update readme
This commit is contained in:
Ian Norden
parent
0d28234804
commit
1dee766014
188
README.md
188
README.md
@ -2,24 +2,23 @@
|
||||
|
||||
[](https://goreportcard.com/report/github.com/vulcanize/ipld-eth-server)
|
||||
|
||||
> ipld-eth-server is used to extract, transform, and load all eth or btc data into an IPFS-backing Postgres datastore while generating useful secondary indexes around the data in other Postgres tables
|
||||
> ipld-eth-server is the server backend for indexed ETH IPLD objects
|
||||
|
||||
## Table of Contents
|
||||
1. [Background](#background)
|
||||
1. [Architecture](#architecture)
|
||||
1. [Install](#install)
|
||||
1. [Usage](#usage)
|
||||
1. [Contributing](#contributing)
|
||||
1. [License](#license)
|
||||
|
||||
## Background
|
||||
ipld-eth-server is a collection of interfaces that are used to extract, process, store, and index
|
||||
all blockchain data in Postgres-IPFS. The raw data indexed by ipld-eth-server serves as the basis for more specific watchers and applications.
|
||||
NOTE: WIP
|
||||
|
||||
Currently the service supports complete processing of all Bitcoin and Ethereum data.
|
||||
ipld-eth-server is used to service queries against the indexed Ethereum IPLD objects indexed by [ipld-eth-indexer](https://github.com/vulcanize/ipld-eth-indexer).
|
||||
|
||||
It exposes standard Ethereum JSON RPC endpoints on top of the database, in some cases these endpoints can leverage the unique indexes to improve query performance.
|
||||
Additional, unique endpoints are exposed which utilize the new indexes and state diff data objects.
|
||||
|
||||
## Architecture
|
||||
More details on the design of ipld-eth-server can be found in [here](./documentation/architecture.md)
|
||||
|
||||
## Dependencies
|
||||
Minimal build dependencies
|
||||
@ -28,111 +27,10 @@ Minimal build dependencies
|
||||
* GCC compiler
|
||||
* This repository
|
||||
|
||||
Potential external dependencies
|
||||
* Goose
|
||||
* Postgres
|
||||
* Statediffing go-ethereum
|
||||
* Bitcoin node
|
||||
External dependency
|
||||
* Postgres database populated by [ipld-eth-indexer](https://github.com/vulcanize/ipld-eth-indexer)
|
||||
|
||||
## Install
|
||||
1. [Goose](#goose)
|
||||
1. [Postgres](#postgres)
|
||||
1. [IPFS](#ipfs)
|
||||
1. [Blockchain](#blockchain)
|
||||
1. [Watcher](#watcher)
|
||||
|
||||
### Goose
|
||||
[goose](https://github.com/pressly/goose) is used for migration management. While it is not necessary to use `goose` for manual setup, it
|
||||
is required for running the automated tests and is used by the `make migrate` command.
|
||||
|
||||
### Postgres
|
||||
1. [Install Postgres](https://wiki.postgresql.org/wiki/Detailed_installation_guides)
|
||||
1. Create a superuser for yourself and make sure `psql --list` works without prompting for a password.
|
||||
1. `createdb vulcanize_public`
|
||||
1. `cd $GOPATH/src/github.com/vulcanize/ipld-eth-server`
|
||||
1. Run the migrations: `make migrate HOST_NAME=localhost NAME=vulcanize_public PORT=5432`
|
||||
- There are optional vars `USER=username:password` if the database user is not the default user `postgres` and/or a password is present
|
||||
- To rollback a single step: `make rollback NAME=vulcanize_public`
|
||||
- To rollback to a certain migration: `make rollback_to MIGRATION=n NAME=vulcanize_public`
|
||||
- To see status of migrations: `make migration_status NAME=vulcanize_public`
|
||||
|
||||
* See below for configuring additional environments
|
||||
|
||||
In some cases (such as recent Ubuntu systems), it may be necessary to overcome failures of password authentication from
|
||||
localhost. To allow access on Ubuntu, set localhost connections via hostname, ipv4, and ipv6 from peer/md5 to trust in: /etc/postgresql/<version>/pg_hba.conf
|
||||
|
||||
(It should be noted that trusted auth should only be enabled on systems without sensitive data in them: development and local test databases)
|
||||
|
||||
### IPFS
|
||||
Data is stored in an [IPFS-backing Postgres datastore](https://github.com/ipfs/go-ds-sql).
|
||||
By default data is written directly to the ipfs blockstore in Postgres; the public.blocks table.
|
||||
In this case no further IPFS configuration is needed at this time.
|
||||
|
||||
Optionally, ipld-eth-server can be configured to function through an internal ipfs node interface using the flag: `-ipfs-mode=interface`.
|
||||
Operating through the ipfs interface provides the option to configure a block exchange that can search remotely for IPLD data found missing in the local datastore.
|
||||
This option is irrelevant in most cases and this mode has some disadvantages, namely:
|
||||
|
||||
1. Environment must have IPFS configured
|
||||
1. Process will contend with the lockfile at `$IPFS_PATH`
|
||||
1. Publishing and indexing of data must occur in separate db transactions
|
||||
|
||||
More information for configuring Postgres-IPFS can be found [here](./documentation/ipfs.md)
|
||||
|
||||
### Blockchain
|
||||
This section describes how to setup an Ethereum or Bitcoin node to serve as a data source for ipld-eth-server
|
||||
|
||||
#### Ethereum
|
||||
For Ethereum, [a special fork of go-ethereum](https://github.com/vulcanize/go-ethereum/tree/statediff_at_anyblock-1.9.11) is currently *requirde*.
|
||||
This can be setup as follows.
|
||||
Skip this step if you already have access to a node that displays the statediffing endpoints.
|
||||
|
||||
Begin by downloading geth and switching to the statediffing branch:
|
||||
|
||||
`GO111MODULE=off go get -d github.com/ethereum/go-ethereum`
|
||||
|
||||
`cd $GOPATH/src/github.com/ethereum/go-ethereum`
|
||||
|
||||
`git remote add vulcanize https://github.com/vulcanize/go-ethereum.git`
|
||||
|
||||
`git fetch vulcanize`
|
||||
|
||||
`git checkout -b statediffing vulcanize/statediff_at_anyblock-1.9.11`
|
||||
|
||||
Now, install this fork of geth (make sure any old versions have been uninstalled/binaries removed first):
|
||||
|
||||
`make geth`
|
||||
|
||||
And run the output binary with statediffing turned on:
|
||||
|
||||
`cd $GOPATH/src/github.com/ethereum/go-ethereum/build/bin`
|
||||
|
||||
`./geth --syncmode=full --statediff --ws`
|
||||
|
||||
Note: to access historical data (perform `backFill`) the node will need to operate as an archival node (`--gcmode=archive`) with rpc endpoints
|
||||
exposed (`--rpc --rpcapi=eth,statediff,net`)
|
||||
|
||||
Warning: There is a good chance even a fully synced archive node has incomplete historical state data to some degree
|
||||
|
||||
The output from geth should mention that it is `Starting statediff service` and block synchronization should begin shortly thereafter.
|
||||
Note that until it receives a subscriber, the statediffing process does nothing but wait for one. Once a subscription is received, this
|
||||
will be indicated in the output and the node will begin processing and sending statediffs.
|
||||
|
||||
Also in the output will be the endpoints that will be used to interface with the node.
|
||||
The default ws url is "127.0.0.1:8546" and the default http url is "127.0.0.1:8545".
|
||||
These values will be used as the `ethereum.wsPath` and `ethereum.httpPath` in the config, respectively.
|
||||
|
||||
#### Bitcoin
|
||||
For Bitcoin, ipld-eth-server is able to operate entirely through the universally exposed JSON-RPC interfaces.
|
||||
This means any of the standard full nodes can be used (e.g. bitcoind, btcd) as the data source.
|
||||
|
||||
Point at a remote node or set one up locally using the instructions for [bitcoind](https://github.com/bitcoin/bitcoin) and [btcd](https://github.com/btcsuite/btcd).
|
||||
|
||||
The default http url is "127.0.0.1:8332". We will use the http endpoint as both the `bitcoin.wsPath` and `bitcoin.httpPath`
|
||||
(bitcoind does not support websocket endpoints, the watcher currently uses a "subscription" wrapper around the http endpoints)
|
||||
|
||||
### Watcher
|
||||
Finally, setup the watcher process itself.
|
||||
|
||||
Start by downloading ipld-eth-server and moving into the repo:
|
||||
|
||||
`GO111MODULE=off go get -d github.com/vulcanize/ipld-eth-server`
|
||||
@ -146,68 +44,50 @@ Then, build the binary:
|
||||
## Usage
|
||||
After building the binary, run as
|
||||
|
||||
`./ipld-eth-server watch --config=<the name of your config file.toml>`
|
||||
`./ipld-eth-server serve --config=<the name of your config file.toml>`
|
||||
|
||||
### Configuration
|
||||
|
||||
Below is the set of universal config parameters for the ipld-eth-server command, in .toml form, with the respective environmental variables commented to the side.
|
||||
This set of parameters needs to be set no matter the chain type.
|
||||
Below is the set of parameters for the ipld-eth-server command, in .toml form, with the respective environmental variables commented to the side.
|
||||
The corresponding CLI flags can be found with the `./ipld-eth-server serve --help` command.
|
||||
|
||||
```toml
|
||||
[database]
|
||||
name = "vulcanize_public" # $DATABASE_NAME
|
||||
hostname = "localhost" # $DATABASE_HOSTNAME
|
||||
port = 5432 # $DATABASE_PORT
|
||||
user = "vdbm" # $DATABASE_USER
|
||||
user = "postgres" # $DATABASE_USER
|
||||
password = "" # $DATABASE_PASSWORD
|
||||
|
||||
[watcher]
|
||||
chain = "bitcoin" # $SUPERNODE_CHAIN
|
||||
server = true # $SUPERNODE_SERVER
|
||||
ipcPath = "~/.vulcanize/vulcanize.ipc" # $SUPERNODE_IPC_PATH
|
||||
wsPath = "127.0.0.1:8082" # $SUPERNODE_WS_PATH
|
||||
httpPath = "127.0.0.1:8083" # $SUPERNODE_HTTP_PATH
|
||||
sync = true # $SUPERNODE_SYNC
|
||||
workers = 1 # $SUPERNODE_WORKERS
|
||||
backFill = true # $SUPERNODE_BACKFILL
|
||||
frequency = 45 # $SUPERNODE_FREQUENCY
|
||||
batchSize = 1 # $SUPERNODE_BATCH_SIZE
|
||||
batchNumber = 50 # $SUPERNODE_BATCH_NUMBER
|
||||
timeout = 300 # $HTTP_TIMEOUT
|
||||
validationLevel = 1 # $SUPERNODE_VALIDATION_LEVEL
|
||||
[log]
|
||||
level = "info" # $LOGRUS_LEVEL
|
||||
|
||||
[server]
|
||||
ipcPath = "~/.vulcanize/vulcanize.ipc" # $SERVER_IPC_PATH
|
||||
wsPath = "127.0.0.1:8081" # $SERVER_WS_PATH
|
||||
httpPath = "127.0.0.1:8082" # $SERVER_HTTP_PATH
|
||||
```
|
||||
|
||||
Additional parameters need to be set depending on the specific chain.
|
||||
The `database` fields are for connecting to a Postgres database that has been/is being populated by [ipld-eth-indexer](https://github.com/vulcanize/ipld-eth-indexer).
|
||||
The `server` fields set the paths for exposing the ipld-eth-server endpoints
|
||||
|
||||
For Bitcoin:
|
||||
|
||||
```toml
|
||||
[bitcoin]
|
||||
wsPath = "127.0.0.1:8332" # $BTC_WS_PATH
|
||||
httpPath = "127.0.0.1:8332" # $BTC_HTTP_PATH
|
||||
pass = "password" # $BTC_NODE_PASSWORD
|
||||
user = "username" # $BTC_NODE_USER
|
||||
nodeID = "ocd0" # $BTC_NODE_ID
|
||||
clientName = "Omnicore" # $BTC_CLIENT_NAME
|
||||
genesisBlock = "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f" # $BTC_GENESIS_BLOCK
|
||||
networkID = "0xD9B4BEF9" # $BTC_NETWORK_ID
|
||||
```
|
||||
### Endpoints
|
||||
#### IPLD subscription
|
||||
TODO: Port the IPLD RPC subscription endpoints after the decoupling
|
||||
|
||||
For Ethereum:
|
||||
#### Ethereum JSON-RPC
|
||||
ipld-eth-server currently recapitulates portions of the Ethereum JSON-RPC api standard.
|
||||
|
||||
```toml
|
||||
[ethereum]
|
||||
wsPath = "127.0.0.1:8546" # $ETH_WS_PATH
|
||||
httpPath = "127.0.0.1:8545" # $ETH_HTTP_PATH
|
||||
nodeID = "arch1" # $ETH_NODE_ID
|
||||
clientName = "Geth" # $ETH_CLIENT_NAME
|
||||
genesisBlock = "0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3" # $ETH_GENESIS_BLOCK
|
||||
networkID = "1" # $ETH_NETWORK_ID
|
||||
chainID = "1" # $ETH_CHAIN_ID
|
||||
```
|
||||
The currently supported standard endpoints are:
|
||||
`eth_blockNumber`
|
||||
`eth_getLogs`
|
||||
`eth_getHeaderByNumber`
|
||||
`eth_getBlockByNumber`
|
||||
`eth_getBlockByHash`
|
||||
`eth_getTransactionByHash`
|
||||
|
||||
### Exposing the data
|
||||
A number of different APIs for remote access to ipld-eth-server data can be exposed, these are discussed in more detail [here](./documentation/apis.md)
|
||||
TODO: Add the rest of the standard endpoints add unique endpoints (e.g. getSlice)
|
||||
|
||||
### Testing
|
||||
`make test` will run the unit tests
|
||||
|
||||
30
environments/subscribeExample.toml
Normal file
30
environments/subscribeExample.toml
Normal file
@ -0,0 +1,30 @@
|
||||
[watcher]
|
||||
[watcher.ethSubscription]
|
||||
historicalData = false
|
||||
historicalDataOnly = false
|
||||
startingBlock = 0
|
||||
endingBlock = 0
|
||||
wsPath = "ws://127.0.0.1:8080"
|
||||
[watcher.ethSubscription.headerFilter]
|
||||
off = false
|
||||
uncles = false
|
||||
[watcher.ethSubscription.txFilter]
|
||||
off = false
|
||||
src = []
|
||||
dst = []
|
||||
[watcher.ethSubscription.receiptFilter]
|
||||
off = false
|
||||
contracts = []
|
||||
topic0s = []
|
||||
topic1s = []
|
||||
topic2s = []
|
||||
topic3s = []
|
||||
[watcher.ethSubscription.stateFilter]
|
||||
off = false
|
||||
addresses = []
|
||||
intermediateNodes = false
|
||||
[watcher.ethSubscription.storageFilter]
|
||||
off = true
|
||||
addresses = []
|
||||
storageKeys = []
|
||||
intermediateNodes = false
|
||||
Loading…
Reference in New Issue
Block a user