cmd | ||
db | ||
dockerfiles | ||
documentation | ||
environments | ||
pkg | ||
scripts | ||
utils | ||
version | ||
.dockerignore | ||
.gitignore | ||
.travis.yml | ||
Dockerfile | ||
go.mod | ||
go.sum | ||
LICENSE | ||
main.go | ||
Makefile | ||
README.md | ||
temp_rsa.enc |
Vulcanize DB
Tool for extracting and indexing blockchain data on PG-IPFS
Table of Contents
Background
ipfs-blockchain-watcher is a collection of interfaces that are used to extract, process, and store in Postgres-IPFS all chain data. The raw data indexed by ipfs-blockchain-watcher serves as the basis for more specific watchers and applications.
Currently the service supports complete processing of all Bitcoin and Ethereum data.
Architecture
More details on the design of ipfs-blockchain-watcher can be found in here
Install
Postgres
-
Create a superuser for yourself and make sure
psql --list
works without prompting for a password. -
createdb vulcanize_public
-
cd $GOPATH/src/github.com/vulcanize/ipfs-blockchain-watcher
-
Run the migrations:
make migrate HOST_NAME=localhost NAME=vulcanize_public PORT=5432
- There are optional vars
USER=username
andPASS=password
if the database user is not the default userpostgres
and/or a password is present - To rollback a single step:
make rollback NAME=vulcanize_public
- To rollback to a certain migration:
make rollback_to MIGRATION=n NAME=vulcanize_public
- To see status of migrations:
make migration_status NAME=vulcanize_public
- See below for configuring additional environments
- There are optional vars
In some cases (such as recent Ubuntu systems), it may be necessary to overcome failures of password authentication from localhost. To allow access on Ubuntu, set localhost connections via hostname, ipv4, and ipv6 from peer/md5 to trust in: /etc/postgresql//pg_hba.conf
(It should be noted that trusted auth should only be enabled on systems without sensitive data in them: development and local test databases)
Goose
We use goose as our migration management tool. While it is not necessary to use goose
for manual setup, it
is required for running the automated tests.
IPFS
We use IPFS to store IPLD objects for each type of data we extract from on chain.
To start, download and install IPFS:
go get github.com/ipfs/go-ipfs
cd $GOPATH/src/github.com/ipfs/go-ipfs
make install
If we want to use Postgres as our backing datastore, we need to use the vulcanize fork of go-ipfs.
Start by adding the fork and switching over to it:
git remote add vulcanize https://github.com/vulcanize/go-ipfs.git
git fetch vulcanize
git checkout -b postgres_update vulcanize/postgres_update
Now install this fork of ipfs, first be sure to remove any previous installation:
make install
Check that is installed properly by running:
ipfs
You should see the CLI info/help output.
And now we initialize with the postgresds
profile.
If ipfs was previously initialized we will need to remove the old profile first.
We also need to provide env variables for the postgres connection:
We can either set these manually, e.g.
export IPFS_PGHOST=
export IPFS_PGUSER=
export IPFS_PGDATABASE=
export IPFS_PGPORT=
export IPFS_PGPASSWORD=
And then run the ipfs command:
ipfs init --profile=postgresds
Or we can use the pre-made script at GOPATH/src/github.com/ipfs/go-ipfs/misc/utility/ipfs_postgres.sh
which has usage:
./ipfs_postgres.sh <IPFS_PGHOST> <IPFS_PGPORT> <IPFS_PGUSER> <IPFS_PGDATABASE>"
and will ask us to enter the password, avoiding storing it to an ENV variable.
Once we have initialized ipfs, that is all we need to do with it- we do not need to run a daemon during the subsequent processes.
Blockchain
This section describes how to setup an Ethereum or Bitcoin node to serve as a data source for ipfs-blockchain-watcher
Ethereum
For Ethereum, we currently require a special fork of go-ethereum. This can be setup as follows. Skip this steps if you already have access to a node that displays the statediffing endpoints.
Begin by downloading geth and switching to the vulcanize/rpc_statediffing branch:
go get github.com/ethereum/go-ethereum
cd $GOPATH/src/github.com/ethereum/go-ethereum
git remote add vulcanize https://github.com/vulcanize/go-ethereum.git
git fetch vulcanize
git checkout -b statediffing vulcanize/statediff_at_anyblock-1.9.11
Now, install this fork of geth (make sure any old versions have been uninstalled/binaries removed first):
make geth
And run the output binary with statediffing turned on:
cd $GOPATH/src/github.com/ethereum/go-ethereum/build/bin
./geth --statediff --statediff.streamblock --ws --syncmode=full
Note: if you wish to access historical data (perform backFill
) then the node will need to operate as an archival node (--gcmode=archive
)
Note: other CLI options- statediff specific ones included- can be explored with ./geth help
The output from geth should mention that it is Starting statediff service
and block synchronization should begin shortly thereafter.
Note that until it receives a subscriber, the statediffing process does nothing but wait for one. Once a subscription is received, this
will be indicated in the output and node will begin processing and sending statediffs.
Also in the output will be the endpoints that we will use to interface with the node.
The default ws url is "127.0.0.1:8546" and the default http url is "127.0.0.1:8545".
These values will be used as the ethereum.wsPath
and ethereum.httpPath
in the config, respectively.
Bitcoin
For Bitcoin, ipfs-blockchain-watcher is able to operate entirely through the universally exposed JSON-RPC interfaces. This means we can use any of the standard full nodes (e.g. bitcoind, btcd) as our data source.
Point at a remote node or set one up locally using the instructions for bitcoind and btcd.
The default http url is "127.0.0.1:8332". We will use the http endpoint as both the bitcoin.wsPath
and bitcoin.httpPath
(bitcoind does not support websocket endpoints, we are currently using a "subscription" wrapper around the http endpoints)
Watcher
Finally, we can setup the watcher process itself.
Start by downloading vulcanizedb and moving into the repo:
go get github.com/vulcanize/ipfs-chain-watcher
cd $GOPATH/src/github.com/vulcanize/ipfs-chain-watcher
Then, build the binary:
make build
Usage
After building the binary, run as
./ipfs-blockchain-watcher watch --config=<config_file.toml
Configuration
Below is the set of universal config parameters for the ipfs-blockchain-watcher command, in .toml form, with the respective environmental variables commented to the side. This set of parameters needs to be set no matter the chain type.
[database]
name = "vulcanize_public" # $DATABASE_NAME
hostname = "localhost" # $DATABASE_HOSTNAME
port = 5432 # $DATABASE_PORT
user = "vdbm" # $DATABASE_USER
password = "" # $DATABASE_PASSWORD
[ipfs]
path = "~/.ipfs" # $IPFS_PATH
mode = "postgres" # $IPFS_MODE
[superNode]
chain = "bitcoin" # $SUPERNODE_CHAIN
server = true # $SUPERNODE_SERVER
ipcPath = "~/.vulcanize/vulcanize.ipc" # $SUPERNODE_IPC_PATH
wsPath = "127.0.0.1:8082" # $SUPERNODE_WS_PATH
httpPath = "127.0.0.1:8083" # $SUPERNODE_HTTP_PATH
sync = true # $SUPERNODE_SYNC
workers = 1 # $SUPERNODE_WORKERS
backFill = true # $SUPERNODE_BACKFILL
frequency = 45 # $SUPERNODE_FREQUENCY
batchSize = 1 # $SUPERNODE_BATCH_SIZE
batchNumber = 50 # $SUPERNODE_BATCH_NUMBER
timeout = 300 # $HTTP_TIMEOUT
validationLevel = 1 # $SUPERNODE_VALIDATION_LEVEL
Additional parameters need to be set depending on the specific chain.
For Bitcoin:
[bitcoin]
wsPath = "127.0.0.1:8332" # $BTC_WS_PATH
httpPath = "127.0.0.1:8332" # $BTC_HTTP_PATH
pass = "password" # $BTC_NODE_PASSWORD
user = "username" # $BTC_NODE_USER
nodeID = "ocd0" # $BTC_NODE_ID
clientName = "Omnicore" # $BTC_CLIENT_NAME
genesisBlock = "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f" # $BTC_GENESIS_BLOCK
networkID = "0xD9B4BEF9" # $BTC_NETWORK_ID
For Ethereum:
[ethereum]
wsPath = "127.0.0.1:8546" # $ETH_WS_PATH
httpPath = "127.0.0.1:8545" # $ETH_HTTP_PATH
nodeID = "arch1" # $ETH_NODE_ID
clientName = "Geth" # $ETH_CLIENT_NAME
genesisBlock = "0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3" # $ETH_GENESIS_BLOCK
networkID = "1" # $ETH_NETWORK_ID
Exposing the data
We can expose a number of different APIs for remote access to ipfs-blockchain-watcher data, these are dicussed in more detail here
Testing
make test
will run the unit tests
make test
setups a clean vulcanize_testing
db
Contributing
Contributions are welcome!
VulcanizeDB follows the Contributor Covenant Code of Conduct.
License
AGPL-3.0 © Vulcanize Inc