59cdaa05e6
execution cycles and fixing test db tear down so that it drops checked_headers table instead of individual columns (droping columns only hides them, they remain in memory and continue to contribute to the 1600 column limit which causes us to eventually exceed it) |
||
---|---|---|
cmd | ||
db | ||
dockerfiles/rinkeby | ||
environments | ||
examples | ||
integration_test | ||
libraries/shared | ||
pkg | ||
scripts | ||
test_config | ||
utils | ||
vendor | ||
.gitignore | ||
.private_blockchain_password | ||
.travis.yml | ||
Gopkg.lock | ||
Gopkg.toml | ||
LICENSE | ||
main.go | ||
Makefile | ||
README.md |
Vulcanize DB
About
Vulcanize DB is a set of tools that make it easier for developers to write application-specific indexes and caches for dapps built on Ethereum.
Dependencies
- Go 1.11+
- Postgres 10
- Ethereum Node
- Go Ethereum (1.8.18+)
- Parity 1.8.11+
Project Setup
Using Vulcanize for the first time requires several steps be done in order to allow use of the software. The following instructions will offer a guide through the steps of the process:
- Fetching the project
- Installing dependencies
- Configuring shell environment
- Database setup
- Configuring synced Ethereum node integration
- Data syncing
Installation
In order to fetch the project codebase for local use or modification, install it to your GOPATH
via:
go get github.com/vulcanize/vulcanizedb
Once fetched, dependencies can be installed via go get
or (the preferred method) at specific versions via golang/dep
, the prototype golang pakcage manager. Installation instructions are here.
In order to install packages with dep
, ensure you are in the project directory now within your GOPATH
(default location is ~/go/src/github.com/vulcanize/vulcanizedb/
) and run:
dep ensure
After dep
finishes, dependencies should be installed within your GOPATH
at the versions specified in Gopkg.toml
.
Lastly, ensure that GOPATH
is defined in your shell. If necessary, GOPATH
can be set in ~/.bashrc
or ~/.bash_profile
, depending upon your system. It can be additionally helpful to add $GOPATH/bin
to your shell's $PATH
.
Setting up the Database
-
Install Postgres
-
Create a superuser for yourself and make sure
psql --list
works without prompting for a password. -
Execute
createdb vulcanize_public
-
Execute
cd $GOPATH/src/github.com/vulcanize/vulcanizedb
-
Run the migrations:
make migrate HOST_NAME=localhost NAME=vulcanize_public PORT=<postgres port, default 5432>
- See below for configuring additional environments
In some cases (such as recent Ubuntu systems), it may be necessary to overcome failures of password authentication from localhost
. To allow access on Ubuntu, set localhost connections via hostname, ipv4, and ipv6 from peer
/md5
to trust
in: /etc/postgresql/<version>/pg_hba.conf
(It should be noted that trusted auth should only be enabled on systems without sensitive data in them: development and local test databases.)
Configuring Ethereum Node Integration
-
To use a local Ethereum node, copy
environments/public.toml.example
toenvironments/public.toml
and update theipcPath
andlevelDbPath
.-
ipcPath
should match the local node's IPC filepath:-
For Geth:
- The IPC file is called
geth.ipc
. - The geth IPC file path is printed to the console when you start geth.
- The default location is:
- Mac:
<full home path>/Library/Ethereum
- Linux:
<full home path>/ethereum/geth.ipc
- Mac:
- The IPC file is called
-
For Parity:
- The IPC file is called
jsonrpc.ipc
. - The default location is:
- Mac:
<full home path>/Library/Application\ Support/io.parity.ethereum/
- Linux:
<full home path>/local/share/io.parity.ethereum/
- Mac:
- The IPC file is called
-
-
levelDbPath
should match Geth's chaindata directory path.- The geth LevelDB chaindata path is printed to the console when you start geth.
- The default location is:
- Mac:
<full home path>/Library/Ethereum/geth/chaindata
- Linux:
<full home path>/ethereum/geth/chaindata
- Mac:
levelDbPath
is irrelevant (andcoldImport
is currently unavailable) if only running parity.
-
-
See
environments/infura.toml
to configure commands to run against infura, if a local node is unavailable. (Support is currently experimental, at this time.)
Start syncing with postgres
Syncs VulcanizeDB with the configured Ethereum node.
- Start the node
- If node state is not yet fully synced, Vulcanize will not be able to operate on the fetched data. You will need to wait for the initial sync to finish.
- Start the vulcanize_db sync or lightSync
- Execute
./vulcanizedb sync --config <path to config.toml>
- Or
./vulcanizedb lightSync --config <path to config.toml>
- Or to sync from a specific block:
./vulcanizedb sync --config <config.toml> --starting-block-number <block-number>
- Or
./vulcanizedb lightSync --config <config.toml> --starting-block-number <block-number>
- Execute
Alternatively, sync from Geth's underlying LevelDB
Sync VulcanizeDB from the LevelDB underlying a Geth node.
- Assure node is not running, and that it has synced to the desired block height.
- Start vulcanize_db
./vulcanizedb coldImport --config <config.toml>
- Optional flags:
--starting-block-number <block number>
/-s <block number>
: block number to start syncing from--ending-block-number <block number>
/-e <block number>
: block number to sync to--all
/-a
: sync all missing blocks
Running the Tests
In order to run the full test suite, a test database must be prepared. By default, the rests use a database named vulcanize_private
. Create the database in Postgres, and run migrations on the new database in preparation for executing tests:
make migrate HOST_NAME=localhost NAME=vulcanize_private PORT=<postgres port, default 5432>
Ginkgo is declared as a dep
package test execution. Linting and tests can be run together via a provided make
task:
make test
Tests can be run directly via Ginkgo in the project's root directory:
ginkgo -r
Start full environment in docker by single command
Geth Rinkeby
make command | description |
---|---|
rinkeby_env_up | start geth, postgres and rolling migrations, after migrations done starting vulcanizedb container |
rinkeby_env_deploy | build and run vulcanizedb container in rinkeby environment |
rinkeby_env_migrate | build and run rinkeby env migrations |
rinkeby_env_down | stop and remove all rinkeby env containers |
Success run of the VulcanizeDB container require full geth state sync, attach to geth console and check sync state:
$ docker exec -it rinkeby_vulcanizedb_geth geth --rinkeby attach
...
> eth.syncing
false
If you have full rinkeby chaindata you can move it to rinkeby_vulcanizedb_geth_data
docker volume to skip long wait of sync.
omniWatcher and lightOmniWatcher
These commands require a pre-synced (full or light) vulcanizeDB (see above sections)
To watch all events of a contract using a light synced vDB:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address>
Or if you are using a full synced vDB, change the mode to full:
- Execute ./vulcanizedb omniWatcher --mode full --config <path to config.toml> --contract-address <contract address>
To watch contracts on a network other than mainnet, use the network flag:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --network <ropsten, kovan, or rinkeby>
To watch events starting at a certain block use the starting block flag:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --starting-block-number <#>
To watch only specified events use the events flag:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --events <EventName1> --events <EventName2>
To watch events and poll the specified methods with any addresses and hashes emitted by the watched events utilize the methods flag:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --methods <methodName1> --methods <methodName2>
To watch specified events and poll the specified method with any addresses and hashes emitted by the watched events:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --events <EventName1> --events <EventName2> --methods <methodName>
To turn on method piping so that values returned from previous method calls are cached and used as arguments in subsequent method calls:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --piping true --contract-address <contract address> --events <EventName1> --events <EventName2> --methods <methodName>
To watch all types of events of the contract but only persist the ones that emit one of the filtered-for argument values:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --event-args <arg1> --event-args <arg2>
To watch all events of the contract but only poll the specified method with specified argument values (if they are emitted from the watched events):
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --methods <methodName> --method-args <arg1> --method-args <arg2>