c5862299e6
Minor Solidity suggestion |
||
---|---|---|
cmd | ||
db | ||
dockerfiles | ||
environments | ||
integration_test | ||
libraries/shared | ||
pkg | ||
plugins | ||
postgraphile | ||
scripts | ||
test_config | ||
utils | ||
vendor | ||
.dockerignore | ||
.gitignore | ||
.travis.yml | ||
Dockerfile | ||
Gopkg.lock | ||
Gopkg.toml | ||
LICENSE | ||
main.go | ||
Makefile | ||
README.md |
Vulcanize DB
About
Vulcanize DB is a set of tools that make it easier for developers to write application-specific indexes and caches for dapps built on Ethereum.
Dependencies
- Go 1.11+
- Postgres 10.6
- Ethereum Node
- Go Ethereum (1.8.23+)
- Parity 1.8.11+
Project Setup
Using Vulcanize for the first time requires several steps be done in order to allow use of the software. The following instructions will offer a guide through the steps of the process:
- Fetching the project
- Installing dependencies
- Configuring shell environment
- Database setup
- Configuring synced Ethereum node integration
- Data syncing
Installation
In order to fetch the project codebase for local use or modification, install it to your GOPATH
via:
go get github.com/vulcanize/vulcanizedb
go get gopkg.in/DataDog/dd-trace-go.v1/ddtrace
Once fetched, dependencies can be installed via go get
or (the preferred method) at specific versions via golang/dep
, the prototype golang pakcage manager. Installation instructions are here.
In order to install packages with dep
, ensure you are in the project directory now within your GOPATH
(default location is ~/go/src/github.com/vulcanize/vulcanizedb/
) and run:
dep ensure
After dep
finishes, dependencies should be installed within your GOPATH
at the versions specified in Gopkg.toml
.
Because we are working with a modified version of the go-ethereum accounts/abi package, after running dep ensure
you will need to run git checkout vendor/github.com/ethereum/go-ethereum/accounts/abi
to checkout the modified dependency
Lastly, ensure that GOPATH
is defined in your shell. If necessary, GOPATH
can be set in ~/.bashrc
or ~/.bash_profile
, depending upon your system. It can be additionally helpful to add $GOPATH/bin
to your shell's $PATH
.
Setting up the Database
-
Install Postgres
-
Create a superuser for yourself and make sure
psql --list
works without prompting for a password. -
createdb vulcanize_public
-
cd $GOPATH/src/github.com/vulcanize/vulcanizedb
-
Run the migrations:
make migrate HOST_NAME=localhost NAME=vulcanize_public PORT=5432
- To rollback a single step:
make rollback NAME=vulcanize_public
- To rollback to a certain migration:
make rollback_to MIGRATION=n NAME=vulcanize_public
- To see status of migrations:
make migration_status NAME=vulcanize_public
- See below for configuring additional environments
- To rollback a single step:
Create a migration file
make new_migration NAME=add_columnA_to_table1
- This will create a new timestamped migration file in
db/migrations
- This will create a new timestamped migration file in
- Write the migration code in the created file, under the respective
goose
pragma- Goose automatically runs each migration in a transaction; don't add
BEGIN
andCOMMIT
statements.
- Goose automatically runs each migration in a transaction; don't add
Configuration
-
To use a local Ethereum node, copy
environments/public.toml.example
toenvironments/public.toml
and update theipcPath
andlevelDbPath
.-
ipcPath
should match the local node's IPC filepath:-
For Geth:
- The IPC file is called
geth.ipc
. - The geth IPC file path is printed to the console when you start geth.
- The default location is:
- Mac:
<full home path>/Library/Ethereum
- Linux:
<full home path>/ethereum/geth.ipc
- Mac:
- The IPC file is called
-
For Parity:
- The IPC file is called
jsonrpc.ipc
. - The default location is:
- Mac:
<full home path>/Library/Application\ Support/io.parity.ethereum/
- Linux:
<full home path>/local/share/io.parity.ethereum/
- Mac:
- The IPC file is called
-
-
levelDbPath
should match Geth's chaindata directory path.- The geth LevelDB chaindata path is printed to the console when you start geth.
- The default location is:
- Mac:
<full home path>/Library/Ethereum/geth/chaindata
- Linux:
<full home path>/ethereum/geth/chaindata
- Mac:
levelDbPath
is irrelevant (andcoldImport
is currently unavailable) if only running parity.
-
-
See
environments/infura.toml
to configure commands to run against infura, if a local node is unavailable. -
Copy
environments/local.toml.example
toenvironments/local.toml
to configure commands to run against a local node such as Ganache or ganache-cli.
Start syncing with postgres
Syncs VulcanizeDB with the configured Ethereum node, populating blocks, transactions, receipts, and logs. This command is useful when you want to maintain a broad cache of what's happening on the blockchain.
- Start Ethereum node (if fast syncing your Ethereum node, wait for initial sync to finish)
- In a separate terminal start VulcanizeDB:
./vulcanizedb sync --config <config.toml> --starting-block-number <block-number>
Alternatively, sync from Geth's underlying LevelDB
Sync VulcanizeDB from the LevelDB underlying a Geth node.
- Assure node is not running, and that it has synced to the desired block height.
- Start vulcanize_db
./vulcanizedb coldImport --config <config.toml>
- Optional flags:
--starting-block-number <block number>
/-s <block number>
: block number to start syncing from--ending-block-number <block number>
/-e <block number>
: block number to sync to--all
/-a
: sync all missing blocks
Alternatively, sync in "light" mode
Syncs VulcanizeDB with the configured Ethereum node, populating only block headers. This command is useful when you want a minimal baseline from which to track targeted data on the blockchain (e.g. individual smart contract storage values).
- Start Ethereum node
- In a separate terminal start VulcanizeDB:
./vulcanizedb lightSync --config <config.toml> --starting-block-number <block-number>
Start full environment in docker by single command
Geth Rinkeby
make command | description |
---|---|
rinkeby_env_up | start geth, postgres and rolling migrations, after migrations done starting vulcanizedb container |
rinkeby_env_deploy | build and run vulcanizedb container in rinkeby environment |
rinkeby_env_migrate | build and run rinkeby env migrations |
rinkeby_env_down | stop and remove all rinkeby env containers |
Success run of the VulcanizeDB container require full geth state sync, attach to geth console and check sync state:
$ docker exec -it rinkeby_vulcanizedb_geth geth --rinkeby attach
...
> eth.syncing
false
If you have full rinkeby chaindata you can move it to rinkeby_vulcanizedb_geth_data
docker volume to skip long wait of sync.
Running the Tests
createdb vulcanize_private
will create the test dbmake migrate NAME=vulcanize_private
will run the db migrationsmake test
will run the unit tests and skip the integration testsmake integrationtest
will run the just the integration tests
Deploying
- you will need to make sure you have ssh agent running and your ssh key added to it. instructions here
go get -u github.com/pressly/sup/cmd/sup
sup staging deploy
Contract Watchers
Contract watchers work with a light or full sync vDB to fetch raw ethereum data and execute a set of transformations over them, persisting the output.
A watcher is composed of at least a fetcher and a transformer or set of transformers, where a fetcher is an interface for retrieving raw Ethereum data from some source (e.g. eth_jsonrpc, IPFS) and a transformer is an interface for filtering through that raw Ethereum data to extract, process, and persist data for specific contracts or accounts.
omniWatcher
The omniWatcher
command is a built-in generic contract watcher. It can watch any and all events for a given contract provided the contract's ABI is available.
It also provides some state variable coverage by automating polling of public methods, with some restrictions.
This command requires a pre-synced (full or light) vulcanizeDB (see above sections) and currently requires the contract ABI be available on etherscan or provided by the user.
To watch all events of a contract using a light synced vDB:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address>
Or if you are using a full synced vDB, change the mode to full:
- Execute ./vulcanizedb omniWatcher --mode full --config <path to config.toml> --contract-address <contract address>
To watch contracts on a network other than mainnet, use the network flag:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --network <ropsten, kovan, or rinkeby>
To watch events starting at a certain block use the starting block flag:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --starting-block-number <#>
To watch only specified events use the events flag:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --events <EventName1> --events <EventName2>
To watch events and poll the specified methods with any addresses and hashes emitted by the watched events utilize the methods flag:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --methods <methodName1> --methods <methodName2>
To watch specified events and poll the specified method with any addresses and hashes emitted by the watched events:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --events <EventName1> --events <EventName2> --methods <methodName>
To turn on method piping so that values returned from previous method calls are cached and used as arguments in subsequent method calls:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --piping true --contract-address <contract address> --events <EventName1> --events <EventName2> --methods <methodName>
To watch all types of events of the contract but only persist the ones that emit one of the filtered-for argument values:
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --event-args <arg1> --event-args <arg2>
To watch all events of the contract but only poll the specified method with specified argument values (if they are emitted from the watched events):
- Execute ./vulcanizedb omniWatcher --config <path to config.toml> --contract-address <contract address> --methods <methodName> --method-args <arg1> --method-args <arg2>
omniWatcher output
Transformed events and polled method results are committed to Postgres in schemas and tables generated according to the contract abi.
Schemas are created for each contract using the naming convention <sync-type>_<lowercase contract-address>
Under this schema, tables are generated for watched events as <lowercase event name>_event
and for polled methods as <lowercase method name>_method
The 'method' and 'event' identifiers are tacked onto the end of the table names to prevent collisions between methods and events of the same lowercase name
Example:
Running ./vulcanizedb omniWatcher --config <path to config> --starting-block-number=5197514 --contract-address=0x8dd5fbce2f6a956c3022ba3663759011dd51e73e --events=Transfer --events=Mint --methods=balanceOf
watches Transfer and Mint events of the TrueUSD contract and polls its balanceOf method using the addresses we find emitted from those events
It produces and populates a schema with three tables:
light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.transfer_event
light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.mint_event
light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.balanceof_method
Column ids and types for these tables are generated based on the event and method argument names and types and method return types, resulting in tables such as:
Table "light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.transfer_event"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description |
---|---|---|---|---|---|---|---|
id | integer | not null | nextval('light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.transfer_event_id_seq'::regclass) | plain | |||
header_id | integer | not null | plain | ||||
token_name | character varying(66) | not null | extended | ||||
raw_log | jsonb | extended | |||||
log_idx | integer | not null | plain | ||||
tx_idx | integer | not null | plain | ||||
from_ | character varying(66) | not null | extended | ||||
to_ | character varying(66) | not null | extended | ||||
value_ | numeric | not null | main |
Table "light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.balanceof_method"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description |
---|---|---|---|---|---|---|---|
id | integer | not null | nextval('light_0x8dd5fbce2f6a956c3022ba3663759011dd51e73e.balanceof_method_id_seq'::regclass) | plain | |||
token_name | character varying(66) | not null | extended | ||||
block | integer | not null | plain | ||||
who_ | character varying(66) | not null | extended | ||||
returned | numeric | not null | main |
The addition of '_' after table names is to prevent collisions with reserved Postgres words
composeAndExecute
The composeAndExecute
command is used to compose and execute over an arbitrary set of custom transformers.
This is accomplished by generating a Go pluggin which allows our vulcanizedb
binary to link to external transformers, so
long as they abide by our standard interfaces.
This command requires Go 1.11+ and Go plugins only work on Unix based systems.
Writing custom transformers
Storage Transformers
Event Transformers
- Guide
- Example
composeAndExecute configuration
A .toml config file is specified when executing the command:
./vulcanizedb composeAndExecute --config=./environments/config_name.toml
The config provides information for composing a set of transformers:
[database]
name = "vulcanize_public"
hostname = "localhost"
user = "vulcanize"
password = "vulcanize"
port = 5432
[client]
ipcPath = "http://kovan0.vulcanize.io:8545"
[exporter]
home = "github.com/vulcanize/vulcanizedb"
name = "exampleTransformerExporter"
save = false
transformerNames = [
"transformer1",
"transformer2",
"transformer3",
"transformer4",
]
[exporter.transformer1]
path = "path/to/transformer1"
type = "eth_event"
repository = "github.com/account/repo"
migrations = "db/migrations"
[exporter.transformer2]
path = "path/to/transformer2"
type = "eth_event"
repository = "github.com/account/repo"
migrations = "db/migrations"
[exporter.transformer3]
path = "path/to/transformer3"
type = "eth_event"
repository = "github.com/account/repo"
migrations = "db/migrations"
[exporter.transformer4]
path = "path/to/transformer4"
type = "eth_storage"
repository = "github.com/account2/repo2"
migrations = "to/db/migrations"
home
is the name of the package you are building the plugin for, in most cases this is github.com/vulcanize/vulcanizedbname
is the name used for the plugin files (.so and .go)save
indicates whether or not the user wants to save the .go file instead of removing it after .so compilation. Sometimes useful for debugging/trouble-shooting purposes.transformerNames
is the list of the names of the transformers we are composing together, so we know how to access their submaps in the exporter mapexporter.<transformerName>
s are the sub-mappings containing config info for the transformersrepository
is the path for the repository which contains the transformer and itsTransformerInitializer
path
is the relative path fromrepository
to the transformer'sTransformerInitializer
directory (initializer package).- Transformer repositories need to be cloned into the user's $GOPATH (
go get
)
- Transformer repositories need to be cloned into the user's $GOPATH (
type
is the type of the transformer; indicating which type of watcher it works with (for now, there are only two options:eth_event
andeth_storage
)eth_storage
indicates the transformer works with the storage watcher that fetches state and storage diffs from an ETH node (instead of, for example, from IPFS)eth_event
indicates the transformer works with the event watcher that fetches event logs from an ETH node
migrations
is the relative path fromrepository
to the db migrations directory for the transformer
- Note: If any of the imported transformers need additional config variables those need to be included as well
This information is used to write and build a Go plugin which exports the configured transformers. These transformers are loaded onto their specified watchers and executed.
Transformers of different types can be run together in the same command using a single config file or in separate instances using different config files
The general structure of a plugin .go file, and what we would see built with the above config is shown below
package main
import (
interface1 "github.com/vulcanize/vulcanizedb/libraries/shared/transformer"
transformer1 "github.com/account/repo/path/to/transformer1"
transformer2 "github.com/account/repo/path/to/transformer2"
transformer3 "github.com/account/repo/path/to/transformer3"
transformer4 "github.com/account2/repo2/path/to/transformer4"
)
type exporter string
var Exporter exporter
func (e exporter) Export() []interface1.TransformerInitializer, []interface1.StorageTransformerInitializer {
return []interface1.TransformerInitializer{
transformer1.TransformerInitializer,
transformer2.TransformerInitializer,
transformer3.TransformerInitializer,
}, []interface1.StorageTransformerInitializer{
transformer4.StorageTransformerInitializer,
}
}
Preparing transformer(s) to work as pluggins for composeAndExecute
To plug in an external transformer we need to:
- Create a package
that exports a variable
TransformerInitializer
orStorageTransformerInitializer
that are of type TransformerInitializer or StorageTransformerInitializer, respectively - Design the transformers to work in the context of their event or storage watchers
- Create db migrations to run against vulcanizeDB so that we can store the transformer output
- Specify migration locations for each transformer in the config with the
exporter.transformer.migrations
fields - Do not
goose fix
the transformer migrations
- Specify migration locations for each transformer in the config with the
To update a plugin repository with changes to the core vulcanizedb repository, replace the vulcanizedb vendored in the plugin repo (plugin_repo/vendor/github.com/vulcanize/vulcanizedb
)
with the newly updated version
- The entire vendor lib within the vendored vulcanizedb needs to be deleted (
plugin_repo/vendor/github.com/vulcanize/vulcanizedb/vendor
) - These complications arise due to this conflict between
dep
and Go plugins