Server backend for indexed ETH IPLD objects
Go to file
2019-04-09 17:08:15 -05:00
cmd PR fixes; remove all infura token references and setup travis to use encrypted env 2019-03-21 18:36:51 -05:00
db Update schema to remove uncles 2019-04-08 15:16:43 -05:00
dockerfiles Nuke OpenRC from docker image, update readme's 2019-02-11 11:27:33 +01:00
documentation review fixes 2019-04-08 11:52:36 -05:00
environments remove LIMIT 100; method to check continuity of headers returned but doesn't require 2019-04-05 17:21:59 -05:00
integration_test adjust block/uncle reward tests and add methods to pkg/geth blockchain and ethclient for direct fetching of eth balances 2019-04-02 10:10:17 -05:00
libraries/shared adjust block/uncle reward tests and add methods to pkg/geth blockchain and ethclient for direct fetching of eth balances 2019-04-02 10:10:17 -05:00
pkg Add timestamp customization for fake header 2019-04-09 17:08:15 -05:00
plugins removing mcd/maker references; delete test_data; update READMEs; use 2019-02-24 21:53:04 -06:00
postgraphile Remove links to Gitter 2019-04-01 16:04:07 -05:00
scripts Remove unused files 2019-03-07 16:29:01 -06:00
test_config full_sync and light_sync receipt tables and light receipt repository methods and test 2019-04-05 16:36:48 -05:00
utils tests for migration path ordering and errors 2019-03-13 11:42:30 -05:00
vendor update vendor dep; constrain to revision of geth with UnpackIntoMap; updates to match 2019-04-03 00:14:02 -05:00
.dockerignore Add container service files and docker README 2019-02-11 11:26:12 +01:00
.gitignore compose and execute as separate commands; update README and add git checkout vendor/github.com/ethereum/go-ethereum/accounts/abi to make build 2019-02-28 18:15:16 -06:00
.travis.yml new travis env variable for tests 2019-03-27 20:10:52 -05:00
Dockerfile Nuke OpenRC from docker image, update readme's 2019-02-11 11:27:33 +01:00
Gopkg.lock update vendor dep; constrain to revision of geth with UnpackIntoMap; updates to match 2019-04-03 00:14:02 -05:00
Gopkg.toml update vendor dep; constrain to revision of geth with UnpackIntoMap; updates to match 2019-04-03 00:14:02 -05:00
LICENSE Edits to address PR issues; change license from apache to AGPL; and work 2018-11-15 12:32:52 -06:00
main.go Assorted cleanups 2019-03-08 11:35:10 -06:00
Makefile update vendor dep; constrain to revision of geth with UnpackIntoMap; updates to match 2019-04-03 00:14:02 -05:00
README.md review fixes 2019-04-08 11:52:36 -05:00

Vulcanize DB

Build Status Go Report Card

Vulcanize DB is a set of tools that make it easier for developers to write application-specific indexes and caches for dapps built on Ethereum.

Table of Contents

  1. Background
  2. Dependencies
  3. Install
  4. Usage
  5. Tests
  6. API
  7. Contributing
  8. License

Background

The same data structures and encodings that make Ethereum an effective and trust-less distributed virtual machine complicate data accessibility and usability for dApp developers.

Dependencies

Install

  1. Building the project
  2. Setting up the database
  3. Configuring a synced Ethereum node

Building the project

Download the codebase to your local GOPATH via:

go get github.com/vulcanize/vulcanizedb

Move to the project directory and use golang/dep to install the dependencies:

cd $GOPATH/src/github.com/vulcanize/vulcanizedb

dep ensure

Once the dependencies have been successfully installed, build the executable with:

make build

If you are running into issues at this stage, ensure that GOPATH is defined in your shell. If necessary, GOPATH can be set in ~/.bashrc or ~/.bash_profile, depending upon your system. It can be additionally helpful to add $GOPATH/bin to your shell's $PATH.

Setting up the database

  1. Install Postgres

  2. Create a superuser for yourself and make sure psql --list works without prompting for a password.

  3. createdb vulcanize_public

  4. cd $GOPATH/src/github.com/vulcanize/vulcanizedb

  5. Run the migrations: make migrate HOST_NAME=localhost NAME=vulcanize_public PORT=5432

    • To rollback a single step: make rollback NAME=vulcanize_public
    • To rollback to a certain migration: make rollback_to MIGRATION=n NAME=vulcanize_public
    • To see status of migrations: make migration_status NAME=vulcanize_public
    • See below for configuring additional environments

In some cases (such as recent Ubuntu systems), it may be necessary to overcome failures of password authentication from localhost. To allow access on Ubuntu, set localhost connections via hostname, ipv4, and ipv6 from peer/md5 to trust in: /etc/postgresql//pg_hba.conf

(It should be noted that trusted auth should only be enabled on systems without sensitive data in them: development and local test databases)

Configuring a synced Ethereum node

  • To use a local Ethereum node, copy environments/public.toml.example to environments/public.toml and update the ipcPath and levelDbPath.
    • ipcPath should match the local node's IPC filepath:

      • For Geth:

        • The IPC file is called geth.ipc.
        • The geth IPC file path is printed to the console when you start geth.
        • The default location is:
          • Mac: <full home path>/Library/Ethereum
          • Linux: <full home path>/ethereum/geth.ipc
      • For Parity:

        • The IPC file is called jsonrpc.ipc.
        • The default location is:
          • Mac: <full home path>/Library/Application\ Support/io.parity.ethereum/
          • Linux: <full home path>/local/share/io.parity.ethereum/
    • levelDbPath should match Geth's chaindata directory path.

      • The geth LevelDB chaindata path is printed to the console when you start geth.
      • The default location is:
        • Mac: <full home path>/Library/Ethereum/geth/chaindata
        • Linux: <full home path>/ethereum/geth/chaindata
      • levelDbPath is irrelevant (and coldImport is currently unavailable) if only running parity.

Usage

Usage is broken up into two processes:

Data syncing

To provide data for transformations, raw Ethereum data must first be synced into vDB. This is accomplished through the use of the lightSync, sync, or coldImport commands. These commands are described in detail here.

Data transformation

Contract watchers use the raw data that has been synced into Postgres to filter out and apply transformations to specific data of interest.

There is a built-in contractWatcher command which provides generic transformation of most contract data. This command is described in detail here.

In many cases a custom transformer or set of transformers will need to be written to provide complete or more comprehensive coverage or to optimize other aspects of the output for a specific end-use. In this case we have provided the compose, execute, and composeAndExecute commands for running custom transformers from external repositories. This is described in detail here.

Tests

  • Replace the empty ipcPath in the environments/infura.toml with a path to a full node's eth_jsonrpc endpoint (e.g. local geth node ipc path or infura url)
    • Note: integration tests require configuration with an archival node
  • createdb vulcanize_private will create the test db
  • make migrate NAME=vulcanize_private will run the db migrations
  • make test will run the unit tests and skip the integration tests
  • make integrationtest will run just the integration tests

API

Postgraphile is used to expose GraphQL endpoints for our database schemas, this is described in detail here.

Contributing

Contributions are welcome! For more on this, please see here.

Small note: If editing the Readme, please conform to the standard-readme specification.

License

AGPL-3.0 © Vulcanize Inc