Server backend for indexed ETH IPLD objects
Go to file
Rob Mulholand d5c2ab33fc Sync only missing blocks on cold import
= Add eth_node_fingerprint to block that can be imitated by both hot and cold imports
- Only sync missing blocks (blocks that are missing or don't share a fingerprint) on cold import
- Set block is_final status after import
2018-05-07 16:38:04 -05:00
cmd Sync only missing blocks on cold import 2018-05-07 16:38:04 -05:00
db Sync only missing blocks on cold import 2018-05-07 16:38:04 -05:00
environments Add cold import script 2018-05-04 10:54:00 -05:00
integration_test Update block reading test to resolve periodic fails (#53) 2018-05-04 10:41:56 -05:00
libraries/shared Rename node table and node column 2018-03-21 13:24:13 -05:00
pkg Sync only missing blocks on cold import 2018-05-07 16:38:04 -05:00
scripts Get transactions (#45) 2018-03-27 16:06:12 -05:00
test_config Remove geth --dev dep 2018-03-21 13:57:18 -05:00
utils Handle events 2018-03-05 10:01:50 -06:00
vendor Add cold import script 2018-05-04 10:54:00 -05:00
.gitignore Handle events 2018-03-05 10:01:50 -06:00
.private_blockchain_password Add integration test 2017-10-24 15:36:50 -05:00
.travis.yml Get transactions (#45) 2018-03-27 16:06:12 -05:00
Gopkg.lock Add cold import script 2018-05-04 10:54:00 -05:00
Gopkg.toml Allow Parity as ingest node (#36) 2018-03-07 15:29:21 -06:00
LICENSE Add LICENSE 2017-11-09 12:58:17 -06:00
main.go Merge old private repo into vulcanize 2018-01-25 18:08:26 -06:00
Makefile Get transactions (#45) 2018-03-27 16:06:12 -05:00
README.md Sync only missing blocks on cold import 2018-05-07 16:38:04 -05:00

Vulcanize DB

Build Status

Dependencies

Installation

go get github.com/vulcanize/vulcanizedb

Setting up the Database

  1. Install Postgres

  2. Create a superuser for yourself and make sure psql --list works without prompting for a password.

  3. createdb vulcanize_public

  4. cd $GOPATH/src/github.com/vulcanize/vulcanizedb

  5. Run the migrations: make migrate HOST_NAME=localhost NAME=vulcanize_public PORT=5432

    • See below for configuring additional environments

Configuration

  • To use a local Ethereum node, copy environments/public.toml.example to environments/public.toml and update the ipcPath and levelDbPath.

    • ipcPath should match the local node's IPC filepath:

      • when using geth:

        • The IPC file is called geth.ipc.
        • The geth IPC file path is printed to the console when you start geth.
        • The default location is:
          • Mac: $HOME/Library/Ethereum
          • Linux: $HOME/.ethereum
      • when using parity:

        • The IPC file is called jsonrpc.ipc.
        • The default location is:
          • Mac: $HOME/Library/Application\ Support/io.parity.ethereum/
          • Linux: $HOME/.local/share/io.parity.ethereum/
    • levelDbPath should match Geth's chaindata directory path.

      • The geth LevelDB chaindata path is printed to the console when you start geth.
      • The default location is:
        • Mac: $HOME/Library/Ethereum/geth/chaindata
        • Linux: $HOME/.ethereum/geth/chaindata
      • levelDbPath is irrelevant (and coldImport is currently unavailable) if only running parity.
  • See environments/infura.toml to configure commands to run against infura, if a local node is unavailable

Start syncing with postgres

Syncs VulcanizeDB with the configured Ethereum node.

  1. Start node (if fast syncing wait for initial sync to finish)
  2. In a separate terminal start vulcanize_db
    • ./vulcanizedb sync --config <config.toml> --starting-block-number <block-number>

Alternatively, sync from Geth's underlying LevelDB

Sync VulcanizeDB from the LevelDB underlying a Geth node.

  1. Assure node is not running, and that it has synced to the desired block height.
  2. Start vulcanize_db
    • ./vulcanizedb coldImport --config <config.toml> --starting-block-number <block-number> --ending-block-number <block-number>
  3. Optional flags:
    • --starting-block-number/-s: block number to start syncing from
    • --ending-block-number/-e: block number to sync to
    • --all/-a: sync all missing blocks

Running the Tests

Unit Tests

  • go test ./pkg/...

Integration Tests

  • go test ./... to run all tests.