Tool for inserting the entire state and storage tries into PG-IPFS
Go to file
Abdul Rabbani a04777cb78 Updating Go and Geth Version
Ran Tests:

```
09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v
# github.com/vulcanize/ipld-eth-state-snapshot/cmd
cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v
?       github.com/vulcanize/ipld-eth-state-snapshot    [no test files]
=== RUN   TestCreateSnapshot
=== RUN   TestCreateSnapshot/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#03
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestCreateSnapshot (0.20s)
    --- PASS: TestCreateSnapshot/case (0.15s)
    --- PASS: TestCreateSnapshot/case#01 (0.01s)
    --- PASS: TestCreateSnapshot/case#02 (0.02s)
    --- PASS: TestCreateSnapshot/case#03 (0.03s)
=== RUN   TestRecovery
=== RUN   TestRecovery/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestRecovery/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestRecovery/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestRecovery (0.40s)
    --- PASS: TestRecovery/case (0.12s)
    --- PASS: TestRecovery/case#01 (0.19s)
    --- PASS: TestRecovery/case#02 (0.09s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot       1.109s
=== RUN   TestWriting
--- PASS: TestWriting (0.00s)
=== RUN   TestPgCopy
    publisher_test.go:84: set TEST_WITH_DB to enable test
--- SKIP: TestPgCopy (0.00s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file  0.279s
=== RUN   TestBasic
    publisher_test.go:47: set TEST_WITH_DB to enable test
--- SKIP: TestBasic (0.00s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg    0.647s
```
2022-06-01 09:09:19 -04:00
cmd Add prometheus metrics collection (#33) 2022-05-23 16:56:48 +05:30
db/migrations remove diff unique constraint 2020-08-06 14:33:02 -05:00
environments Add ethereum config to fix db import in file mode 2022-05-25 18:15:08 +05:30
fixture add some logs and guards, update module name, update readme 2022-03-30 18:57:30 -05:00
pkg Add prometheus metrics collection (#33) 2022-05-23 16:56:48 +05:30
test implement job recovery; fix traversal & iterator 2022-03-03 17:12:29 +08:00
.gitignore add some logs and guards, update module name, update readme 2022-03-30 18:57:30 -05:00
go.mod Updating Go and Geth Version 2022-06-01 09:09:19 -04:00
go.sum Updating Go and Geth Version 2022-06-01 09:09:19 -04:00
LICENSE cobra init 2020-06-30 11:53:18 -05:00
main.go add some logs and guards, update module name, update readme 2022-03-30 18:57:30 -05:00
Makefile Add prometheus metrics collection (#33) 2022-05-23 16:56:48 +05:30
README.md Add ethereum config to fix db import in file mode 2022-05-25 18:15:08 +05:30

ipld-eth-state-snapshot

Tool for extracting the entire Ethereum state at a particular block height from leveldb into Postgres-backed IPFS

Go Report Card

Usage

./ipld-eth-state-snapshot stateSnapshot --config={path to toml config file}

Config

Config format:

[snapshot]
    mode = "file" # indicates output mode ("postgres" or "file")
    workers = 4 # degree of concurrency, the state trie is subdivided into sectiosn that are traversed and processed concurrently
    blockHeight = -1 # blockheight to perform the snapshot at (-1 indicates to use the latest blockheight found in leveldb)
    recoveryFile = "recovery_file" # specifies a file to output recovery information on error or premature closure

[leveldb]
    path = "/Users/user/Library/Ethereum/geth/chaindata" # path to geth leveldb
    ancient = "/Users/user/Library/Ethereum/geth/chaindata/ancient" # path to geth ancient database

[database]
    name     = "vulcanize_public" # postgres database name
    hostname = "localhost" # postgres host
    port     = 5432 # postgres port
    user     = "postgres" # postgres user
    password = "" # postgres password

[file]
    outputDir = "output_dir/" # when operating in 'file' output mode, this is the directory the files are written to

[log]
    level = "info" # log level (trace, debug, info, warn, error, fatal, panic) (default: info)
    file = "log_file" # file path for logging

[prom]
    metrics = true # enable prometheus metrics (default: false)
    http = true # enable prometheus http service (default: false)
    httpAddr = "0.0.0.0" # prometheus http host (default: 127.0.0.1)
    httpPort = 9101 # prometheus http port (default: 8086)
    dbStats = true # enable prometheus db stats (default: false)

# node info
[ethereum]
    clientName = "Geth" # $ETH_CLIENT_NAME
    nodeID = "arch1" # $ETH_NODE_ID
    networkID = "1" # $ETH_NETWORK_ID
    chainID = "1" # $ETH_CHAIN_ID
    genesisBlock = "0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3" # $ETH_GENESIS_BLOCK