Add helper scripts for data dump correction (#57 )

* Add a script to find bad data in CSV file dumps

* Add a script to delete bad rows from CSV file dumps

* Add instructions to run the scripts

* Reorganize instructions

2022-08-17 15:14:14 +05:30

7.5 KiB

Raw Blame History

ipld-eth-state-snapshot

Tool for extracting the entire Ethereum state at a particular block height from leveldb into Postgres-backed IPFS

Setup

Build the binary:
```
make build
```

Configuration

Config format:

[snapshot]
    mode         = "file"           # indicates output mode <postgres | file>
    workers      = 4                # degree of concurrency, the state trie is subdivided into sections that are traversed and processed concurrently
    blockHeight  = -1               # blockheight to perform the snapshot at (-1 indicates to use the latest blockheight found in leveldb)
    recoveryFile = "recovery_file"  # specifies a file to output recovery information on error or premature closure
    accounts = []                   # list of accounts (addresses) to take the snapshot for # SNAPSHOT_ACCOUNTS

[leveldb]
    # path to geth leveldb
    path    = "/Users/user/Library/Ethereum/geth/chaindata"         # LVL_DB_PATH
    # path to geth ancient database
    ancient = "/Users/user/Library/Ethereum/geth/chaindata/ancient" # ANCIENT_DB_PATH

[database]
    # when operating in 'postgres' output mode
    # db credentials
    name     = "vulcanize_public"   # DATABASE_NAME
    hostname = "localhost"          # DATABASE_HOSTNAME
    port     = 5432                 # DATABASE_PORT
    user     = "postgres"           # DATABASE_USER
    password = ""                   # DATABASE_PASSWORD

[file]
    # when operating in 'file' output mode
    # directory the CSV files are written to
    outputDir = "output_dir/"   # FILE_OUTPUT_DIR

[log]
    level = "info"      # log level (trace, debug, info, warn, error, fatal, panic) (default: info)
    file  = "log_file"  # file path for logging, leave unset to log to stdout

[prom]
    # prometheus metrics
    metrics  = true         # enable prometheus metrics         (default: false)
    http     = true         # enable prometheus http service    (default: false)
    httpAddr = "0.0.0.0"    # prometheus http host              (default: 127.0.0.1)
    httpPort = 9101         # prometheus http port              (default: 8086)
    dbStats  = true         # enable prometheus db stats        (default: false)

[ethereum]
    # node info
    clientName   = "Geth"   # ETH_CLIENT_NAME
    nodeID       = "arch1"  # ETH_NODE_ID
    networkID    = "1"      # ETH_NETWORK_ID
    chainID      = "1"      # ETH_CHAIN_ID
    genesisBlock = "0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3" # ETH_GENESIS_BLOCK

Usage

For state snapshot from LevelDB:
```
./ipld-eth-state-snapshot stateSnapshot --config={path to toml config file}
```
- Account selective snapshot: To restrict the snapshot to a list of accounts (addresses), provide the addresses in config parameter snapshot.accounts or env variable SNAPSHOT_ACCOUNTS. Only nodes related to provided addresses will be indexed.
  
  Example:
```
[snapshot]
    accounts = [
        "0x825a6eec09e44Cb0fa19b84353ad0f7858d7F61a"
    ]
```

For in-place snapshot in the database:

./ipld-eth-state-snapshot inPlaceStateSnapshot --config={path to toml config file}

Monitoring

Enable metrics using config parameters prom.metrics and prom.http.
ipld-eth-state-snapshot exposes following prometheus metrics at /metrics endpoint:
- state_node_count: Number of state nodes processed.
- storage_node_count: Number of storage nodes processed.
- code_node_count: Number of code nodes processed.
- DB stats if operating in postgres mode.

Tests

Run unit tests:

# setup db
docker-compose up -d

# run tests after db migrations are run
make dbtest

# tear down db
docker-compose down -v --remove-orphans

Import output data in file mode into a database

When ipld-eth-state-snapshot stateSnapshot is run in file mode (database.type), the output is in form of CSV files.
Assuming the output files are located in host's ./output_dir directory.

Data post-processing:

Create a directory to store post-processed output:
```
mkdir -p output_dir/processed_output
```

Combine output from multiple workers and copy to post-processed output directory:

# public.blocks
cat {output_dir,output_dir/*}/public.blocks.csv > output_dir/processed_output/combined-public.blocks.csv

# eth.state_cids
cat output_dir/*/eth.state_cids.csv > output_dir/processed_output/combined-eth.state_cids.csv

# eth.storage_cids
cat output_dir/*/eth.storage_cids.csv > output_dir/processed_output/combined-eth.storage_cids.csv

# public.nodes
cp output_dir/public.nodes.csv output_dir/processed_output/public.nodes.csv

# eth.header_cids
cp output_dir/eth.header_cids.csv output_dir/processed_output/eth.header_cids.csv

De-duplicate data:

# public.blocks
sort -u output_dir/processed_output/combined-public.blocks.csv -o output_dir/processed_output/deduped-combined-public.blocks.csv

# eth.header_cids
sort -u output_dir/processed_output/eth.header_cids.csv -o output_dir/processed_output/deduped-eth.header_cids.csv

# eth.state_cids
sort -u output_dir/processed_output/combined-eth.state_cids.csv -o output_dir/processed_output/deduped-combined-eth.state_cids.csv

# eth.storage_cids
sort -u output_dir/processed_output/combined-eth.storage_cids.csv -o output_dir/processed_output/deduped-combined-eth.storage_cids.csv

Copy over the post-processed output files to the DB server (say in /output_dir).

Start psql to run the import commands:

psql -U <DATABASE_USER> -h <DATABASE_HOSTNAME> -p <DATABASE_PORT> <DATABASE_NAME>

Run the following to import data:

# public.nodes
COPY public.nodes FROM '/output_dir/processed_output/public.nodes.csv' CSV;

# public.blocks
COPY public.blocks FROM '/output_dir/processed_output/deduped-combined-public.blocks.csv' CSV;

# eth.header_cids
COPY eth.header_cids FROM '/output_dir/processed_output/deduped-eth.header_cids.csv' CSV;

# eth.state_cids
COPY eth.state_cids FROM '/output_dir/processed_output/deduped-combined-eth.state_cids.csv' CSV FORCE NOT NULL state_leaf_key;

# eth.storage_cids
COPY eth.storage_cids FROM '/output_dir/processed_output/deduped-combined-eth.storage_cids.csv' CSV FORCE NOT NULL storage_leaf_key;

NOTE: COPY command on CSVs inserts empty strings as NULL in the DB. Passing FORCE_NOT_NULL <COLUMN_NAME> forces it to insert empty strings instead. This is required to maintain compatibility of the imported snapshot data with the data generated by statediffing. Reference: https://www.postgresql.org/docs/14/sql-copy.html

Troubleshooting

Run the following command to find any rows (in data dumps in file mode) having unexpected number of columns:
```
./scripts/find-bad-rows.sh -i <input-file> -c <expected-columns> -o [output-file] -d true
```
Run the following command to select rows (from data dumps in file mode) other than the ones having unexpected number of columns:
```
./scripts/filter-bad-rows.sh -i <input-file> -c <expected-columns> -o <output-file>
```
See scripts for more details.

7.5 KiB Raw Blame History