cerc-io/ipld-eth-state-snapshot
16
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity
be544a3424
ipld-eth-state-snapshot/go.mod

111 lines
4.9 KiB
Modula-2
Raw Normal View History

add some logs and guards, update module name, update readme
2022-03-30 23:57:30 +00:00
module github.com/vulcanize/ipld-eth-state-snapshot
go modules
2020-07-01 19:10:25 +00:00
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
go 1.18
go modules
2020-07-01 19:10:25 +00:00
require (
Upgrade dependencies (#56)
2022-08-03 10:41:38 +00:00
github.com/ethereum/go-ethereum v1.10.21
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/golang/mock v1.6.0
Upgrade dependencies (#56)
2022-08-03 10:41:38 +00:00
github.com/ipfs/go-cid v0.2.0
github.com/ipfs/go-ipfs-blockstore v1.2.0
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/ipfs/go-ipfs-ds-help v1.1.0
Account selective snapshot (#46) * snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided * config and env updates * cmd update * Encode watched address path bytes to hex for comparison * actually ignore the subtries that are not along the paths of interest * Fixes for account selective snapshot * Use non-concurrent iterator when having a single worker * Only index root node when starting path of an iterator is nil * Upgrade deps * Avoid tracking iterators and skip recovery test * Fix recovery mechanism, use sync Map instead of buffered channels * Add test for account selective snapshot * Continue traversal with concurrent iterators with starting path nil * Use errgroup to simplify error handling with concurrent iterators * Check if all the nodes are indexed in the recovery test * Use concurrency safe sync Map in account selective snapshot test * Only track concurrent iterators and refactor code * Fix node and recovered path comparison * Revert back to using buffered channels for tracking iterators * Add a metric to monitor number of active iterators * Update docs * Update seeked path after node is processed * Return error on context cancellation from subtrie iteration * Add tests for account selective snapshot recovery * Explicity enforce concurrent iterator bounds to avoid duplicate nodes * Update full snapshot test to check nodes being indexed * Refactor code to simplify snapshot logic * Remove unnecessary function argument * Use ctx cancellation for handling signals * Add descriptive comments Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 11:35:04 +00:00
github.com/jmoiron/sqlx v1.3.5
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/multiformats/go-multihash v0.1.0
github.com/prometheus/client_golang v1.3.0
Account selective snapshot (#46) * snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided * config and env updates * cmd update * Encode watched address path bytes to hex for comparison * actually ignore the subtries that are not along the paths of interest * Fixes for account selective snapshot * Use non-concurrent iterator when having a single worker * Only index root node when starting path of an iterator is nil * Upgrade deps * Avoid tracking iterators and skip recovery test * Fix recovery mechanism, use sync Map instead of buffered channels * Add test for account selective snapshot * Continue traversal with concurrent iterators with starting path nil * Use errgroup to simplify error handling with concurrent iterators * Check if all the nodes are indexed in the recovery test * Use concurrency safe sync Map in account selective snapshot test * Only track concurrent iterators and refactor code * Fix node and recovered path comparison * Revert back to using buffered channels for tracking iterators * Add a metric to monitor number of active iterators * Update docs * Update seeked path after node is processed * Return error on context cancellation from subtrie iteration * Add tests for account selective snapshot recovery * Explicity enforce concurrent iterator bounds to avoid duplicate nodes * Update full snapshot test to check nodes being indexed * Refactor code to simplify snapshot logic * Remove unnecessary function argument * Use ctx cancellation for handling signals * Add descriptive comments Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 11:35:04 +00:00
github.com/sirupsen/logrus v1.9.0
github.com/spf13/cobra v1.5.0
github.com/spf13/viper v1.12.0
github.com/vulcanize/go-eth-state-node-iterator v1.1.4
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
)
require (
github.com/VictoriaMetrics/fastcache v1.6.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
Add prometheus metrics collection (#33) * Upgrade geth * Add prometheus metrics collection * Update README
2022-05-23 11:26:48 +00:00
github.com/btcsuite/btcd/btcec/v2 v2.2.0 // indirect
Update geth and implement transaction batching.
2021-12-13 15:01:32 +00:00
github.com/cespare/xxhash/v2 v2.1.2 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1 // indirect
Upgrade dependencies (#56)
2022-08-03 10:41:38 +00:00
github.com/fsnotify/fsnotify v1.5.4 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/georgysavva/scany v0.2.9 // indirect
Update geth and implement transaction batching.
2021-12-13 15:01:32 +00:00
github.com/go-kit/kit v0.10.0 // indirect
Upgrade dependencies (#56)
2022-08-03 10:41:38 +00:00
github.com/go-ole/go-ole v1.2.6 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/go-stack/stack v1.8.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/golang/snappy v0.0.4 // indirect
Update geth and implement transaction batching.
2021-12-13 15:01:32 +00:00
github.com/google/uuid v1.3.0 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/holiman/bloomfilter/v2 v2.0.3 // indirect
github.com/inconshreveable/mousetrap v1.0.0 // indirect
github.com/ipfs/bbloom v0.0.4 // indirect
github.com/ipfs/go-block-format v0.0.3 // indirect
Update geth and implement transaction batching.
2021-12-13 15:01:32 +00:00
github.com/ipfs/go-datastore v0.5.1 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/ipfs/go-ipfs-util v0.0.2 // indirect
Upgrade dependencies (#56)
2022-08-03 10:41:38 +00:00
github.com/ipfs/go-ipld-format v0.4.0 // indirect
Update geth and implement transaction batching.
2021-12-13 15:01:32 +00:00
github.com/ipfs/go-log v1.0.5 // indirect
github.com/ipfs/go-log/v2 v2.4.0 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/ipfs/go-metrics-interface v0.0.1 // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
github.com/jackc/pgconn v1.11.0 // indirect
github.com/jackc/pgio v1.0.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgproto3/v2 v2.2.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b // indirect
github.com/jackc/pgtype v1.10.0 // indirect
Upgrade geth (#51) * Fix Makefile for installing mockgen tool * Upgrade geth * Add sleep in the failing recovery test Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-07-13 06:09:36 +00:00
github.com/jackc/pgx/v4 v4.15.0 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/jackc/puddle v1.2.1 // indirect
github.com/jbenet/goprocess v0.1.4 // indirect
github.com/klauspost/cpuid/v2 v2.0.9 // indirect
github.com/lib/pq v1.10.2 // indirect
Account selective snapshot (#46) * snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided * config and env updates * cmd update * Encode watched address path bytes to hex for comparison * actually ignore the subtries that are not along the paths of interest * Fixes for account selective snapshot * Use non-concurrent iterator when having a single worker * Only index root node when starting path of an iterator is nil * Upgrade deps * Avoid tracking iterators and skip recovery test * Fix recovery mechanism, use sync Map instead of buffered channels * Add test for account selective snapshot * Continue traversal with concurrent iterators with starting path nil * Use errgroup to simplify error handling with concurrent iterators * Check if all the nodes are indexed in the recovery test * Use concurrency safe sync Map in account selective snapshot test * Only track concurrent iterators and refactor code * Fix node and recovered path comparison * Revert back to using buffered channels for tracking iterators * Add a metric to monitor number of active iterators * Update docs * Update seeked path after node is processed * Return error on context cancellation from subtrie iteration * Add tests for account selective snapshot recovery * Explicity enforce concurrent iterator bounds to avoid duplicate nodes * Update full snapshot test to check nodes being indexed * Refactor code to simplify snapshot logic * Remove unnecessary function argument * Use ctx cancellation for handling signals * Add descriptive comments Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 11:35:04 +00:00
github.com/magiconair/properties v1.8.6 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/mattn/go-isatty v0.0.14 // indirect
github.com/mattn/go-runewidth v0.0.9 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 // indirect
github.com/minio/sha256-simd v1.0.0 // indirect
Account selective snapshot (#46) * snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided * config and env updates * cmd update * Encode watched address path bytes to hex for comparison * actually ignore the subtries that are not along the paths of interest * Fixes for account selective snapshot * Use non-concurrent iterator when having a single worker * Only index root node when starting path of an iterator is nil * Upgrade deps * Avoid tracking iterators and skip recovery test * Fix recovery mechanism, use sync Map instead of buffered channels * Add test for account selective snapshot * Continue traversal with concurrent iterators with starting path nil * Use errgroup to simplify error handling with concurrent iterators * Check if all the nodes are indexed in the recovery test * Use concurrency safe sync Map in account selective snapshot test * Only track concurrent iterators and refactor code * Fix node and recovered path comparison * Revert back to using buffered channels for tracking iterators * Add a metric to monitor number of active iterators * Update docs * Update seeked path after node is processed * Return error on context cancellation from subtrie iteration * Add tests for account selective snapshot recovery * Explicity enforce concurrent iterator bounds to avoid duplicate nodes * Update full snapshot test to check nodes being indexed * Refactor code to simplify snapshot logic * Remove unnecessary function argument * Use ctx cancellation for handling signals * Add descriptive comments Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 11:35:04 +00:00
github.com/mitchellh/mapstructure v1.5.0 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/mr-tron/base58 v1.2.0 // indirect
Update geth and implement transaction batching.
2021-12-13 15:01:32 +00:00
github.com/multiformats/go-base32 v0.0.4 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/multiformats/go-base36 v0.1.0 // indirect
github.com/multiformats/go-multibase v0.0.3 // indirect
github.com/multiformats/go-varint v0.0.6 // indirect
github.com/olekukonko/tablewriter v0.0.5 // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
Account selective snapshot (#46) * snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided * config and env updates * cmd update * Encode watched address path bytes to hex for comparison * actually ignore the subtries that are not along the paths of interest * Fixes for account selective snapshot * Use non-concurrent iterator when having a single worker * Only index root node when starting path of an iterator is nil * Upgrade deps * Avoid tracking iterators and skip recovery test * Fix recovery mechanism, use sync Map instead of buffered channels * Add test for account selective snapshot * Continue traversal with concurrent iterators with starting path nil * Use errgroup to simplify error handling with concurrent iterators * Check if all the nodes are indexed in the recovery test * Use concurrency safe sync Map in account selective snapshot test * Only track concurrent iterators and refactor code * Fix node and recovered path comparison * Revert back to using buffered channels for tracking iterators * Add a metric to monitor number of active iterators * Update docs * Update seeked path after node is processed * Return error on context cancellation from subtrie iteration * Add tests for account selective snapshot recovery * Explicity enforce concurrent iterator bounds to avoid duplicate nodes * Update full snapshot test to check nodes being indexed * Refactor code to simplify snapshot logic * Remove unnecessary function argument * Use ctx cancellation for handling signals * Add descriptive comments Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 11:35:04 +00:00
github.com/pelletier/go-toml v1.9.5 // indirect
github.com/pelletier/go-toml/v2 v2.0.1 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_model v0.1.0 // indirect
github.com/prometheus/common v0.7.0 // indirect
github.com/prometheus/procfs v0.0.8 // indirect
Upgrade dependencies (#56)
2022-08-03 10:41:38 +00:00
github.com/prometheus/tsdb v0.10.0 // indirect
github.com/shirou/gopsutil v3.21.11+incompatible // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/spaolacci/murmur3 v1.1.0 // indirect
Account selective snapshot (#46) * snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided * config and env updates * cmd update * Encode watched address path bytes to hex for comparison * actually ignore the subtries that are not along the paths of interest * Fixes for account selective snapshot * Use non-concurrent iterator when having a single worker * Only index root node when starting path of an iterator is nil * Upgrade deps * Avoid tracking iterators and skip recovery test * Fix recovery mechanism, use sync Map instead of buffered channels * Add test for account selective snapshot * Continue traversal with concurrent iterators with starting path nil * Use errgroup to simplify error handling with concurrent iterators * Check if all the nodes are indexed in the recovery test * Use concurrency safe sync Map in account selective snapshot test * Only track concurrent iterators and refactor code * Fix node and recovered path comparison * Revert back to using buffered channels for tracking iterators * Add a metric to monitor number of active iterators * Update docs * Update seeked path after node is processed * Return error on context cancellation from subtrie iteration * Add tests for account selective snapshot recovery * Explicity enforce concurrent iterator bounds to avoid duplicate nodes * Update full snapshot test to check nodes being indexed * Refactor code to simplify snapshot logic * Remove unnecessary function argument * Use ctx cancellation for handling signals * Add descriptive comments Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 11:35:04 +00:00
github.com/spf13/afero v1.8.2 // indirect
github.com/spf13/cast v1.5.0 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/stretchr/objx v0.2.0 // indirect
update for go-ethereum 1.10.20
2022-07-19 19:12:44 +00:00
github.com/stretchr/testify v1.7.2 // indirect
Account selective snapshot (#46) * snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided * config and env updates * cmd update * Encode watched address path bytes to hex for comparison * actually ignore the subtries that are not along the paths of interest * Fixes for account selective snapshot * Use non-concurrent iterator when having a single worker * Only index root node when starting path of an iterator is nil * Upgrade deps * Avoid tracking iterators and skip recovery test * Fix recovery mechanism, use sync Map instead of buffered channels * Add test for account selective snapshot * Continue traversal with concurrent iterators with starting path nil * Use errgroup to simplify error handling with concurrent iterators * Check if all the nodes are indexed in the recovery test * Use concurrency safe sync Map in account selective snapshot test * Only track concurrent iterators and refactor code * Fix node and recovered path comparison * Revert back to using buffered channels for tracking iterators * Add a metric to monitor number of active iterators * Update docs * Update seeked path after node is processed * Return error on context cancellation from subtrie iteration * Add tests for account selective snapshot recovery * Explicity enforce concurrent iterator bounds to avoid duplicate nodes * Update full snapshot test to check nodes being indexed * Refactor code to simplify snapshot logic * Remove unnecessary function argument * Use ctx cancellation for handling signals * Add descriptive comments Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 11:35:04 +00:00
github.com/subosito/gotenv v1.3.0 // indirect
Upgrade dependencies (#56)
2022-08-03 10:41:38 +00:00
github.com/syndtr/goleveldb v1.0.1-0.20220614013038-64ee5596c38a // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
github.com/tklauser/go-sysconf v0.3.5 // indirect
github.com/tklauser/numcpus v0.2.2 // indirect
Upgrade dependencies (#56)
2022-08-03 10:41:38 +00:00
github.com/yusufpapurcu/wmi v1.2.2 // indirect
Update geth and implement transaction batching.
2021-12-13 15:01:32 +00:00
go.uber.org/atomic v1.9.0 // indirect
go.uber.org/goleak v1.1.11 // indirect
go.uber.org/multierr v1.7.0 // indirect
go.uber.org/zap v1.19.1 // indirect
Account selective snapshot (#46) * snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided * config and env updates * cmd update * Encode watched address path bytes to hex for comparison * actually ignore the subtries that are not along the paths of interest * Fixes for account selective snapshot * Use non-concurrent iterator when having a single worker * Only index root node when starting path of an iterator is nil * Upgrade deps * Avoid tracking iterators and skip recovery test * Fix recovery mechanism, use sync Map instead of buffered channels * Add test for account selective snapshot * Continue traversal with concurrent iterators with starting path nil * Use errgroup to simplify error handling with concurrent iterators * Check if all the nodes are indexed in the recovery test * Use concurrency safe sync Map in account selective snapshot test * Only track concurrent iterators and refactor code * Fix node and recovered path comparison * Revert back to using buffered channels for tracking iterators * Add a metric to monitor number of active iterators * Update docs * Update seeked path after node is processed * Return error on context cancellation from subtrie iteration * Add tests for account selective snapshot recovery * Explicity enforce concurrent iterator bounds to avoid duplicate nodes * Update full snapshot test to check nodes being indexed * Refactor code to simplify snapshot logic * Remove unnecessary function argument * Use ctx cancellation for handling signals * Add descriptive comments Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 11:35:04 +00:00
golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4 // indirect
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
golang.org/x/text v0.3.7 // indirect
Account selective snapshot (#46) * snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided * config and env updates * cmd update * Encode watched address path bytes to hex for comparison * actually ignore the subtries that are not along the paths of interest * Fixes for account selective snapshot * Use non-concurrent iterator when having a single worker * Only index root node when starting path of an iterator is nil * Upgrade deps * Avoid tracking iterators and skip recovery test * Fix recovery mechanism, use sync Map instead of buffered channels * Add test for account selective snapshot * Continue traversal with concurrent iterators with starting path nil * Use errgroup to simplify error handling with concurrent iterators * Check if all the nodes are indexed in the recovery test * Use concurrency safe sync Map in account selective snapshot test * Only track concurrent iterators and refactor code * Fix node and recovered path comparison * Revert back to using buffered channels for tracking iterators * Add a metric to monitor number of active iterators * Update docs * Update seeked path after node is processed * Return error on context cancellation from subtrie iteration * Add tests for account selective snapshot recovery * Explicity enforce concurrent iterator bounds to avoid duplicate nodes * Update full snapshot test to check nodes being indexed * Refactor code to simplify snapshot logic * Remove unnecessary function argument * Use ctx cancellation for handling signals * Add descriptive comments Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 11:35:04 +00:00
google.golang.org/protobuf v1.28.0 // indirect
gopkg.in/ini.v1 v1.66.4 // indirect
Updating Go and Geth Version Ran Tests: ``` 09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v # github.com/vulcanize/ipld-eth-state-snapshot/cmd cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v ? github.com/vulcanize/ipld-eth-state-snapshot [no test files] === RUN TestCreateSnapshot === RUN TestCreateSnapshot/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestCreateSnapshot/case#03 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestCreateSnapshot (0.20s) --- PASS: TestCreateSnapshot/case (0.15s) --- PASS: TestCreateSnapshot/case#01 (0.01s) --- PASS: TestCreateSnapshot/case#02 (0.02s) --- PASS: TestCreateSnapshot/case#03 (0.03s) === RUN TestRecovery === RUN TestRecovery/case time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#01 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" === RUN TestRecovery/case#02 time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv" time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1" time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1" --- PASS: TestRecovery (0.40s) --- PASS: TestRecovery/case (0.12s) --- PASS: TestRecovery/case#01 (0.19s) --- PASS: TestRecovery/case#02 (0.09s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s === RUN TestWriting --- PASS: TestWriting (0.00s) === RUN TestPgCopy publisher_test.go:84: set TEST_WITH_DB to enable test --- SKIP: TestPgCopy (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s === RUN TestBasic publisher_test.go:47: set TEST_WITH_DB to enable test --- SKIP: TestBasic (0.00s) PASS ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s ```
2022-06-01 13:09:19 +00:00
gopkg.in/yaml.v2 v2.4.0 // indirect
update for go-ethereum 1.10.20
2022-07-19 19:12:44 +00:00
gopkg.in/yaml.v3 v3.0.1 // indirect
Update geth and implement transaction batching.
2021-12-13 15:01:32 +00:00
lukechampine.com/blake3 v1.1.7 // indirect
go modules
2020-07-01 19:10:25 +00:00
)
Upgrade dependencies (#56)
2022-08-03 10:41:38 +00:00
replace github.com/ethereum/go-ethereum v1.10.21 => github.com/vulcanize/go-ethereum v1.10.21-statediff-4.1.2-alpha
Reference in New Issue Copy Permalink