cerc-io/ipld-eth-state-snapshot
16
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity
135 Commits 10 Branches 27 Tags 7.2 MiB
14b1180161
Commit Graph

135 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
i-norden
14b1180161 update Block1 and Chain2_Block32 test fixtures 2023-04-12 13:13:04 -05:00
i-norden
ead007f159 update csv file publisher 2023-04-12 13:13:04 -05:00
i-norden
f1a980f37c update direct pg publisher 2023-04-12 13:13:04 -05:00
i-norden
f83ab82424 use types/models exported from vdb geth 2023-04-12 13:13:04 -05:00
Ian Norden
fb5a95d874
Merge pull request #72 from cerc-io/ian/v5_rebase 
Rebase
 2023-04-12 13:12:35 -05:00
Roy Crihfield
97ee99a449 Update schema (#61) 
Updates table to add header_cids.uncles_hash
 2023-04-12 13:09:58 -05:00
Ian Norden
b6d7695536
Merge pull request #70 from cerc-io/ian/v4_dev 
update db version
 2023-04-11 09:06:03 -05:00
i-norden
3f3e77cbac update db version 2023-04-11 09:01:50 -05:00
Ian Norden
e63ffbf0ad
Merge pull request #69 from cerc-io/ian/v4_dev 
Update to use v4 vdb geth v1.11.5
 2023-03-31 12:50:23 -05:00
i-norden
6ede522ae0 missing .meta files in fixtures 2023-03-31 12:36:30 -05:00
i-norden
5c13d59515 refactor to work with v4 vdb geth v1.11.5 2023-03-31 10:39:27 -05:00
i-norden
6fe54aa8b4 use go1.19 2023-03-31 10:39:13 -05:00
i-norden
c9dd85488b bump vdb v4 geth to v1.11.5 2023-03-31 10:39:00 -05:00
Michael
ac85fe29eb
dependency updates for geth 1.10.26 (#67) 2022-11-08 14:11:11 -05:00
Roy Crihfield
c270f39de9
Update schema (#61) 
Updates table to add header_cids.uncles_hash
 2022-10-13 13:19:47 -05:00
Michael
9ea78c4ecf
dependency updates for geth 1.10.25 (#65) 2022-09-26 09:14:15 -04:00
Michael
989bd1c0f1
Cerc refactor (#64) 
* cerc refactor waiting on unpublished dependencies

* more cerc_refactor with published ipld-eth-db unstable

* TearDownDB refactor

* missed second publisher_test TearDownDB
 2022-09-20 13:47:34 -04:00
Michael
3f93a989dc
Merge pull request #63 from vulcanize/rebase-1.10.23-wip 
updates for geth rebase on 1.10.23
 2022-09-02 15:39:23 -04:00
Michael Shaw
321e37584d updates for geth rebase on 1.10.23 2022-09-02 15:21:29 -04:00
prathamesh0
be544a3424
Add helper scripts for data dump correction (#57) 
* Add a script to find bad data in CSV file dumps

* Add a script to delete bad rows from CSV file dumps

* Add instructions to run the scripts

* Reorganize instructions
 2022-08-17 15:14:14 +05:30
Ian Norden
05aeeab581
Account selective snapshot (#46) 
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided

* config and env updates

* cmd update

* Encode watched address path bytes to hex for comparison

* actually ignore the subtries that are not along the paths of interest

* Fixes for account selective snapshot

* Use non-concurrent iterator when having a single worker

* Only index root node when starting path of an iterator is nil

* Upgrade deps

* Avoid tracking iterators and skip recovery test

* Fix recovery mechanism, use sync Map instead of buffered channels

* Add test for account selective snapshot

* Continue traversal with concurrent iterators with starting path nil

* Use errgroup to simplify error handling with concurrent iterators

* Check if all the nodes are indexed in the recovery test

* Use concurrency safe sync Map in account selective snapshot test

* Only track concurrent iterators and refactor code

* Fix node and recovered path comparison

* Revert back to using buffered channels for tracking iterators

* Add a metric to monitor number of active iterators

* Update docs

* Update seeked path after node is processed

* Return error on context cancellation from subtrie iteration

* Add tests for account selective snapshot recovery

* Explicity enforce concurrent iterator bounds to avoid duplicate nodes

* Update full snapshot test to check nodes being indexed

* Refactor code to simplify snapshot logic

* Remove unnecessary function argument

* Use ctx cancellation for handling signals

* Add descriptive comments

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
 2022-08-03 17:05:04 +05:30
prathamesh0
5002c82038
Upgrade dependencies (#56) 2022-08-03 16:11:38 +05:30
Michael
289fb63568
Merge pull request #54 from vulcanize/release-v4.1.1-alpha 
update for go-ethereum 1.10.20
 2022-07-20 17:18:58 -04:00
Michael Shaw
257fac1a0c update for go-ethereum 1.10.20 2022-07-19 15:12:44 -04:00
prathamesh0
c0a7fdf9e8
Fix typo in config format in README (#53) 2022-07-18 16:48:58 +05:30
prathamesh0
2af26ad583
Add instructions to import snapshot data into database (#52) 
* Add instructions to import snapshot data into database

* Add monitoring and update data processing in README

* Update instructions to import snapshot
 2022-07-18 15:30:23 +05:30
prathamesh0
ac00ee280f
Upgrade geth (#51) 
* Fix Makefile for installing mockgen tool

* Upgrade geth

* Add sleep in the failing recovery test

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
 2022-07-13 11:39:36 +05:30
prathamesh0
b241bf05eb
Add command to take an in-place snapshot (#42) 
* Add command to take an in-place snapshot

* Add test data for in place snapshot unit test

* Implement unit test for inplace snapshot

* Add check for storage IPLD

* Run unit tests sequentially

* Add github workflow for unit test

* Add missing checks for state and storage cid fields

* Add more storage nodes to test

* Update ipld-eth-db version for tests

* Add comments for inplace snapshot test data

* Add in-place snapshot cmd in readme

* Implement defer pattern for db transaction

* Log transaction commit or rollback error

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
 2022-06-15 12:51:26 +05:30
Ashwin Phatak
c2ecea6d61
Merge pull request #45 from vulcanize/ng-merge-fix-iterator 
Merge changes to fix iterator and update
 2022-06-10 14:24:33 +05:30
nabarun
baef795834 Merge changes to fix iterator and update 2022-06-10 14:12:34 +05:30
nikugogoi
4979e4090d
Merge pull request #43 from roysc/fix-iterator-and-update 
Fix iterator and update
 2022-06-10 14:03:43 +05:30
nabarun
59011cba3c Update go-eth-state-node-iterator version 2022-06-10 14:02:13 +05:30
Roy Crihfield
8371344756 clean up logging 2022-06-08 20:12:22 +08:00
Roy Crihfield
ef391d8d31 simplify tracker, prevent send to closed chan 2022-06-08 20:12:22 +08:00
Roy Crihfield
838dabcba6 update iterator mod 2022-06-08 20:12:22 +08:00
Roy Crihfield
8572c36ee7 decrement path from restored iterator 
to cover node gaps
 2022-06-08 20:12:22 +08:00
Ashwin Phatak
74f82a641d Fix dependency 2022-06-08 20:05:38 +08:00
Roy Crihfield
3e24972a89 try to fix hanging iterator 2022-06-08 20:05:38 +08:00
Ashwin Phatak
7129044eae Log progress info 2022-06-08 18:32:10 +08:00
Abdul Rabbani
c9d398b5af
Merge pull request #40 from vulcanize/feature/update-go-geth-sharding 
Update go and geth versions
 2022-06-01 09:22:04 -04:00
Abdul Rabbani
8aab6a5137 Update geth versions 2022-06-01 09:20:45 -04:00
Abdul Rabbani
cd2559e434 Merge branch 'master' into feature/update-go-geth-sharding 2022-06-01 09:19:18 -04:00
Abdul Rabbani
d213644ff4
Merge pull request #39 from vulcanize/feature/update-go-geth 
Updating Go and Geth Version
 2022-06-01 09:10:35 -04:00
Abdul Rabbani
a04777cb78 Updating Go and Geth Version 
Ran Tests:

```
09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v
# github.com/vulcanize/ipld-eth-state-snapshot/cmd
cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v
?       github.com/vulcanize/ipld-eth-state-snapshot    [no test files]
=== RUN   TestCreateSnapshot
=== RUN   TestCreateSnapshot/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#03
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestCreateSnapshot (0.20s)
    --- PASS: TestCreateSnapshot/case (0.15s)
    --- PASS: TestCreateSnapshot/case#01 (0.01s)
    --- PASS: TestCreateSnapshot/case#02 (0.02s)
    --- PASS: TestCreateSnapshot/case#03 (0.03s)
=== RUN   TestRecovery
=== RUN   TestRecovery/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestRecovery/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestRecovery/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestRecovery (0.40s)
    --- PASS: TestRecovery/case (0.12s)
    --- PASS: TestRecovery/case#01 (0.19s)
    --- PASS: TestRecovery/case#02 (0.09s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot       1.109s
=== RUN   TestWriting
--- PASS: TestWriting (0.00s)
=== RUN   TestPgCopy
    publisher_test.go:84: set TEST_WITH_DB to enable test
--- SKIP: TestPgCopy (0.00s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file  0.279s
=== RUN   TestBasic
    publisher_test.go:47: set TEST_WITH_DB to enable test
--- SKIP: TestBasic (0.00s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg    0.647s
```
 2022-06-01 09:09:19 -04:00
Ashwin Phatak
c785c12274
Merge pull request #35 from vulcanize/ng-file-ethereum-nodeId 
Add ethereum config to fix db import in file mode
 2022-05-25 18:38:09 +05:30
nabarun
23c4d22885 Add ethereum config to fix db import in file mode 2022-05-25 18:15:08 +05:30
Ashwin Phatak
d3a149899b
Merge pull request #34 from deep-stack/pm-merge-metrics 
Merge changes to add env bindings and prometheus metrics
 2022-05-23 18:14:06 +05:30
prathamesh0
dc27487b66 Merge changes to add env bindings and prometheus metrics 2022-05-23 17:11:51 +05:30
prathamesh0
cce45bda57
Add prometheus metrics collection (#33) 
* Upgrade geth

* Add prometheus metrics collection

* Update README
 2022-05-23 16:56:48 +05:30
Ian Norden
751e064f8f
Merge pull request #28 from vulcanize/release-v3.0.2 
add env bindings
 2022-05-17 22:36:08 -05:00