cerc-io/ipld-eth-state-snapshot
16
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity
116 Commits 10 Branches 27 Tags 7.2 MiB
321e37584d
Commit Graph

116 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Michael Shaw
321e37584d updates for geth rebase on 1.10.23 2022-09-02 15:21:29 -04:00
Ian Norden
05aeeab581
Account selective snapshot (#46) 
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided

* config and env updates

* cmd update

* Encode watched address path bytes to hex for comparison

* actually ignore the subtries that are not along the paths of interest

* Fixes for account selective snapshot

* Use non-concurrent iterator when having a single worker

* Only index root node when starting path of an iterator is nil

* Upgrade deps

* Avoid tracking iterators and skip recovery test

* Fix recovery mechanism, use sync Map instead of buffered channels

* Add test for account selective snapshot

* Continue traversal with concurrent iterators with starting path nil

* Use errgroup to simplify error handling with concurrent iterators

* Check if all the nodes are indexed in the recovery test

* Use concurrency safe sync Map in account selective snapshot test

* Only track concurrent iterators and refactor code

* Fix node and recovered path comparison

* Revert back to using buffered channels for tracking iterators

* Add a metric to monitor number of active iterators

* Update docs

* Update seeked path after node is processed

* Return error on context cancellation from subtrie iteration

* Add tests for account selective snapshot recovery

* Explicity enforce concurrent iterator bounds to avoid duplicate nodes

* Update full snapshot test to check nodes being indexed

* Refactor code to simplify snapshot logic

* Remove unnecessary function argument

* Use ctx cancellation for handling signals

* Add descriptive comments

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
 2022-08-03 17:05:04 +05:30
prathamesh0
5002c82038
Upgrade dependencies (#56) 2022-08-03 16:11:38 +05:30
Michael
289fb63568
Merge pull request #54 from vulcanize/release-v4.1.1-alpha 
update for go-ethereum 1.10.20
 2022-07-20 17:18:58 -04:00
Michael Shaw
257fac1a0c update for go-ethereum 1.10.20 2022-07-19 15:12:44 -04:00
prathamesh0
c0a7fdf9e8
Fix typo in config format in README (#53) 2022-07-18 16:48:58 +05:30
prathamesh0
2af26ad583
Add instructions to import snapshot data into database (#52) 
* Add instructions to import snapshot data into database

* Add monitoring and update data processing in README

* Update instructions to import snapshot
 2022-07-18 15:30:23 +05:30
prathamesh0
ac00ee280f
Upgrade geth (#51) 
* Fix Makefile for installing mockgen tool

* Upgrade geth

* Add sleep in the failing recovery test

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
 2022-07-13 11:39:36 +05:30
prathamesh0
b241bf05eb
Add command to take an in-place snapshot (#42) 
* Add command to take an in-place snapshot

* Add test data for in place snapshot unit test

* Implement unit test for inplace snapshot

* Add check for storage IPLD

* Run unit tests sequentially

* Add github workflow for unit test

* Add missing checks for state and storage cid fields

* Add more storage nodes to test

* Update ipld-eth-db version for tests

* Add comments for inplace snapshot test data

* Add in-place snapshot cmd in readme

* Implement defer pattern for db transaction

* Log transaction commit or rollback error

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
 2022-06-15 12:51:26 +05:30
Ashwin Phatak
c2ecea6d61
Merge pull request #45 from vulcanize/ng-merge-fix-iterator 
Merge changes to fix iterator and update
 2022-06-10 14:24:33 +05:30
nabarun
baef795834 Merge changes to fix iterator and update 2022-06-10 14:12:34 +05:30
nikugogoi
4979e4090d
Merge pull request #43 from roysc/fix-iterator-and-update 
Fix iterator and update
 2022-06-10 14:03:43 +05:30
nabarun
59011cba3c Update go-eth-state-node-iterator version 2022-06-10 14:02:13 +05:30
Roy Crihfield
8371344756 clean up logging 2022-06-08 20:12:22 +08:00
Roy Crihfield
ef391d8d31 simplify tracker, prevent send to closed chan 2022-06-08 20:12:22 +08:00
Roy Crihfield
838dabcba6 update iterator mod 2022-06-08 20:12:22 +08:00
Roy Crihfield
8572c36ee7 decrement path from restored iterator 
to cover node gaps
 2022-06-08 20:12:22 +08:00
Ashwin Phatak
74f82a641d Fix dependency 2022-06-08 20:05:38 +08:00
Roy Crihfield
3e24972a89 try to fix hanging iterator 2022-06-08 20:05:38 +08:00
Ashwin Phatak
7129044eae Log progress info 2022-06-08 18:32:10 +08:00
Abdul Rabbani
c9d398b5af
Merge pull request #40 from vulcanize/feature/update-go-geth-sharding 
Update go and geth versions
 2022-06-01 09:22:04 -04:00
Abdul Rabbani
8aab6a5137 Update geth versions 2022-06-01 09:20:45 -04:00
Abdul Rabbani
cd2559e434 Merge branch 'master' into feature/update-go-geth-sharding 2022-06-01 09:19:18 -04:00
Abdul Rabbani
d213644ff4
Merge pull request #39 from vulcanize/feature/update-go-geth 
Updating Go and Geth Version
 2022-06-01 09:10:35 -04:00
Abdul Rabbani
a04777cb78 Updating Go and Geth Version 
Ran Tests:

```
09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v
# github.com/vulcanize/ipld-eth-state-snapshot/cmd
cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v
?       github.com/vulcanize/ipld-eth-state-snapshot    [no test files]
=== RUN   TestCreateSnapshot
=== RUN   TestCreateSnapshot/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#03
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestCreateSnapshot (0.20s)
    --- PASS: TestCreateSnapshot/case (0.15s)
    --- PASS: TestCreateSnapshot/case#01 (0.01s)
    --- PASS: TestCreateSnapshot/case#02 (0.02s)
    --- PASS: TestCreateSnapshot/case#03 (0.03s)
=== RUN   TestRecovery
=== RUN   TestRecovery/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestRecovery/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestRecovery/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestRecovery (0.40s)
    --- PASS: TestRecovery/case (0.12s)
    --- PASS: TestRecovery/case#01 (0.19s)
    --- PASS: TestRecovery/case#02 (0.09s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot       1.109s
=== RUN   TestWriting
--- PASS: TestWriting (0.00s)
=== RUN   TestPgCopy
    publisher_test.go:84: set TEST_WITH_DB to enable test
--- SKIP: TestPgCopy (0.00s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file  0.279s
=== RUN   TestBasic
    publisher_test.go:47: set TEST_WITH_DB to enable test
--- SKIP: TestBasic (0.00s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg    0.647s
```
 2022-06-01 09:09:19 -04:00
Ashwin Phatak
c785c12274
Merge pull request #35 from vulcanize/ng-file-ethereum-nodeId 
Add ethereum config to fix db import in file mode
 2022-05-25 18:38:09 +05:30
nabarun
23c4d22885 Add ethereum config to fix db import in file mode 2022-05-25 18:15:08 +05:30
Ashwin Phatak
d3a149899b
Merge pull request #34 from deep-stack/pm-merge-metrics 
Merge changes to add env bindings and prometheus metrics
 2022-05-23 18:14:06 +05:30
prathamesh0
dc27487b66 Merge changes to add env bindings and prometheus metrics 2022-05-23 17:11:51 +05:30
prathamesh0
cce45bda57
Add prometheus metrics collection (#33) 
* Upgrade geth

* Add prometheus metrics collection

* Update README
 2022-05-23 16:56:48 +05:30
Ian Norden
751e064f8f
Merge pull request #28 from vulcanize/release-v3.0.2 
add env bindings
 2022-05-17 22:36:08 -05:00
i-norden
004f3e6665 constants for CLI binding strings 2022-05-17 22:33:20 -05:00
prathamesh0
0a1ea3ae4a
Fix insert statements and upgrade dependencies (#31) 2022-05-17 19:19:52 +05:30
Ashwin Phatak
b3af4e9e75
Merge pull request #30 from deep-stack/pm-upgrade-geth 
Upgrade vulcanize-geth to v4
 2022-05-16 14:05:28 +05:30
prathamesh0
67ec456e88 Upgrade vulcanize-geth to v4 2022-05-16 13:48:26 +05:30
Ashwin Phatak
04b57ca186
Merge pull request #29 from deep-stack/pm-db-sharding 
Updates to use v4 schema
 2022-05-13 16:06:09 +05:30
prathamesh0
fc9e82f717 Index block number using string in file mode 2022-05-13 14:00:40 +05:30
prathamesh0
44d48b8644 Change ON CONFLICT clauses to do nothing and upgrade geth 2022-05-13 09:28:36 +05:30
prathamesh0
26ae3ab008 Updates to use v4 schema 2022-04-19 18:14:25 +05:30
i-norden
31105a009c add env bindings 2022-04-04 14:35:18 -05:00
Ian Norden
560199f930
Merge pull request #25 from vulcanize/release-v3.0.1 
fix minor bug
 2022-03-31 10:34:36 -05:00
i-norden
060eb5cf67 add some logs and guards, update module name, update readme 2022-03-30 18:57:30 -05:00
i-norden
de39842049 fix minor bug 2022-03-30 14:09:32 -05:00
Ian Norden
9e768dc2ba
Merge pull request #22 from vulcanize/recovery 
Service state recovery
 2022-03-17 07:09:25 -05:00
Roy Crihfield
8d1e79d22e handle sigterm 2022-03-16 18:48:51 +08:00
Roy Crihfield
b732ffc5f2 fix race condition 2022-03-14 23:34:26 +08:00
Roy Crihfield
2481d0a28d recovery - handle signals 2022-03-14 23:34:26 +08:00
Roy Crihfield
02488e2b79 logging tweaks 2022-03-14 23:32:00 +08:00
Roy Crihfield
7a2ef4f542 fix fixtures 2022-03-14 23:32:00 +08:00
Roy Crihfield
56acb77829 update iterator mod 2022-03-14 23:31:48 +08:00