cerc-io/ipld-eth-state-snapshot
16
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity
163 Commits 10 Branches 27 Tags 7.2 MiB
e954370cd6
Commit Graph

30 Commits

Author SHA1 Message Date
Roy Crihfield
00c2cd66bc [wip] update deps 2024-04-24 17:37:05 +08:00
Roy Crihfield
00141776bf Refactor to use statediff plugin (#1) 
* Refactors to replace most of the code with the statediff plugin.
* Adds basic CI test workflows for Gitea
* Refactors fixtures to use https://git.vdb.to/cerc-io/eth-testing
* Renames env vars for consistency with flags and other services:
  - LOGRUS_{LEVEL,FILE} => LOG_LEVEL, etc.
  - LVL_DB_PATH => LEVELDB_PATH
  - ANCIENT_DB_PATH => LEVELDB_ANCIENT
  - These will need to be updated wherever they are used

Reviewed-on: #1
 2023-09-29 18:43:26 +00:00
Thomas E Lackey
4e0b481ea5
Update to geth v1.11.6-statediff-5.0.8 (#79) 
* Update to geth v1.11.6-statediff-5.0.8
 2023-07-21 19:05:01 -05:00
i-norden
81e286399b update geth and ipld-eth-db deps 2023-05-15 11:32:29 -05:00
i-norden
2dd9221467 update go mod 2023-05-12 10:27:20 -05:00
i-norden
32b637671d use v5.0.2 statediffing geth 2023-05-10 13:11:23 -05:00
i-norden
4245b80a4a go mod tidy 2023-04-12 13:13:04 -05:00
i-norden
c9dd85488b bump vdb v4 geth to v1.11.5 2023-03-31 10:39:00 -05:00
Michael
ac85fe29eb
dependency updates for geth 1.10.26 (#67) 2022-11-08 14:11:11 -05:00
Michael
9ea78c4ecf
dependency updates for geth 1.10.25 (#65) 2022-09-26 09:14:15 -04:00
Michael
989bd1c0f1
Cerc refactor (#64) 
* cerc refactor waiting on unpublished dependencies

* more cerc_refactor with published ipld-eth-db unstable

* TearDownDB refactor

* missed second publisher_test TearDownDB
 2022-09-20 13:47:34 -04:00
Michael Shaw
321e37584d updates for geth rebase on 1.10.23 2022-09-02 15:21:29 -04:00
Ian Norden
05aeeab581
Account selective snapshot (#46) 
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided

* config and env updates

* cmd update

* Encode watched address path bytes to hex for comparison

* actually ignore the subtries that are not along the paths of interest

* Fixes for account selective snapshot

* Use non-concurrent iterator when having a single worker

* Only index root node when starting path of an iterator is nil

* Upgrade deps

* Avoid tracking iterators and skip recovery test

* Fix recovery mechanism, use sync Map instead of buffered channels

* Add test for account selective snapshot

* Continue traversal with concurrent iterators with starting path nil

* Use errgroup to simplify error handling with concurrent iterators

* Check if all the nodes are indexed in the recovery test

* Use concurrency safe sync Map in account selective snapshot test

* Only track concurrent iterators and refactor code

* Fix node and recovered path comparison

* Revert back to using buffered channels for tracking iterators

* Add a metric to monitor number of active iterators

* Update docs

* Update seeked path after node is processed

* Return error on context cancellation from subtrie iteration

* Add tests for account selective snapshot recovery

* Explicity enforce concurrent iterator bounds to avoid duplicate nodes

* Update full snapshot test to check nodes being indexed

* Refactor code to simplify snapshot logic

* Remove unnecessary function argument

* Use ctx cancellation for handling signals

* Add descriptive comments

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
 2022-08-03 17:05:04 +05:30
prathamesh0
5002c82038
Upgrade dependencies (#56) 2022-08-03 16:11:38 +05:30
Michael Shaw
257fac1a0c update for go-ethereum 1.10.20 2022-07-19 15:12:44 -04:00
prathamesh0
ac00ee280f
Upgrade geth (#51) 
* Fix Makefile for installing mockgen tool

* Upgrade geth

* Add sleep in the failing recovery test

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
 2022-07-13 11:39:36 +05:30
nabarun
baef795834 Merge changes to fix iterator and update 2022-06-10 14:12:34 +05:30
nabarun
59011cba3c Update go-eth-state-node-iterator version 2022-06-10 14:02:13 +05:30
Roy Crihfield
838dabcba6 update iterator mod 2022-06-08 20:12:22 +08:00
Abdul Rabbani
8aab6a5137 Update geth versions 2022-06-01 09:20:45 -04:00
Abdul Rabbani
a04777cb78 Updating Go and Geth Version 
Ran Tests:

```
09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v
# github.com/vulcanize/ipld-eth-state-snapshot/cmd
cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v
?       github.com/vulcanize/ipld-eth-state-snapshot    [no test files]
=== RUN   TestCreateSnapshot
=== RUN   TestCreateSnapshot/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#03
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestCreateSnapshot (0.20s)
    --- PASS: TestCreateSnapshot/case (0.15s)
    --- PASS: TestCreateSnapshot/case#01 (0.01s)
    --- PASS: TestCreateSnapshot/case#02 (0.02s)
    --- PASS: TestCreateSnapshot/case#03 (0.03s)
=== RUN   TestRecovery
=== RUN   TestRecovery/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestRecovery/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestRecovery/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestRecovery (0.40s)
    --- PASS: TestRecovery/case (0.12s)
    --- PASS: TestRecovery/case#01 (0.19s)
    --- PASS: TestRecovery/case#02 (0.09s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot       1.109s
=== RUN   TestWriting
--- PASS: TestWriting (0.00s)
=== RUN   TestPgCopy
    publisher_test.go:84: set TEST_WITH_DB to enable test
--- SKIP: TestPgCopy (0.00s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file  0.279s
=== RUN   TestBasic
    publisher_test.go:47: set TEST_WITH_DB to enable test
--- SKIP: TestBasic (0.00s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg    0.647s
```
 2022-06-01 09:09:19 -04:00
prathamesh0
cce45bda57
Add prometheus metrics collection (#33) 
* Upgrade geth

* Add prometheus metrics collection

* Update README
 2022-05-23 16:56:48 +05:30
i-norden
060eb5cf67 add some logs and guards, update module name, update readme 2022-03-30 18:57:30 -05:00
Roy Crihfield
56acb77829 update iterator mod 2022-03-14 23:31:48 +08:00
Roy Crihfield
c4b4f232e9 create csv file publisher; update geth and schema 2022-02-14 07:22:31 +08:00
i-norden
1919738fec update go mod 2022-01-26 12:07:52 -06:00
Roy Crihfield
159f93902e tests - mock publisher with fixtures 
* go structs for header, etc.
* refactor: pkg/types

* add fixture chaindata
 2022-01-14 21:43:00 -06:00
i-norden
b5d31748e0 bump geth version 2021-12-28 23:30:17 -06:00
Arijit Das
ba8e32ce97 Update geth and implement transaction batching. 2021-12-13 20:31:32 +05:30
Ian Norden
e50dd95c29 go modules 2020-07-01 14:10:44 -05:00