cerc-io/ipld-eth-state-snapshot
16
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity
165 Commits 10 Branches 27 Tags 7.2 MiB
6bee3b8a0d
Commit Graph

31 Commits

Author SHA1 Message Date
Roy Crihfield
6bee3b8a0d update deps 2024-04-24 18:00:28 +08:00
Roy Crihfield
00c2cd66bc [wip] update deps 2024-04-24 17:37:05 +08:00
Roy Crihfield
00141776bf Refactor to use statediff plugin (#1) 
* Refactors to replace most of the code with the statediff plugin.
* Adds basic CI test workflows for Gitea
* Refactors fixtures to use https://git.vdb.to/cerc-io/eth-testing
* Renames env vars for consistency with flags and other services:
  - LOGRUS_{LEVEL,FILE} => LOG_LEVEL, etc.
  - LVL_DB_PATH => LEVELDB_PATH
  - ANCIENT_DB_PATH => LEVELDB_ANCIENT
  - These will need to be updated wherever they are used

Reviewed-on: #1
 2023-09-29 18:43:26 +00:00
Thomas E Lackey
4e0b481ea5
Update to geth v1.11.6-statediff-5.0.8 (#79) 
* Update to geth v1.11.6-statediff-5.0.8
 2023-07-21 19:05:01 -05:00
i-norden
81e286399b update geth and ipld-eth-db deps 2023-05-15 11:32:29 -05:00
i-norden
2dd9221467 update go mod 2023-05-12 10:27:20 -05:00
i-norden
32b637671d use v5.0.2 statediffing geth 2023-05-10 13:11:23 -05:00
i-norden
4245b80a4a go mod tidy 2023-04-12 13:13:04 -05:00
i-norden
c9dd85488b bump vdb v4 geth to v1.11.5 2023-03-31 10:39:00 -05:00
Michael
ac85fe29eb
dependency updates for geth 1.10.26 (#67) 2022-11-08 14:11:11 -05:00
Michael
9ea78c4ecf
dependency updates for geth 1.10.25 (#65) 2022-09-26 09:14:15 -04:00
Michael
989bd1c0f1
Cerc refactor (#64) 
* cerc refactor waiting on unpublished dependencies

* more cerc_refactor with published ipld-eth-db unstable

* TearDownDB refactor

* missed second publisher_test TearDownDB
 2022-09-20 13:47:34 -04:00
Michael Shaw
321e37584d updates for geth rebase on 1.10.23 2022-09-02 15:21:29 -04:00
Ian Norden
05aeeab581
Account selective snapshot (#46) 
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided

* config and env updates

* cmd update

* Encode watched address path bytes to hex for comparison

* actually ignore the subtries that are not along the paths of interest

* Fixes for account selective snapshot

* Use non-concurrent iterator when having a single worker

* Only index root node when starting path of an iterator is nil

* Upgrade deps

* Avoid tracking iterators and skip recovery test

* Fix recovery mechanism, use sync Map instead of buffered channels

* Add test for account selective snapshot

* Continue traversal with concurrent iterators with starting path nil

* Use errgroup to simplify error handling with concurrent iterators

* Check if all the nodes are indexed in the recovery test

* Use concurrency safe sync Map in account selective snapshot test

* Only track concurrent iterators and refactor code

* Fix node and recovered path comparison

* Revert back to using buffered channels for tracking iterators

* Add a metric to monitor number of active iterators

* Update docs

* Update seeked path after node is processed

* Return error on context cancellation from subtrie iteration

* Add tests for account selective snapshot recovery

* Explicity enforce concurrent iterator bounds to avoid duplicate nodes

* Update full snapshot test to check nodes being indexed

* Refactor code to simplify snapshot logic

* Remove unnecessary function argument

* Use ctx cancellation for handling signals

* Add descriptive comments

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
 2022-08-03 17:05:04 +05:30
prathamesh0
5002c82038
Upgrade dependencies (#56) 2022-08-03 16:11:38 +05:30
Michael Shaw
257fac1a0c update for go-ethereum 1.10.20 2022-07-19 15:12:44 -04:00
prathamesh0
ac00ee280f
Upgrade geth (#51) 
* Fix Makefile for installing mockgen tool

* Upgrade geth

* Add sleep in the failing recovery test

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
 2022-07-13 11:39:36 +05:30
nabarun
baef795834 Merge changes to fix iterator and update 2022-06-10 14:12:34 +05:30
nabarun
59011cba3c Update go-eth-state-node-iterator version 2022-06-10 14:02:13 +05:30
Roy Crihfield
838dabcba6 update iterator mod 2022-06-08 20:12:22 +08:00
Abdul Rabbani
8aab6a5137 Update geth versions 2022-06-01 09:20:45 -04:00
Abdul Rabbani
a04777cb78 Updating Go and Geth Version 
Ran Tests:

```
09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v
# github.com/vulcanize/ipld-eth-state-snapshot/cmd
cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v
?       github.com/vulcanize/ipld-eth-state-snapshot    [no test files]
=== RUN   TestCreateSnapshot
=== RUN   TestCreateSnapshot/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestCreateSnapshot/case#03
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestCreateSnapshot (0.20s)
    --- PASS: TestCreateSnapshot/case (0.15s)
    --- PASS: TestCreateSnapshot/case#01 (0.01s)
    --- PASS: TestCreateSnapshot/case#02 (0.02s)
    --- PASS: TestCreateSnapshot/case#03 (0.03s)
=== RUN   TestRecovery
=== RUN   TestRecovery/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestRecovery/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN   TestRecovery/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestRecovery (0.40s)
    --- PASS: TestRecovery/case (0.12s)
    --- PASS: TestRecovery/case#01 (0.19s)
    --- PASS: TestRecovery/case#02 (0.09s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot       1.109s
=== RUN   TestWriting
--- PASS: TestWriting (0.00s)
=== RUN   TestPgCopy
    publisher_test.go:84: set TEST_WITH_DB to enable test
--- SKIP: TestPgCopy (0.00s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file  0.279s
=== RUN   TestBasic
    publisher_test.go:47: set TEST_WITH_DB to enable test
--- SKIP: TestBasic (0.00s)
PASS
ok      github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg    0.647s
```
 2022-06-01 09:09:19 -04:00
prathamesh0
cce45bda57
Add prometheus metrics collection (#33) 
* Upgrade geth

* Add prometheus metrics collection

* Update README
 2022-05-23 16:56:48 +05:30
i-norden
060eb5cf67 add some logs and guards, update module name, update readme 2022-03-30 18:57:30 -05:00
Roy Crihfield
56acb77829 update iterator mod 2022-03-14 23:31:48 +08:00
Roy Crihfield
c4b4f232e9 create csv file publisher; update geth and schema 2022-02-14 07:22:31 +08:00
i-norden
1919738fec update go mod 2022-01-26 12:07:52 -06:00
Roy Crihfield
159f93902e tests - mock publisher with fixtures 
* go structs for header, etc.
* refactor: pkg/types

* add fixture chaindata
 2022-01-14 21:43:00 -06:00
i-norden
b5d31748e0 bump geth version 2021-12-28 23:30:17 -06:00
Arijit Das
ba8e32ce97 Update geth and implement transaction batching. 2021-12-13 20:31:32 +05:30
Ian Norden
e50dd95c29 go modules 2020-07-01 14:10:44 -05:00