cerc-io/ipld-eth-state-snapshot
11
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor to use statediff plugin #1

Merged
roysc merged 27 commits from refactor-use-plugin into v5 2023-09-29 18:43:28 +00:00
Conversation 6 Commits 27 Files Changed 96
+2036 -3564
roysc commented 2023-08-30 17:20:51 +00:00
Owner
Copy Link
Copy Source
  • Refactors to replace most of the code with the statediff plugin (from branch cerc-io/plugeth-statediff#15).
  • Adds basic CI test workflows for Gitea
  • Refactors fixtures to use https://git.vdb.to/cerc-io/eth-testing
  • Renames env vars for consistency with flags and other services:
    • LOGRUS_{LEVEL,FILE} => LOG_LEVEL, etc.
    • LVL_DB_PATH => LEVELDB_PATH
    • ANCIENT_DB_PATH => LEVELDB_ANCIENT
    • These will need to be updated wherever they are used
* Refactors to replace most of the code with the statediff plugin (from branch cerc-io/plugeth-statediff#15). * Adds basic CI test workflows for Gitea * Refactors fixtures to use https://git.vdb.to/cerc-io/eth-testing * Renames env vars for consistency with flags and other services: - LOGRUS_{LEVEL,FILE} => LOG_LEVEL, etc. - LVL_DB_PATH => LEVELDB_PATH - ANCIENT_DB_PATH => LEVELDB_ANCIENT - These will need to be updated wherever they are used
roysc added 72 commits 2023-08-30 17:20:53 +00:00
Updates to use v4 schema 26ae3ab008
Change ON CONFLICT clauses to do nothing and upgrade geth 44d48b8644
Index block number using string in file mode fc9e82f717
Merge pull request #29 from deep-stack/pm-db-sharding 04b57ca186 
Updates to use v4 schema
Upgrade vulcanize-geth to v4 67ec456e88
Merge pull request #30 from deep-stack/pm-upgrade-geth b3af4e9e75 
Upgrade vulcanize-geth to v4
Fix insert statements and upgrade dependencies (#31) 0a1ea3ae4a
Merge changes to add env bindings and prometheus metrics dc27487b66
Merge pull request #34 from deep-stack/pm-merge-metrics d3a149899b 
Merge changes to add env bindings and prometheus metrics
Merge branch 'master' into feature/update-go-geth-sharding cd2559e434
Update geth versions 8aab6a5137
Merge pull request #40 from vulcanize/feature/update-go-geth-sharding c9d398b5af 
Update go and geth versions
Merge changes to fix iterator and update baef795834
Merge pull request #45 from vulcanize/ng-merge-fix-iterator c2ecea6d61 
Merge changes to fix iterator and update
Add command to take an in-place snapshot (#42) b241bf05eb 
* Add command to take an in-place snapshot

* Add test data for in place snapshot unit test

* Implement unit test for inplace snapshot

* Add check for storage IPLD

* Run unit tests sequentially

* Add github workflow for unit test

* Add missing checks for state and storage cid fields

* Add more storage nodes to test

* Update ipld-eth-db version for tests

* Add comments for inplace snapshot test data

* Add in-place snapshot cmd in readme

* Implement defer pattern for db transaction

* Log transaction commit or rollback error

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
Upgrade geth (#51) ac00ee280f 
* Fix Makefile for installing mockgen tool

* Upgrade geth

* Add sleep in the failing recovery test

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
Add instructions to import snapshot data into database (#52) 2af26ad583 
* Add instructions to import snapshot data into database

* Add monitoring and update data processing in README

* Update instructions to import snapshot
Fix typo in config format in README (#53) c0a7fdf9e8
update for go-ethereum 1.10.20 257fac1a0c
Merge pull request #54 from vulcanize/release-v4.1.1-alpha 289fb63568 
update for go-ethereum 1.10.20
Upgrade dependencies (#56) 5002c82038
Account selective snapshot (#46) 05aeeab581 
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided

* config and env updates

* cmd update

* Encode watched address path bytes to hex for comparison

* actually ignore the subtries that are not along the paths of interest

* Fixes for account selective snapshot

* Use non-concurrent iterator when having a single worker

* Only index root node when starting path of an iterator is nil

* Upgrade deps

* Avoid tracking iterators and skip recovery test

* Fix recovery mechanism, use sync Map instead of buffered channels

* Add test for account selective snapshot

* Continue traversal with concurrent iterators with starting path nil

* Use errgroup to simplify error handling with concurrent iterators

* Check if all the nodes are indexed in the recovery test

* Use concurrency safe sync Map in account selective snapshot test

* Only track concurrent iterators and refactor code

* Fix node and recovered path comparison

* Revert back to using buffered channels for tracking iterators

* Add a metric to monitor number of active iterators

* Update docs

* Update seeked path after node is processed

* Return error on context cancellation from subtrie iteration

* Add tests for account selective snapshot recovery

* Explicity enforce concurrent iterator bounds to avoid duplicate nodes

* Update full snapshot test to check nodes being indexed

* Refactor code to simplify snapshot logic

* Remove unnecessary function argument

* Use ctx cancellation for handling signals

* Add descriptive comments

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
Add helper scripts for data dump correction (#57) be544a3424 
* Add a script to find bad data in CSV file dumps

* Add a script to delete bad rows from CSV file dumps

* Add instructions to run the scripts

* Reorganize instructions
updates for geth rebase on 1.10.23 321e37584d
Merge pull request #63 from vulcanize/rebase-1.10.23-wip 3f93a989dc 
updates for geth rebase on 1.10.23
Cerc refactor (#64) 989bd1c0f1 
* cerc refactor waiting on unpublished dependencies

* more cerc_refactor with published ipld-eth-db unstable

* TearDownDB refactor

* missed second publisher_test TearDownDB
dependency updates for geth 1.10.25 (#65) 9ea78c4ecf
Update schema (#61) c270f39de9 
Updates table to add header_cids.uncles_hash
dependency updates for geth 1.10.26 (#67) ac85fe29eb
bump vdb v4 geth to v1.11.5 c9dd85488b
use go1.19 6fe54aa8b4
refactor to work with v4 vdb geth v1.11.5 5c13d59515
missing .meta files in fixtures 6ede522ae0
Merge pull request #69 from cerc-io/ian/v4_dev e63ffbf0ad 
Update to use v4 vdb geth v1.11.5
update db version 3f3e77cbac
Merge pull request #70 from cerc-io/ian/v4_dev b6d7695536 
update db version
Update schema (#61) 97ee99a449 
Updates table to add header_cids.uncles_hash
Merge pull request #72 from cerc-io/ian/v5_rebase fb5a95d874 
Rebase
use types/models exported from vdb geth f83ab82424
update direct pg publisher f1a980f37c
update csv file publisher ead007f159
update Block1 and Chain2_Block32 test fixtures 14b1180161
removei in-place snapshot stuff 382ad92701
update service 22ecd4065a
go mod tidy 4245b80a4a
bump db version in docker compose 0a04baab17
Add Docker files (#73) 82176ea41c
update publisher interface 4ee75a3371
use v5.0.2 statediffing geth 32b637671d
update compose fdb105b769
finish updating test fixtures 3d8064ccbb
fix service.go: prevent panics on subtrie iterators with odd-length prefix-paths 4081787b03
update test helpers 7af17b1851
update service_test with new fixtures ea4c1042c4
update go mod 2dd9221467
update misc 6de5b9e96c
fix NUMERIC formattign error 98015c4c87
Merge pull request #71 from cerc-io/ian/v5_dev 7e426b2ca5 
v5 upgrade
update geth and ipld-eth-db deps 81e286399b
Merge pull request #74 from cerc-io/ian/v5_dev 0cab03b98e 
update geth and ipld-eth-db deps
fix overshadowing of snap.Tx 891fca89bd
Merge pull request #75 from cerc-io/ian/v5_dev dd86f02997 
fix overshadowing of snap.Tx
Add progress counter to prometheus output (#76) 768357293c 
* Add a progress counter by checking the distance already traversed from the startPath to endPath in a bounded iterator vs the estimated number of iterations.
avoid overshadowing err in defers fe88e90181
Merge pull request #77 from cerc-io/ian/v5_dev da02e5ac12 
fix: err overshadowing in defers
Update to geth v1.11.6-statediff-5.0.8 (#79) 4e0b481ea5 
* Update to geth v1.11.6-statediff-5.0.8
refactor to use statediff plugin 4699fba64d
refactor tests 5e7465565e
rm unused flags 63516f2895
readme cleanup aa3d1aecf5
[wip] CI tests af452b174c
[wip] go mod update 357585b1c0
roysc changed title from WIP: Refactor to use statediff plugin to WIP: Refactor to use statediff plugin 2023-08-30 17:23:58 +00:00
roysc changed target branch from master to v5 2023-08-30 17:23:59 +00:00
roysc requested review from i-norden 2023-09-01 21:09:04 +00:00
roysc force-pushed refactor-use-plugin from 357585b1c0 to c6170f063c 2023-09-02 11:40:52 +00:00 Compare
roysc marked the pull request as ready for review 2023-09-02 11:41:49 +00:00
roysc added 1 commit 2023-09-05 16:46:12 +00:00
rm old todo
Test / Run unit and integration tests (pull_request) Successful in 4m44s
Details
6969595d37
roysc marked the pull request as work in progress 2023-09-06 10:04:49 +00:00
roysc commented 2023-09-06 10:06:19 +00:00
Author
Owner
Copy Link
Copy Source

Still need to refactor the per-iterator metrics for this

Still need to refactor the per-iterator metrics for this
roysc requested review from telackey 2023-09-07 03:37:43 +00:00
cmd/stateSnapshot.go Outdated
@@ -87,3 +100,2 @@
stateSnapshotCmd.PersistentFlags().String(snapshot.LVL_DB_PATH_CLI, "", "path to primary datastore")
stateSnapshotCmd.PersistentFlags().String(snapshot.ANCIENT_DB_PATH_CLI, "", "path to ancient datastore")
stateSnapshotCmd.PersistentFlags().String(snapshot.LEVELDB_PATH_CLI, "", "path to primary datastore")
telackey commented 2023-09-15 21:30:45 +00:00
Contributor
Copy Link
Copy Source

Can you make sure these match eth-statediff-service and any changes are documented? We will need to change all the code that uses these (eg, chain-chunker) to make sure we use the right env.

Can you make sure these match eth-statediff-service and any changes are documented? We will need to change all the code that uses these (eg, chain-chunker) to make sure we use the right env.
roysc commented 2023-09-19 18:58:04 +00:00
Author
Owner
Copy Link
Copy Source

I mainly changed them to match eth-statediff-service so we're good there, but yes I'll review if anything needs an update including deployments. I'll mention it in Git and the readme, but I'm not actually sure if there's a better changelog-type place to document it.

I mainly changed them to match eth-statediff-service so we're good there, but yes I'll review if anything needs an update including deployments. I'll mention it in Git and the readme, but I'm not actually sure if there's a better changelog-type place to document it.
telackey marked this conversation as resolved
fixture/leaf_keys.go Outdated
@@ -1,264 +1,619 @@
package fixture
var Block1_StateNodeLeafKeys = []string{
"0x291b6bbca879d684337ee5e2bca5d18cfb2d3b7a97c8187e703309564280295e",
telackey commented 2023-09-15 21:31:36 +00:00
Contributor
Copy Link
Copy Source

Did the data change?

Did the data change?
roysc commented 2023-09-19 17:35:28 +00:00
Author
Owner
Copy Link
Copy Source

No, it's just sorted. I'll add a comment.

No, it's just sorted. I'll add a comment.
telackey marked this conversation as resolved
telackey commented 2023-09-15 21:35:45 +00:00
Contributor
Copy Link
Copy Source

We may need a new test, similar to what we talked about in #11, which compares before / after output.

We may need a new test, similar to what we talked about in #11, which compares before / after output.
telackey commented 2023-09-26 01:55:40 +00:00
Contributor
Copy Link
Copy Source

Is this basically good to go?

Is this basically good to go?
roysc added 4 commits 2023-09-26 18:01:38 +00:00
misc 5b0cc124a5
per-iterator prom metrics e6c103bfb6
refactor fixtures 80dcf3019d
update go mods
Test / Run unit and integration tests (pull_request) Failing after 2m7s
Details
5302122509
roysc added 2 commits 2023-09-27 13:57:04 +00:00
rename LOGRUS_* to LOG_* 190a714e53
update plugin
Test / Run unit and integration tests (pull_request) Failing after 4m0s
Details
66857e6ce1
roysc added 1 commit 2023-09-27 14:18:20 +00:00
document changed vars
Test / Run unit and integration tests (pull_request) Has been cancelled
Details
a0a620a09d
roysc marked the pull request as ready for review 2023-09-27 14:19:06 +00:00
roysc force-pushed refactor-use-plugin from a0a620a09d to 7f32e9f699 2023-09-27 14:20:52 +00:00 Compare
roysc added 1 commit 2023-09-27 15:05:51 +00:00
update ci test
Test / Run unit and integration tests (pull_request) Successful in 4m20s
Details
29a470378d
roysc added 1 commit 2023-09-27 16:04:07 +00:00
use the old fixture
Test / Run unit and integration tests (pull_request) Has been cancelled
Details
b0900a90e2
roysc force-pushed refactor-use-plugin from b0900a90e2 to 674c7e75f3 2023-09-27 16:06:55 +00:00 Compare
roysc added 1 commit 2023-09-28 00:36:47 +00:00
fix
Test / Run unit and integration tests (pull_request) Successful in 4m6s
Details
d1a6e33713
roysc referenced this issue from a commit 2023-09-28 03:35:49 +00:00
Add WriteStateSnapshot (#15)
roysc added 1 commit 2023-09-28 04:07:27 +00:00
update plugin
Test / Run unit and integration tests (pull_request) Successful in 4m11s
Details
40146d9da2
roysc added 3 commits 2023-09-28 12:09:53 +00:00
missing env wiring b988bf7b59
compare-snapshots script 0baaa39e63 
add compliance test to ci
update plugin
Test / Run unit and integration tests (pull_request) Successful in 4m29s
Details
Test / Run compliance tests (pull_request) Failing after 4m24s
Details
0906cf3ab9
roysc requested review from telackey 2023-09-28 12:10:00 +00:00
roysc commented 2023-09-28 12:12:28 +00:00
Author
Owner
Copy Link
Copy Source

Compliance test is added to CI.

Compliance test is added to CI.
roysc added 1 commit 2023-09-28 12:43:36 +00:00
make sure db is ready
Test / Run unit and integration tests (pull_request) Successful in 4m39s
Details
Test / Run compliance tests (pull_request) Successful in 5m7s
Details
027f014d31
telackey approved these changes 2023-09-28 17:11:09 +00:00
telackey left a comment
Contributor
Copy Link
Copy Source

Looks great.

Looks great.
.gitea/workflows/test.yml
@@ -0,0 +101,4 @@
working-directory: ./ipld-eth-state-snapshot
run: go build -o ../snapshot-current .
- name: Checkout canonical version
telackey commented 2023-09-28 17:09:05 +00:00
Contributor
Copy Link
Copy Source

This is a thing of beauty.

This is a thing of beauty.
😆 1
roysc marked this conversation as resolved
roysc added 1 commit 2023-09-29 12:39:55 +00:00
ci- rm redundant vars
Test / Run unit and integration tests (pull_request) Successful in 4m48s
Details
Test / Run compliance tests (pull_request) Successful in 5m10s
Details
d9a9929130
roysc added 1 commit 2023-09-29 18:09:48 +00:00
update dep
Test / Run unit and integration tests (pull_request) Successful in 5m6s
Details
Test / Run compliance tests (pull_request) Successful in 6m3s
Details
226bb04264
roysc merged commit 00141776bf into v5 2023-09-29 18:43:28 +00:00
roysc deleted branch refactor-use-plugin 2023-09-29 18:43:28 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
i-norden
telackey
Labels
Clear labels
Copied from Github
An issue or PR manually copied from GitHub.
 Kind/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
AFDudley (A. F. Dudley) Alex cerccicd ericlewis i-norden (Ian Norden) rachmaninovquar rjk-laconic roysc (Roy Crihfield) srw (srw) zramsay
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cerc-io/ipld-eth-state-snapshot#1