cerc-io/ipld-eth-state-snapshot
15
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor to use statediff plugin #1

Merged
roysc merged 27 commits from refactor-use-plugin into v5 2023-09-29 18:43:28 +00:00
Conversation 4 Commits 27 Files Changed 96 +2036 -3564
roysc commented 2023-08-30 17:20:51 +00:00
Member
Copy Link
  • Refactors to replace most of the code with the statediff plugin (from branch cerc-io/plugeth-statediff#15).
  • Adds basic CI test workflows for Gitea
  • Refactors fixtures to use https://git.vdb.to/cerc-io/eth-testing
  • Renames env vars for consistency with flags and other services:
    • LOGRUS_{LEVEL,FILE} => LOG_LEVEL, etc.
    • LVL_DB_PATH => LEVELDB_PATH
    • ANCIENT_DB_PATH => LEVELDB_ANCIENT
    • These will need to be updated wherever they are used
* Refactors to replace most of the code with the statediff plugin (from branch cerc-io/plugeth-statediff#15). * Adds basic CI test workflows for Gitea * Refactors fixtures to use https://git.vdb.to/cerc-io/eth-testing * Renames env vars for consistency with flags and other services: - LOGRUS_{LEVEL,FILE} => LOG_LEVEL, etc. - LVL_DB_PATH => LEVELDB_PATH - ANCIENT_DB_PATH => LEVELDB_ANCIENT - These will need to be updated wherever they are used
roysc added 72 commits 2023-08-30 17:20:53 +00:00 
Updates to use v4 schema
Upgrade vulcanize-geth to v4
Merge changes to add env bindings and prometheus metrics
Update go and geth versions
Merge changes to fix iterator and update
* Add command to take an in-place snapshot

* Add test data for in place snapshot unit test

* Implement unit test for inplace snapshot

* Add check for storage IPLD

* Run unit tests sequentially

* Add github workflow for unit test

* Add missing checks for state and storage cid fields

* Add more storage nodes to test

* Update ipld-eth-db version for tests

* Add comments for inplace snapshot test data

* Add in-place snapshot cmd in readme

* Implement defer pattern for db transaction

* Log transaction commit or rollback error

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Fix Makefile for installing mockgen tool

* Upgrade geth

* Add sleep in the failing recovery test

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
* Add instructions to import snapshot data into database

* Add monitoring and update data processing in README

* Update instructions to import snapshot
update for go-ethereum 1.10.20
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided

* config and env updates

* cmd update

* Encode watched address path bytes to hex for comparison

* actually ignore the subtries that are not along the paths of interest

* Fixes for account selective snapshot

* Use non-concurrent iterator when having a single worker

* Only index root node when starting path of an iterator is nil

* Upgrade deps

* Avoid tracking iterators and skip recovery test

* Fix recovery mechanism, use sync Map instead of buffered channels

* Add test for account selective snapshot

* Continue traversal with concurrent iterators with starting path nil

* Use errgroup to simplify error handling with concurrent iterators

* Check if all the nodes are indexed in the recovery test

* Use concurrency safe sync Map in account selective snapshot test

* Only track concurrent iterators and refactor code

* Fix node and recovered path comparison

* Revert back to using buffered channels for tracking iterators

* Add a metric to monitor number of active iterators

* Update docs

* Update seeked path after node is processed

* Return error on context cancellation from subtrie iteration

* Add tests for account selective snapshot recovery

* Explicity enforce concurrent iterator bounds to avoid duplicate nodes

* Update full snapshot test to check nodes being indexed

* Refactor code to simplify snapshot logic

* Remove unnecessary function argument

* Use ctx cancellation for handling signals

* Add descriptive comments

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
* Add a script to find bad data in CSV file dumps

* Add a script to delete bad rows from CSV file dumps

* Add instructions to run the scripts

* Reorganize instructions
updates for geth rebase on 1.10.23
* cerc refactor waiting on unpublished dependencies

* more cerc_refactor with published ipld-eth-db unstable

* TearDownDB refactor

* missed second publisher_test TearDownDB
Updates table to add header_cids.uncles_hash
Update to use v4 vdb geth v1.11.5
update db version
Updates table to add header_cids.uncles_hash
Rebase
v5 upgrade
update geth and ipld-eth-db deps
fix overshadowing of snap.Tx
* Add a progress counter by checking the distance already traversed from the startPath to endPath in a bounded iterator vs the estimated number of iterations.
fix: err overshadowing in defers
* Update to geth v1.11.6-statediff-5.0.8
roysc changed title from WIP: Refactor to use statediff plugin to WIP: Refactor to use statediff plugin 2023-08-30 17:23:58 +00:00
roysc changed target branch from master to v5 2023-08-30 17:23:59 +00:00
roysc requested review from i-norden 2023-09-01 21:09:04 +00:00
roysc force-pushed refactor-use-plugin from 357585b1c0 to c6170f063c 2023-09-02 11:40:52 +00:00 Compare
roysc changed title from WIP: Refactor to use statediff plugin to Refactor to use statediff plugin 2023-09-02 11:41:49 +00:00
roysc added 1 commit 2023-09-05 16:46:12 +00:00
rm old todo
All checks were successful
Test / Run unit and integration tests (pull_request) Successful in 4m44s
Details
6969595d37
roysc changed title from Refactor to use statediff plugin to WIP: Refactor to use statediff plugin 2023-09-06 10:04:49 +00:00
roysc commented 2023-09-06 10:06:19 +00:00
Author
Member
Copy Link

Still need to refactor the per-iterator metrics for this

Still need to refactor the per-iterator metrics for this
roysc requested review from telackey 2023-09-07 03:37:43 +00:00
telackey requested changes 2023-09-15 21:33:28 +00:00
cmd/stateSnapshot.go Outdated
@ -87,3 +100,2 @@
stateSnapshotCmd.PersistentFlags().String(snapshot.LVL_DB_PATH_CLI, "", "path to primary datastore")
stateSnapshotCmd.PersistentFlags().String(snapshot.ANCIENT_DB_PATH_CLI, "", "path to ancient datastore")
stateSnapshotCmd.PersistentFlags().String(snapshot.LEVELDB_PATH_CLI, "", "path to primary datastore")
telackey commented 2023-09-15 21:30:45 +00:00
Member
Copy Link

Can you make sure these match eth-statediff-service and any changes are documented? We will need to change all the code that uses these (eg, chain-chunker) to make sure we use the right env.

Can you make sure these match eth-statediff-service and any changes are documented? We will need to change all the code that uses these (eg, chain-chunker) to make sure we use the right env.
roysc commented 2023-09-19 18:58:04 +00:00
Author
Member
Copy Link

I mainly changed them to match eth-statediff-service so we're good there, but yes I'll review if anything needs an update including deployments. I'll mention it in Git and the readme, but I'm not actually sure if there's a better changelog-type place to document it.

I mainly changed them to match eth-statediff-service so we're good there, but yes I'll review if anything needs an update including deployments. I'll mention it in Git and the readme, but I'm not actually sure if there's a better changelog-type place to document it.
telackey marked this conversation as resolved
fixture/leaf_keys.go Outdated
@ -1,264 +1,619 @@
package fixture
var Block1_StateNodeLeafKeys = []string{
"0x291b6bbca879d684337ee5e2bca5d18cfb2d3b7a97c8187e703309564280295e",
telackey commented 2023-09-15 21:31:36 +00:00
Member
Copy Link

Did the data change?

Did the data change?
roysc commented 2023-09-19 17:35:28 +00:00
Author
Member
Copy Link

No, it's just sorted. I'll add a comment.

No, it's just sorted. I'll add a comment.
telackey marked this conversation as resolved
telackey commented 2023-09-15 21:35:45 +00:00
Member
Copy Link

We may need a new test, similar to what we talked about in #11, which compares before / after output.

We may need a new test, similar to what we talked about in #11, which compares before / after output.
telackey commented 2023-09-26 01:55:40 +00:00
Member
Copy Link

Is this basically good to go?

Is this basically good to go?
roysc added 4 commits 2023-09-26 18:01:38 +00:00
update go mods
Some checks failed
Test / Run unit and integration tests (pull_request) Failing after 2m7s
Details
5302122509
roysc added 2 commits 2023-09-27 13:57:04 +00:00
update plugin
Some checks failed
Test / Run unit and integration tests (pull_request) Failing after 4m0s
Details
66857e6ce1
roysc added 1 commit 2023-09-27 14:18:20 +00:00
document changed vars
Some checks reported warnings
Test / Run unit and integration tests (pull_request) Has been cancelled
Details
a0a620a09d
roysc changed title from WIP: Refactor to use statediff plugin to Refactor to use statediff plugin 2023-09-27 14:19:06 +00:00
roysc force-pushed refactor-use-plugin from a0a620a09d to 7f32e9f699 2023-09-27 14:20:52 +00:00 Compare
roysc added 1 commit 2023-09-27 15:05:51 +00:00
update ci test
All checks were successful
Test / Run unit and integration tests (pull_request) Successful in 4m20s
Details
29a470378d
roysc added 1 commit 2023-09-27 16:04:07 +00:00
use the old fixture
Some checks reported warnings
Test / Run unit and integration tests (pull_request) Has been cancelled
Details
b0900a90e2
roysc force-pushed refactor-use-plugin from b0900a90e2 to 674c7e75f3 2023-09-27 16:06:55 +00:00 Compare
roysc added 1 commit 2023-09-28 00:36:47 +00:00
fix
All checks were successful
Test / Run unit and integration tests (pull_request) Successful in 4m6s
Details
d1a6e33713
roysc referenced this issue from a commit 2023-09-28 03:35:49 +00:00
Add WriteStateSnapshot (#15)
roysc added 1 commit 2023-09-28 04:07:27 +00:00
update plugin
All checks were successful
Test / Run unit and integration tests (pull_request) Successful in 4m11s
Details
40146d9da2
roysc added 3 commits 2023-09-28 12:09:53 +00:00 
add compliance test to ci
update plugin
Some checks failed
Test / Run unit and integration tests (pull_request) Successful in 4m29s
Details
Test / Run compliance tests (pull_request) Failing after 4m24s
Details
0906cf3ab9
roysc requested review from telackey 2023-09-28 12:10:00 +00:00
roysc commented 2023-09-28 12:12:28 +00:00
Author
Member
Copy Link

Compliance test is added to CI.

Compliance test is added to CI.
roysc added 1 commit 2023-09-28 12:43:36 +00:00
make sure db is ready
All checks were successful
Test / Run unit and integration tests (pull_request) Successful in 4m39s
Details
Test / Run compliance tests (pull_request) Successful in 5m7s
Details
027f014d31
telackey approved these changes 2023-09-28 17:11:09 +00:00
telackey left a comment
Member
Copy Link

Looks great.

Looks great.
.gitea/workflows/test.yml
@ -0,0 +101,4 @@
working-directory: ./ipld-eth-state-snapshot
run: go build -o ../snapshot-current .
- name: Checkout canonical version
telackey commented 2023-09-28 17:09:05 +00:00
Member
Copy Link

This is a thing of beauty.

This is a thing of beauty.
😆 1
roysc marked this conversation as resolved
roysc added 1 commit 2023-09-29 12:39:55 +00:00
ci- rm redundant vars
All checks were successful
Test / Run unit and integration tests (pull_request) Successful in 4m48s
Details
Test / Run compliance tests (pull_request) Successful in 5m10s
Details
d9a9929130
roysc added 1 commit 2023-09-29 18:09:48 +00:00
update dep
All checks were successful
Test / Run unit and integration tests (pull_request) Successful in 5m6s
Details
Test / Run compliance tests (pull_request) Successful in 6m3s
Details
226bb04264
roysc merged commit 00141776bf into v5 2023-09-29 18:43:28 +00:00
roysc deleted branch refactor-use-plugin 2023-09-29 18:43:28 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
i-norden
telackey
Labels
Clear labels
  
Copied from Github

An issue or PR manually copied from GitHub.

  
Kind/Breaking

Breaking change that won't be backward compatible

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Security

This is security issue

  
Kind/Testing

Issue or pull request related to testing

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

No Label
Copied from Github
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/ipld-eth-state-snapshot#1
No description provided.
Delete Branch "refactor-use-plugin"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?