cerc-io/plugeth-statediff
11
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Use symmetric difference iterator #11

Merged
roysc merged 27 commits from symmetric-diff-iterator into main 2023-09-20 03:22:19 +00:00
Conversation 2 Commits 27 Files Changed 26
+865 -702
roysc commented 2023-07-23 17:11:52 +00:00
Owner
Copy Link
Copy Source

Implements https://github.com/cerc-io/go-ethereum/issues/319

With this we can perform a single pass to process updates. This lets us simplify the builder code somewhat, and I've done some additional refactoring for clarity.

Implements https://github.com/cerc-io/go-ethereum/issues/319 With this we can perform a single pass to process updates. This lets us simplify the builder code somewhat, and I've done some additional refactoring for clarity.
roysc added 5 commits 2023-07-23 17:11:59 +00:00
update deps b49c61d9e4
update plugeth 2907882c1a
Builder clean up 6b1f676c60 
- Remove unused bits
- simplify and normalize test data
- Renames for clarity
Implement symmetric difference iterator b9d988e0ec
[wip] use symmetric difference iterator
Test / Run unit tests (pull_request) Failing after 5s
Details
Test / Run integration tests (pull_request) Failing after 8s
Details
79178e0f29
roysc force-pushed symmetric-diff-iterator from 79178e0f29 to 6de2326666 2023-07-23 17:16:04 +00:00 Compare
roysc force-pushed symmetric-diff-iterator from 6de2326666 to 663e762977 2023-07-26 04:20:22 +00:00 Compare
roysc force-pushed symmetric-diff-iterator from 663e762977 to 06723ee4e0 2023-07-26 04:24:11 +00:00 Compare
roysc added 1 commit 2023-08-03 10:34:55 +00:00
iterator fixes
Test / Run unit tests (pull_request) Failing after 10s
Details
Test / Run integration tests (pull_request) Failing after 10s
Details
99ac6bb9ae
roysc added 2 commits 2023-08-03 10:38:58 +00:00
builder test 84def829e6
builder cleanup
Test / Run unit tests (pull_request) Failing after 7s
Details
Test / Run integration tests (pull_request) Failing after 8s
Details
a11d68c38d
roysc marked the pull request as ready for review 2023-08-03 10:49:03 +00:00
roysc requested review from i-norden 2023-08-03 10:49:14 +00:00
roysc requested review from telackey 2023-08-03 10:49:27 +00:00
roysc force-pushed symmetric-diff-iterator from a11d68c38d to bd19a9cfa8 2023-08-03 14:53:15 +00:00 Compare
roysc force-pushed symmetric-diff-iterator from bd19a9cfa8 to 23204cb810 2023-08-05 09:25:17 +00:00 Compare
telackey left a comment
Contributor
Copy Link
Copy Source

The tests are executing in CI now, so taking for granted that those test/build failures will be fixed...

I had a question or two about metrics. It is somewhat difficult to follow the diff. Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)?

The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results.

If that were completely impossible within the context of an in-process test (eg, because too much of relevant indexer code also changed), then an alternative idea would be run the data through an older version of the plugin, using the old iterator scheme, and then through the new one, and compare the complete statediff output.

There is an opportunity for performance comparison here as well.

Thoughts?

The tests are executing in CI now, so taking for granted that those test/build failures will be fixed... I had a question or two about metrics. It is somewhat difficult to follow the diff. Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)? The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results. If that were completely impossible within the context of an in-process test (eg, because too much of relevant indexer code also changed), then an alternative idea would be run the data through an older version of the plugin, using the old iterator scheme, and then through the new one, and compare the complete statediff output. There is an opportunity for performance comparison here as well. Thoughts?
indexer/database/metrics/metrics.go Outdated
@@ -144,28 +120,15 @@ func RegisterIndexerMetrics(reg metrics.Registry) IndexerMetricsHandles {
reg.Register(metricName(subsys, "t_uncle_processing"), ctx.UncleProcessingTimer)
reg.Register(metricName(subsys, "t_tx_receipt_processing"), ctx.TxAndRecProcessingTimer)
reg.Register(metricName(subsys, "t_state_store_code_processing"), ctx.StateStoreCodeProcessingTimer)
reg.Register(metricName(subsys, "t_build_statediff"), ctx.BuildStateDiffTimer)
telackey commented 2023-09-05 18:53:00 +00:00
Contributor
Copy Link
Copy Source

Are all of these timers irrelevant with the new iterator?

Are all of these timers irrelevant with the new iterator?
roysc commented 2023-09-06 13:25:47 +00:00
Author
Owner
Copy Link
Copy Source

Some were not actually used anywhere - ResolveNodeTimer, DifferenceIteratorNextTimer (AccessListEntriesCounter is also unused)

The rest are either renamed or deleted along with the functions they tracked.

Removed include:

  • BuildRemovedStorageNodesFromTrieTimer
  • SortKeysTimer
  • FindIntersectionTimer
  • BuildStateDiffObjectTimer
Some were not actually used anywhere - ResolveNodeTimer, DifferenceIteratorNextTimer (AccessListEntriesCounter is also unused) The rest are either renamed or deleted along with the functions they tracked. Removed include: - BuildRemovedStorageNodesFromTrieTimer - SortKeysTimer - FindIntersectionTimer - BuildStateDiffObjectTimer
roysc marked this conversation as resolved
utils/iterator.go Outdated
@@ -2,0 +156,4 @@
}
func compareNodes(a, b trie.NodeIterator) int {
// if cmp := bytes.Compare(a.Path(), b.Path()); cmp != 0 {
telackey commented 2023-09-05 18:49:55 +00:00
Contributor
Copy Link
Copy Source

Remove dead code?

Remove dead code?
roysc marked this conversation as resolved
roysc commented 2023-09-06 13:20:49 +00:00
Author
Owner
Copy Link
Copy Source

The build failures are fixed by #10, so I'll rebase.

Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)?

All the same data is there, just renamed or removed where appropriate, along with their functions.

The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results.

That's a good idea, I'm adding a test which traverses a sample of mainnet data and compares all the results.

Comparing the full set of (inner & leaf) nodes in-process wouldn't be too complicated, but would still mean comparing the old code with the new, which would rightly need to be done in a separate repo running them against each other. That's something I already want to do for benchmarking, though. To be thorough, we can wait on merging this until that's in place.

The build failures are fixed by https://git.vdb.to/cerc-io/plugeth-statediff/pulls/10, so I'll rebase. >Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)? All the same data is there, just renamed or removed where appropriate, along with their functions. > The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results. That's a good idea, I'm adding a test which traverses a sample of mainnet data and compares all the results. Comparing the full set of (inner & leaf) nodes in-process wouldn't be too complicated, but would still mean comparing the old code with the new, which would rightly need to be done in a separate repo running them against each other. That's something I already want to do for benchmarking, though. To be thorough, we can wait on merging this until that's in place.
roysc force-pushed symmetric-diff-iterator from 23204cb810 to ae33dce5af 2023-09-06 13:31:30 +00:00 Compare
roysc added 3 commits 2023-09-09 10:14:09 +00:00
nits 480b38cb8a 
error msg

note

update make docker-image
get fixture data from test data repo 9c4e00cd69 
rename eth-testing module
utils: load config
Test / Run unit tests (pull_request) Failing after 7m13s
Details
Test / Run integration tests (pull_request) Failing after 38m19s
Details
b7c3789e3c
roysc force-pushed symmetric-diff-iterator from b7c3789e3c to e43cbb30f9 2023-09-10 10:22:04 +00:00 Compare
roysc added 2 commits 2023-09-14 12:10:57 +00:00
fix service init 2e496f3af2
misc
Test / Run unit tests (pull_request) Failing after 7m4s
Details
Test / Run integration tests (pull_request) Failing after 7m5s
Details
80c3540ddb
roysc added 1 commit 2023-09-15 04:02:35 +00:00
system tests has no default branch
Test / Run unit tests (pull_request) Failing after 6m14s
Details
Test / Run integration tests (pull_request) Failing after 8m1s
Details
5ab353605c
roysc added 2 commits 2023-09-19 03:29:08 +00:00
CI fixes 47ebfa205c 
install SO via curl

install py

setup go v4
run unit tests non parallel
Test / Run unit tests (pull_request) Successful in 17m0s
Details
Test / Run integration tests (pull_request) Successful in 35m55s
Details
fc54269146
roysc added 1 commit 2023-09-19 10:09:03 +00:00
fn name
Test / Run unit tests (pull_request) Successful in 17m52s
Details
Test / Run integration tests (pull_request) Successful in 35m2s
Details
2446393db4
roysc referenced this issue from a commit 2023-09-19 12:26:20 +00:00
Add test data & fixes (#1)
roysc added 2 commits 2023-09-19 18:18:11 +00:00
update deps c54b3b0e39
use setup-python
Test / Run unit tests (pull_request) Successful in 18m58s
Details
Test / Run integration tests (pull_request) Failing after 39m25s
Details
8df61cada7
roysc force-pushed symmetric-diff-iterator from 8df61cada7 to 7d223d9c0b 2023-09-19 19:02:54 +00:00 Compare
roysc merged commit 981bfb5895 into main 2023-09-20 03:22:19 +00:00
roysc deleted branch symmetric-diff-iterator 2023-09-20 03:22:21 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
i-norden
telackey
Labels
Clear labels
Copied from Github
An issue or PR manually copied from GitHub.
 Kind/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
AFDudley Alex cerccicd ericlewis i-norden (Ian Norden) rachmaninovquar rjk-laconic roysc (Roy Crihfield) srw zramsay
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cerc-io/plugeth-statediff#11