cerc-io/plugeth-statediff
15
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Use symmetric difference iterator #11

Merged
roysc merged 27 commits from symmetric-diff-iterator into main 2023-09-20 03:22:19 +00:00
Conversation 1 Commits 27 Files Changed 26 +865 -702
roysc commented 2023-07-23 17:11:52 +00:00
Member
Copy Link

Implements https://github.com/cerc-io/go-ethereum/issues/319

With this we can perform a single pass to process updates. This lets us simplify the builder code somewhat, and I've done some additional refactoring for clarity.

Implements https://github.com/cerc-io/go-ethereum/issues/319 With this we can perform a single pass to process updates. This lets us simplify the builder code somewhat, and I've done some additional refactoring for clarity.
roysc added 5 commits 2023-07-23 17:11:59 +00:00 
- Remove unused bits
- simplify and normalize test data
- Renames for clarity
[wip] use symmetric difference iterator
Some checks failed
Test / Run unit tests (pull_request) Failing after 5s
Details
Test / Run integration tests (pull_request) Failing after 8s
Details
79178e0f29
roysc force-pushed symmetric-diff-iterator from 79178e0f29 to 6de2326666 2023-07-23 17:16:04 +00:00 Compare
roysc force-pushed symmetric-diff-iterator from 6de2326666 to 663e762977 2023-07-26 04:20:22 +00:00 Compare
roysc force-pushed symmetric-diff-iterator from 663e762977 to 06723ee4e0 2023-07-26 04:24:11 +00:00 Compare
roysc added 1 commit 2023-08-03 10:34:55 +00:00
iterator fixes
Some checks failed
Test / Run unit tests (pull_request) Failing after 10s
Details
Test / Run integration tests (pull_request) Failing after 10s
Details
99ac6bb9ae
roysc added 2 commits 2023-08-03 10:38:58 +00:00
builder cleanup
Some checks failed
Test / Run unit tests (pull_request) Failing after 7s
Details
Test / Run integration tests (pull_request) Failing after 8s
Details
a11d68c38d
roysc changed title from WIP: Use symmetric difference iterator to Use symmetric difference iterator 2023-08-03 10:49:03 +00:00
roysc requested review from i-norden 2023-08-03 10:49:14 +00:00
roysc requested review from telackey 2023-08-03 10:49:27 +00:00
roysc force-pushed symmetric-diff-iterator from a11d68c38d to bd19a9cfa8 2023-08-03 14:53:15 +00:00 Compare
roysc force-pushed symmetric-diff-iterator from bd19a9cfa8 to 23204cb810 2023-08-05 09:25:17 +00:00 Compare
telackey requested changes 2023-09-05 19:12:29 +00:00
telackey left a comment
Member
Copy Link

The tests are executing in CI now, so taking for granted that those test/build failures will be fixed...

I had a question or two about metrics. It is somewhat difficult to follow the diff. Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)?

The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results.

If that were completely impossible within the context of an in-process test (eg, because too much of relevant indexer code also changed), then an alternative idea would be run the data through an older version of the plugin, using the old iterator scheme, and then through the new one, and compare the complete statediff output.

There is an opportunity for performance comparison here as well.

Thoughts?

The tests are executing in CI now, so taking for granted that those test/build failures will be fixed... I had a question or two about metrics. It is somewhat difficult to follow the diff. Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)? The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results. If that were completely impossible within the context of an in-process test (eg, because too much of relevant indexer code also changed), then an alternative idea would be run the data through an older version of the plugin, using the old iterator scheme, and then through the new one, and compare the complete statediff output. There is an opportunity for performance comparison here as well. Thoughts?
indexer/database/metrics/metrics.go Outdated
@ -144,28 +120,15 @@ func RegisterIndexerMetrics(reg metrics.Registry) IndexerMetricsHandles {
reg.Register(metricName(subsys, "t_uncle_processing"), ctx.UncleProcessingTimer)
reg.Register(metricName(subsys, "t_tx_receipt_processing"), ctx.TxAndRecProcessingTimer)
reg.Register(metricName(subsys, "t_state_store_code_processing"), ctx.StateStoreCodeProcessingTimer)
reg.Register(metricName(subsys, "t_build_statediff"), ctx.BuildStateDiffTimer)
telackey commented 2023-09-05 18:53:00 +00:00
Member
Copy Link

Are all of these timers irrelevant with the new iterator?

Are all of these timers irrelevant with the new iterator?
roysc commented 2023-09-06 13:25:47 +00:00
Author
Member
Copy Link

Some were not actually used anywhere - ResolveNodeTimer, DifferenceIteratorNextTimer (AccessListEntriesCounter is also unused)

The rest are either renamed or deleted along with the functions they tracked.

Removed include:

  • BuildRemovedStorageNodesFromTrieTimer
  • SortKeysTimer
  • FindIntersectionTimer
  • BuildStateDiffObjectTimer
Some were not actually used anywhere - ResolveNodeTimer, DifferenceIteratorNextTimer (AccessListEntriesCounter is also unused) The rest are either renamed or deleted along with the functions they tracked. Removed include: - BuildRemovedStorageNodesFromTrieTimer - SortKeysTimer - FindIntersectionTimer - BuildStateDiffObjectTimer
roysc marked this conversation as resolved
utils/iterator.go Outdated
@ -2,0 +156,4 @@
}
func compareNodes(a, b trie.NodeIterator) int {
// if cmp := bytes.Compare(a.Path(), b.Path()); cmp != 0 {
telackey commented 2023-09-05 18:49:55 +00:00
Member
Copy Link

Remove dead code?

Remove dead code?
roysc marked this conversation as resolved
roysc commented 2023-09-06 13:20:49 +00:00
Author
Member
Copy Link

The build failures are fixed by #10, so I'll rebase.

Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)?

All the same data is there, just renamed or removed where appropriate, along with their functions.

The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results.

That's a good idea, I'm adding a test which traverses a sample of mainnet data and compares all the results.

Comparing the full set of (inner & leaf) nodes in-process wouldn't be too complicated, but would still mean comparing the old code with the new, which would rightly need to be done in a separate repo running them against each other. That's something I already want to do for benchmarking, though. To be thorough, we can wait on merging this until that's in place.

The build failures are fixed by https://git.vdb.to/cerc-io/plugeth-statediff/pulls/10, so I'll rebase. >Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)? All the same data is there, just renamed or removed where appropriate, along with their functions. > The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results. That's a good idea, I'm adding a test which traverses a sample of mainnet data and compares all the results. Comparing the full set of (inner & leaf) nodes in-process wouldn't be too complicated, but would still mean comparing the old code with the new, which would rightly need to be done in a separate repo running them against each other. That's something I already want to do for benchmarking, though. To be thorough, we can wait on merging this until that's in place.
roysc force-pushed symmetric-diff-iterator from 23204cb810 to ae33dce5af 2023-09-06 13:31:30 +00:00 Compare
roysc added 3 commits 2023-09-09 10:14:09 +00:00 
error msg

note

update make docker-image
rename eth-testing module
utils: load config
Some checks failed
Test / Run unit tests (pull_request) Failing after 7m13s
Details
Test / Run integration tests (pull_request) Failing after 38m19s
Details
b7c3789e3c
roysc force-pushed symmetric-diff-iterator from b7c3789e3c to e43cbb30f9 2023-09-10 10:22:04 +00:00 Compare
roysc added 2 commits 2023-09-14 12:10:57 +00:00
misc
Some checks failed
Test / Run unit tests (pull_request) Failing after 7m4s
Details
Test / Run integration tests (pull_request) Failing after 7m5s
Details
80c3540ddb
roysc added 1 commit 2023-09-15 04:02:35 +00:00
system tests has no default branch
Some checks failed
Test / Run unit tests (pull_request) Failing after 6m14s
Details
Test / Run integration tests (pull_request) Failing after 8m1s
Details
5ab353605c
roysc added 2 commits 2023-09-19 03:29:08 +00:00 
install SO via curl

install py

setup go v4
run unit tests non parallel
All checks were successful
Test / Run unit tests (pull_request) Successful in 17m0s
Details
Test / Run integration tests (pull_request) Successful in 35m55s
Details
fc54269146
roysc added 1 commit 2023-09-19 10:09:03 +00:00
fn name
All checks were successful
Test / Run unit tests (pull_request) Successful in 17m52s
Details
Test / Run integration tests (pull_request) Successful in 35m2s
Details
2446393db4
roysc referenced this issue from a commit 2023-09-19 12:26:20 +00:00
Add test data & fixes (#1)
roysc added 2 commits 2023-09-19 18:18:11 +00:00
use setup-python
Some checks failed
Test / Run unit tests (pull_request) Successful in 18m58s
Details
Test / Run integration tests (pull_request) Failing after 39m25s
Details
8df61cada7
roysc force-pushed symmetric-diff-iterator from 8df61cada7 to 7d223d9c0b 2023-09-19 19:02:54 +00:00 Compare
roysc merged commit 981bfb5895 into main 2023-09-20 03:22:19 +00:00
roysc deleted branch symmetric-diff-iterator 2023-09-20 03:22:21 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
i-norden
telackey
Labels
Clear labels
  
Copied from Github

An issue or PR manually copied from GitHub.

  
Kind/Breaking

Breaking change that won't be backward compatible

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Security

This is security issue

  
Kind/Testing

Issue or pull request related to testing

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

No Label
Copied from Github
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/plugeth-statediff#11
No description provided.
Delete Branch "symmetric-diff-iterator"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?