Use symmetric difference iterator #11

Merged
roysc merged 27 commits from symmetric-diff-iterator into main 2023-09-20 03:22:19 +00:00
Member

Implements https://github.com/cerc-io/go-ethereum/issues/319

With this we can perform a single pass to process updates. This lets us simplify the builder code somewhat, and I've done some additional refactoring for clarity.

Implements https://github.com/cerc-io/go-ethereum/issues/319 With this we can perform a single pass to process updates. This lets us simplify the builder code somewhat, and I've done some additional refactoring for clarity.
roysc added 5 commits 2023-07-23 17:11:59 +00:00
- Remove unused bits
- simplify and normalize test data
- Renames for clarity
[wip] use symmetric difference iterator
Some checks failed
Test / Run unit tests (pull_request) Failing after 5s
Test / Run integration tests (pull_request) Failing after 8s
79178e0f29
roysc force-pushed symmetric-diff-iterator from 79178e0f29 to 6de2326666 2023-07-23 17:16:04 +00:00 Compare
roysc force-pushed symmetric-diff-iterator from 6de2326666 to 663e762977 2023-07-26 04:20:22 +00:00 Compare
roysc force-pushed symmetric-diff-iterator from 663e762977 to 06723ee4e0 2023-07-26 04:24:11 +00:00 Compare
roysc added 1 commit 2023-08-03 10:34:55 +00:00
iterator fixes
Some checks failed
Test / Run unit tests (pull_request) Failing after 10s
Test / Run integration tests (pull_request) Failing after 10s
99ac6bb9ae
roysc added 2 commits 2023-08-03 10:38:58 +00:00
builder cleanup
Some checks failed
Test / Run unit tests (pull_request) Failing after 7s
Test / Run integration tests (pull_request) Failing after 8s
a11d68c38d
roysc changed title from WIP: Use symmetric difference iterator to Use symmetric difference iterator 2023-08-03 10:49:03 +00:00
roysc requested review from i-norden 2023-08-03 10:49:14 +00:00
roysc requested review from telackey 2023-08-03 10:49:27 +00:00
roysc force-pushed symmetric-diff-iterator from a11d68c38d to bd19a9cfa8 2023-08-03 14:53:15 +00:00 Compare
roysc force-pushed symmetric-diff-iterator from bd19a9cfa8 to 23204cb810 2023-08-05 09:25:17 +00:00 Compare
telackey requested changes 2023-09-05 19:12:29 +00:00
telackey left a comment
Member

The tests are executing in CI now, so taking for granted that those test/build failures will be fixed...

I had a question or two about metrics. It is somewhat difficult to follow the diff. Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)?

The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results.

If that were completely impossible within the context of an in-process test (eg, because too much of relevant indexer code also changed), then an alternative idea would be run the data through an older version of the plugin, using the old iterator scheme, and then through the new one, and compare the complete statediff output.

There is an opportunity for performance comparison here as well.

Thoughts?

The tests are executing in CI now, so taking for granted that those test/build failures will be fixed... I had a question or two about metrics. It is somewhat difficult to follow the diff. Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)? The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results. If that were completely impossible within the context of an in-process test (eg, because too much of relevant indexer code also changed), then an alternative idea would be run the data through an older version of the plugin, using the old iterator scheme, and then through the new one, and compare the complete statediff output. There is an opportunity for performance comparison here as well. Thoughts?
@ -144,28 +120,15 @@ func RegisterIndexerMetrics(reg metrics.Registry) IndexerMetricsHandles {
reg.Register(metricName(subsys, "t_uncle_processing"), ctx.UncleProcessingTimer)
reg.Register(metricName(subsys, "t_tx_receipt_processing"), ctx.TxAndRecProcessingTimer)
reg.Register(metricName(subsys, "t_state_store_code_processing"), ctx.StateStoreCodeProcessingTimer)
reg.Register(metricName(subsys, "t_build_statediff"), ctx.BuildStateDiffTimer)
Member

Are all of these timers irrelevant with the new iterator?

Are all of these timers irrelevant with the new iterator?
Author
Member

Some were not actually used anywhere - ResolveNodeTimer, DifferenceIteratorNextTimer (AccessListEntriesCounter is also unused)

The rest are either renamed or deleted along with the functions they tracked.

Removed include:

  • BuildRemovedStorageNodesFromTrieTimer
  • SortKeysTimer
  • FindIntersectionTimer
  • BuildStateDiffObjectTimer
Some were not actually used anywhere - ResolveNodeTimer, DifferenceIteratorNextTimer (AccessListEntriesCounter is also unused) The rest are either renamed or deleted along with the functions they tracked. Removed include: - BuildRemovedStorageNodesFromTrieTimer - SortKeysTimer - FindIntersectionTimer - BuildStateDiffObjectTimer
roysc marked this conversation as resolved
@ -2,0 +156,4 @@
}
func compareNodes(a, b trie.NodeIterator) int {
// if cmp := bytes.Compare(a.Path(), b.Path()); cmp != 0 {
Member

Remove dead code?

Remove dead code?
roysc marked this conversation as resolved
Author
Member

The build failures are fixed by #10, so I'll rebase.

Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)?

All the same data is there, just renamed or removed where appropriate, along with their functions.

The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results.

That's a good idea, I'm adding a test which traverses a sample of mainnet data and compares all the results.

Comparing the full set of (inner & leaf) nodes in-process wouldn't be too complicated, but would still mean comparing the old code with the new, which would rightly need to be done in a separate repo running them against each other. That's something I already want to do for benchmarking, though. To be thorough, we can wait on merging this until that's in place.

The build failures are fixed by https://git.vdb.to/cerc-io/plugeth-statediff/pulls/10, so I'll rebase. >Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)? All the same data is there, just renamed or removed where appropriate, along with their functions. > The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results. That's a good idea, I'm adding a test which traverses a sample of mainnet data and compares all the results. Comparing the full set of (inner & leaf) nodes in-process wouldn't be too complicated, but would still mean comparing the old code with the new, which would rightly need to be done in a separate repo running them against each other. That's something I already want to do for benchmarking, though. To be thorough, we can wait on merging this until that's in place.
roysc force-pushed symmetric-diff-iterator from 23204cb810 to ae33dce5af 2023-09-06 13:31:30 +00:00 Compare
roysc added 3 commits 2023-09-09 10:14:09 +00:00
error msg

note

update make docker-image
rename eth-testing module
utils: load config
Some checks failed
Test / Run unit tests (pull_request) Failing after 7m13s
Test / Run integration tests (pull_request) Failing after 38m19s
b7c3789e3c
roysc force-pushed symmetric-diff-iterator from b7c3789e3c to e43cbb30f9 2023-09-10 10:22:04 +00:00 Compare
roysc added 2 commits 2023-09-14 12:10:57 +00:00
misc
Some checks failed
Test / Run unit tests (pull_request) Failing after 7m4s
Test / Run integration tests (pull_request) Failing after 7m5s
80c3540ddb
roysc added 1 commit 2023-09-15 04:02:35 +00:00
system tests has no default branch
Some checks failed
Test / Run unit tests (pull_request) Failing after 6m14s
Test / Run integration tests (pull_request) Failing after 8m1s
5ab353605c
roysc added 2 commits 2023-09-19 03:29:08 +00:00
install SO via curl

install py

setup go v4
run unit tests non parallel
All checks were successful
Test / Run unit tests (pull_request) Successful in 17m0s
Test / Run integration tests (pull_request) Successful in 35m55s
fc54269146
roysc added 1 commit 2023-09-19 10:09:03 +00:00
fn name
All checks were successful
Test / Run unit tests (pull_request) Successful in 17m52s
Test / Run integration tests (pull_request) Successful in 35m2s
2446393db4
roysc referenced this issue from a commit 2023-09-19 12:26:20 +00:00
roysc added 2 commits 2023-09-19 18:18:11 +00:00
use setup-python
Some checks failed
Test / Run unit tests (pull_request) Successful in 18m58s
Test / Run integration tests (pull_request) Failing after 39m25s
8df61cada7
roysc force-pushed symmetric-diff-iterator from 8df61cada7 to 7d223d9c0b 2023-09-19 19:02:54 +00:00 Compare
roysc merged commit 981bfb5895 into main 2023-09-20 03:22:19 +00:00
roysc deleted branch symmetric-diff-iterator 2023-09-20 03:22:21 +00:00
Sign in to join this conversation.
No reviewers
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/plugeth-statediff#11
No description provided.