Use symmetric difference iterator #11
Labels
No Label
Copied from Github
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: cerc-io/plugeth-statediff#11
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "symmetric-diff-iterator"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implements https://github.com/cerc-io/go-ethereum/issues/319
With this we can perform a single pass to process updates. This lets us simplify the builder code somewhat, and I've done some additional refactoring for clarity.
79178e0f29
to6de2326666
6de2326666
to663e762977
663e762977
to06723ee4e0
WIP: Use symmetric difference iteratorto Use symmetric difference iteratora11d68c38d
tobd19a9cfa8
bd19a9cfa8
to23204cb810
The tests are executing in CI now, so taking for granted that those test/build failures will be fixed...
I had a question or two about metrics. It is somewhat difficult to follow the diff. Are all the fine-grained counters maintained (or equivalents added appropriate to the new strategy)?
The second question is about testing. Given the complexity and centrality of the iterator, it would be excellent if we had a test which ran the same (ideally real) data through both the trie.NewDifferenceIterator and utils.NewSymmetricDifferenceIterator and compared the results.
If that were completely impossible within the context of an in-process test (eg, because too much of relevant indexer code also changed), then an alternative idea would be run the data through an older version of the plugin, using the old iterator scheme, and then through the new one, and compare the complete statediff output.
There is an opportunity for performance comparison here as well.
Thoughts?
@ -144,28 +120,15 @@ func RegisterIndexerMetrics(reg metrics.Registry) IndexerMetricsHandles {
reg.Register(metricName(subsys, "t_uncle_processing"), ctx.UncleProcessingTimer)
reg.Register(metricName(subsys, "t_tx_receipt_processing"), ctx.TxAndRecProcessingTimer)
reg.Register(metricName(subsys, "t_state_store_code_processing"), ctx.StateStoreCodeProcessingTimer)
reg.Register(metricName(subsys, "t_build_statediff"), ctx.BuildStateDiffTimer)
Are all of these timers irrelevant with the new iterator?
Some were not actually used anywhere - ResolveNodeTimer, DifferenceIteratorNextTimer (AccessListEntriesCounter is also unused)
The rest are either renamed or deleted along with the functions they tracked.
Removed include:
@ -2,0 +156,4 @@
}
func compareNodes(a, b trie.NodeIterator) int {
// if cmp := bytes.Compare(a.Path(), b.Path()); cmp != 0 {
Remove dead code?
The build failures are fixed by #10, so I'll rebase.
All the same data is there, just renamed or removed where appropriate, along with their functions.
That's a good idea, I'm adding a test which traverses a sample of mainnet data and compares all the results.
Comparing the full set of (inner & leaf) nodes in-process wouldn't be too complicated, but would still mean comparing the old code with the new, which would rightly need to be done in a separate repo running them against each other. That's something I already want to do for benchmarking, though. To be thorough, we can wait on merging this until that's in place.
23204cb810
toae33dce5af
b7c3789e3c
toe43cbb30f9
8df61cada7
to7d223d9c0b