Commit Graph

5 Commits

Author SHA1 Message Date
Ian Norden
05aeeab581
Account selective snapshot (#46)
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided

* config and env updates

* cmd update

* Encode watched address path bytes to hex for comparison

* actually ignore the subtries that are not along the paths of interest

* Fixes for account selective snapshot

* Use non-concurrent iterator when having a single worker

* Only index root node when starting path of an iterator is nil

* Upgrade deps

* Avoid tracking iterators and skip recovery test

* Fix recovery mechanism, use sync Map instead of buffered channels

* Add test for account selective snapshot

* Continue traversal with concurrent iterators with starting path nil

* Use errgroup to simplify error handling with concurrent iterators

* Check if all the nodes are indexed in the recovery test

* Use concurrency safe sync Map in account selective snapshot test

* Only track concurrent iterators and refactor code

* Fix node and recovered path comparison

* Revert back to using buffered channels for tracking iterators

* Add a metric to monitor number of active iterators

* Update docs

* Update seeked path after node is processed

* Return error on context cancellation from subtrie iteration

* Add tests for account selective snapshot recovery

* Explicity enforce concurrent iterator bounds to avoid duplicate nodes

* Update full snapshot test to check nodes being indexed

* Refactor code to simplify snapshot logic

* Remove unnecessary function argument

* Use ctx cancellation for handling signals

* Add descriptive comments

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 17:05:04 +05:30
prathamesh0
b241bf05eb
Add command to take an in-place snapshot (#42)
* Add command to take an in-place snapshot

* Add test data for in place snapshot unit test

* Implement unit test for inplace snapshot

* Add check for storage IPLD

* Run unit tests sequentially

* Add github workflow for unit test

* Add missing checks for state and storage cid fields

* Add more storage nodes to test

* Update ipld-eth-db version for tests

* Add comments for inplace snapshot test data

* Add in-place snapshot cmd in readme

* Implement defer pattern for db transaction

* Log transaction commit or rollback error

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-06-15 12:51:26 +05:30
Ashwin Phatak
7129044eae Log progress info 2022-06-08 18:32:10 +08:00
i-norden
060eb5cf67 add some logs and guards, update module name, update readme 2022-03-30 18:57:30 -05:00
Ian Norden
d903cdbaa4 gitignore 2020-06-30 11:57:28 -05:00