* Add a progress counter by checking the distance already traversed from the startPath to endPath in a bounded iterator vs the estimated number of iterations.
* cerc refactor waiting on unpublished dependencies
* more cerc_refactor with published ipld-eth-db unstable
* TearDownDB refactor
* missed second publisher_test TearDownDB
* Add a script to find bad data in CSV file dumps
* Add a script to delete bad rows from CSV file dumps
* Add instructions to run the scripts
* Reorganize instructions
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided
* config and env updates
* cmd update
* Encode watched address path bytes to hex for comparison
* actually ignore the subtries that are not along the paths of interest
* Fixes for account selective snapshot
* Use non-concurrent iterator when having a single worker
* Only index root node when starting path of an iterator is nil
* Upgrade deps
* Avoid tracking iterators and skip recovery test
* Fix recovery mechanism, use sync Map instead of buffered channels
* Add test for account selective snapshot
* Continue traversal with concurrent iterators with starting path nil
* Use errgroup to simplify error handling with concurrent iterators
* Check if all the nodes are indexed in the recovery test
* Use concurrency safe sync Map in account selective snapshot test
* Only track concurrent iterators and refactor code
* Fix node and recovered path comparison
* Revert back to using buffered channels for tracking iterators
* Add a metric to monitor number of active iterators
* Update docs
* Update seeked path after node is processed
* Return error on context cancellation from subtrie iteration
* Add tests for account selective snapshot recovery
* Explicity enforce concurrent iterator bounds to avoid duplicate nodes
* Update full snapshot test to check nodes being indexed
* Refactor code to simplify snapshot logic
* Remove unnecessary function argument
* Use ctx cancellation for handling signals
* Add descriptive comments
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
* Add instructions to import snapshot data into database
* Add monitoring and update data processing in README
* Update instructions to import snapshot
* Add command to take an in-place snapshot
* Add test data for in place snapshot unit test
* Implement unit test for inplace snapshot
* Add check for storage IPLD
* Run unit tests sequentially
* Add github workflow for unit test
* Add missing checks for state and storage cid fields
* Add more storage nodes to test
* Update ipld-eth-db version for tests
* Add comments for inplace snapshot test data
* Add in-place snapshot cmd in readme
* Implement defer pattern for db transaction
* Log transaction commit or rollback error
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
telackey
Ian Norden
roysc
Michael
prathamesh0
Ashwin Phatak