Commit Graph

6 Commits

Author SHA1 Message Date
b4367dff3b 6: Restore previous scheme for estimating progress. (#7)
```
ipld_eth_state_snapshot_stats_tracked_iterator_1 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_10 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_11 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_12 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_13 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_14 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_15 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_16 0
ipld_eth_state_snapshot_stats_tracked_iterator_2 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_3 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_4 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_5 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_6 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_7 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_8 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_9 37.5
```

Reviewed-on: #7
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2023-10-06 23:20:05 +00:00
00141776bf Refactor to use statediff plugin (#1)
* Refactors to replace most of the code with the statediff plugin.
* Adds basic CI test workflows for Gitea
* Refactors fixtures to use https://git.vdb.to/cerc-io/eth-testing
* Renames env vars for consistency with flags and other services:
  - LOGRUS_{LEVEL,FILE} => LOG_LEVEL, etc.
  - LVL_DB_PATH => LEVELDB_PATH
  - ANCIENT_DB_PATH => LEVELDB_ANCIENT
  - These will need to be updated wherever they are used

Reviewed-on: #1
2023-09-29 18:43:26 +00:00
768357293c
Add progress counter to prometheus output (#76)
* Add a progress counter by checking the distance already traversed from the startPath to endPath in a bounded iterator vs the estimated number of iterations.
2023-05-23 11:23:58 -05:00
i-norden
5c13d59515 refactor to work with v4 vdb geth v1.11.5 2023-03-31 10:39:27 -05:00
Ian Norden
05aeeab581
Account selective snapshot (#46)
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided

* config and env updates

* cmd update

* Encode watched address path bytes to hex for comparison

* actually ignore the subtries that are not along the paths of interest

* Fixes for account selective snapshot

* Use non-concurrent iterator when having a single worker

* Only index root node when starting path of an iterator is nil

* Upgrade deps

* Avoid tracking iterators and skip recovery test

* Fix recovery mechanism, use sync Map instead of buffered channels

* Add test for account selective snapshot

* Continue traversal with concurrent iterators with starting path nil

* Use errgroup to simplify error handling with concurrent iterators

* Check if all the nodes are indexed in the recovery test

* Use concurrency safe sync Map in account selective snapshot test

* Only track concurrent iterators and refactor code

* Fix node and recovered path comparison

* Revert back to using buffered channels for tracking iterators

* Add a metric to monitor number of active iterators

* Update docs

* Update seeked path after node is processed

* Return error on context cancellation from subtrie iteration

* Add tests for account selective snapshot recovery

* Explicity enforce concurrent iterator bounds to avoid duplicate nodes

* Update full snapshot test to check nodes being indexed

* Refactor code to simplify snapshot logic

* Remove unnecessary function argument

* Use ctx cancellation for handling signals

* Add descriptive comments

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 17:05:04 +05:30
prathamesh0
cce45bda57
Add prometheus metrics collection (#33)
* Upgrade geth

* Add prometheus metrics collection

* Update README
2022-05-23 16:56:48 +05:30