Commit Graph

162 Commits

Author SHA1 Message Date
e67c7f030e Missing ENV bindings.
All checks were successful
Test / Run unit and integration tests (pull_request) Successful in 4m34s
Test / Run compliance tests (pull_request) Successful in 4m58s
2023-10-09 12:57:29 -05:00
b4367dff3b 6: Restore previous scheme for estimating progress. (#7)
```
ipld_eth_state_snapshot_stats_tracked_iterator_1 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_10 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_11 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_12 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_13 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_14 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_15 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_16 0
ipld_eth_state_snapshot_stats_tracked_iterator_2 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_3 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_4 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_5 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_6 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_7 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_8 37.5
ipld_eth_state_snapshot_stats_tracked_iterator_9 37.5
```

Reviewed-on: #7
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2023-10-06 23:20:05 +00:00
00141776bf Refactor to use statediff plugin (#1)
* Refactors to replace most of the code with the statediff plugin.
* Adds basic CI test workflows for Gitea
* Refactors fixtures to use https://git.vdb.to/cerc-io/eth-testing
* Renames env vars for consistency with flags and other services:
  - LOGRUS_{LEVEL,FILE} => LOG_LEVEL, etc.
  - LVL_DB_PATH => LEVELDB_PATH
  - ANCIENT_DB_PATH => LEVELDB_ANCIENT
  - These will need to be updated wherever they are used

Reviewed-on: #1
2023-09-29 18:43:26 +00:00
4e0b481ea5
Update to geth v1.11.6-statediff-5.0.8 (#79)
* Update to geth v1.11.6-statediff-5.0.8
2023-07-21 19:05:01 -05:00
Ian Norden
da02e5ac12
Merge pull request #77 from cerc-io/ian/v5_dev
fix: err overshadowing in defers
2023-06-01 08:23:02 -05:00
i-norden
fe88e90181 avoid overshadowing err in defers 2023-05-31 18:08:02 -05:00
768357293c
Add progress counter to prometheus output (#76)
* Add a progress counter by checking the distance already traversed from the startPath to endPath in a bounded iterator vs the estimated number of iterations.
2023-05-23 11:23:58 -05:00
Ian Norden
dd86f02997
Merge pull request #75 from cerc-io/ian/v5_dev
fix overshadowing of snap.Tx
2023-05-18 09:37:26 -05:00
i-norden
891fca89bd fix overshadowing of snap.Tx 2023-05-17 08:16:50 -05:00
Ian Norden
0cab03b98e
Merge pull request #74 from cerc-io/ian/v5_dev
update geth and ipld-eth-db deps
2023-05-15 11:38:24 -05:00
i-norden
81e286399b update geth and ipld-eth-db deps 2023-05-15 11:32:29 -05:00
Ian Norden
7e426b2ca5
Merge pull request #71 from cerc-io/ian/v5_dev
v5 upgrade
2023-05-12 11:29:14 -05:00
i-norden
98015c4c87 fix NUMERIC formattign error 2023-05-12 11:11:00 -05:00
i-norden
6de5b9e96c update misc 2023-05-12 10:41:18 -05:00
i-norden
2dd9221467 update go mod 2023-05-12 10:27:20 -05:00
i-norden
ea4c1042c4 update service_test with new fixtures 2023-05-12 10:07:28 -05:00
i-norden
7af17b1851 update test helpers 2023-05-12 10:07:12 -05:00
i-norden
4081787b03 fix service.go: prevent panics on subtrie iterators with odd-length prefix-paths 2023-05-12 10:06:54 -05:00
i-norden
3d8064ccbb finish updating test fixtures 2023-05-12 09:58:46 -05:00
i-norden
fdb105b769 update compose 2023-05-10 13:11:42 -05:00
i-norden
32b637671d use v5.0.2 statediffing geth 2023-05-10 13:11:23 -05:00
i-norden
4ee75a3371 update publisher interface 2023-05-10 13:11:12 -05:00
82176ea41c
Add Docker files (#73) 2023-05-09 10:33:32 -05:00
i-norden
0a04baab17 bump db version in docker compose 2023-04-12 13:13:04 -05:00
i-norden
4245b80a4a go mod tidy 2023-04-12 13:13:04 -05:00
i-norden
22ecd4065a update service 2023-04-12 13:13:04 -05:00
i-norden
382ad92701 removei in-place snapshot stuff 2023-04-12 13:13:04 -05:00
i-norden
14b1180161 update Block1 and Chain2_Block32 test fixtures 2023-04-12 13:13:04 -05:00
i-norden
ead007f159 update csv file publisher 2023-04-12 13:13:04 -05:00
i-norden
f1a980f37c update direct pg publisher 2023-04-12 13:13:04 -05:00
i-norden
f83ab82424 use types/models exported from vdb geth 2023-04-12 13:13:04 -05:00
Ian Norden
fb5a95d874
Merge pull request #72 from cerc-io/ian/v5_rebase
Rebase
2023-04-12 13:12:35 -05:00
97ee99a449 Update schema (#61)
Updates table to add header_cids.uncles_hash
2023-04-12 13:09:58 -05:00
Ian Norden
b6d7695536
Merge pull request #70 from cerc-io/ian/v4_dev
update db version
2023-04-11 09:06:03 -05:00
i-norden
3f3e77cbac update db version 2023-04-11 09:01:50 -05:00
Ian Norden
e63ffbf0ad
Merge pull request #69 from cerc-io/ian/v4_dev
Update to use v4 vdb geth v1.11.5
2023-03-31 12:50:23 -05:00
i-norden
6ede522ae0 missing .meta files in fixtures 2023-03-31 12:36:30 -05:00
i-norden
5c13d59515 refactor to work with v4 vdb geth v1.11.5 2023-03-31 10:39:27 -05:00
i-norden
6fe54aa8b4 use go1.19 2023-03-31 10:39:13 -05:00
i-norden
c9dd85488b bump vdb v4 geth to v1.11.5 2023-03-31 10:39:00 -05:00
Michael
ac85fe29eb
dependency updates for geth 1.10.26 (#67) 2022-11-08 14:11:11 -05:00
c270f39de9
Update schema (#61)
Updates table to add header_cids.uncles_hash
2022-10-13 13:19:47 -05:00
Michael
9ea78c4ecf
dependency updates for geth 1.10.25 (#65) 2022-09-26 09:14:15 -04:00
Michael
989bd1c0f1
Cerc refactor (#64)
* cerc refactor waiting on unpublished dependencies

* more cerc_refactor with published ipld-eth-db unstable

* TearDownDB refactor

* missed second publisher_test TearDownDB
2022-09-20 13:47:34 -04:00
Michael
3f93a989dc
Merge pull request #63 from vulcanize/rebase-1.10.23-wip
updates for geth rebase on 1.10.23
2022-09-02 15:39:23 -04:00
Michael Shaw
321e37584d updates for geth rebase on 1.10.23 2022-09-02 15:21:29 -04:00
prathamesh0
be544a3424
Add helper scripts for data dump correction (#57)
* Add a script to find bad data in CSV file dumps

* Add a script to delete bad rows from CSV file dumps

* Add instructions to run the scripts

* Reorganize instructions
2022-08-17 15:14:14 +05:30
Ian Norden
05aeeab581
Account selective snapshot (#46)
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided

* config and env updates

* cmd update

* Encode watched address path bytes to hex for comparison

* actually ignore the subtries that are not along the paths of interest

* Fixes for account selective snapshot

* Use non-concurrent iterator when having a single worker

* Only index root node when starting path of an iterator is nil

* Upgrade deps

* Avoid tracking iterators and skip recovery test

* Fix recovery mechanism, use sync Map instead of buffered channels

* Add test for account selective snapshot

* Continue traversal with concurrent iterators with starting path nil

* Use errgroup to simplify error handling with concurrent iterators

* Check if all the nodes are indexed in the recovery test

* Use concurrency safe sync Map in account selective snapshot test

* Only track concurrent iterators and refactor code

* Fix node and recovered path comparison

* Revert back to using buffered channels for tracking iterators

* Add a metric to monitor number of active iterators

* Update docs

* Update seeked path after node is processed

* Return error on context cancellation from subtrie iteration

* Add tests for account selective snapshot recovery

* Explicity enforce concurrent iterator bounds to avoid duplicate nodes

* Update full snapshot test to check nodes being indexed

* Refactor code to simplify snapshot logic

* Remove unnecessary function argument

* Use ctx cancellation for handling signals

* Add descriptive comments

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 17:05:04 +05:30
prathamesh0
5002c82038
Upgrade dependencies (#56) 2022-08-03 16:11:38 +05:30
Michael
289fb63568
Merge pull request #54 from vulcanize/release-v4.1.1-alpha
update for go-ethereum 1.10.20
2022-07-20 17:18:58 -04:00