Commit Graph

46 Commits

Author SHA1 Message Date
i-norden
4081787b03 fix service.go: prevent panics on subtrie iterators with odd-length prefix-paths 2023-05-12 10:06:54 -05:00
i-norden
22ecd4065a update service 2023-04-12 13:13:04 -05:00
i-norden
5c13d59515 refactor to work with v4 vdb geth v1.11.5 2023-03-31 10:39:27 -05:00
Michael
989bd1c0f1
Cerc refactor (#64)
* cerc refactor waiting on unpublished dependencies

* more cerc_refactor with published ipld-eth-db unstable

* TearDownDB refactor

* missed second publisher_test TearDownDB
2022-09-20 13:47:34 -04:00
Ian Norden
05aeeab581
Account selective snapshot (#46)
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided

* config and env updates

* cmd update

* Encode watched address path bytes to hex for comparison

* actually ignore the subtries that are not along the paths of interest

* Fixes for account selective snapshot

* Use non-concurrent iterator when having a single worker

* Only index root node when starting path of an iterator is nil

* Upgrade deps

* Avoid tracking iterators and skip recovery test

* Fix recovery mechanism, use sync Map instead of buffered channels

* Add test for account selective snapshot

* Continue traversal with concurrent iterators with starting path nil

* Use errgroup to simplify error handling with concurrent iterators

* Check if all the nodes are indexed in the recovery test

* Use concurrency safe sync Map in account selective snapshot test

* Only track concurrent iterators and refactor code

* Fix node and recovered path comparison

* Revert back to using buffered channels for tracking iterators

* Add a metric to monitor number of active iterators

* Update docs

* Update seeked path after node is processed

* Return error on context cancellation from subtrie iteration

* Add tests for account selective snapshot recovery

* Explicity enforce concurrent iterator bounds to avoid duplicate nodes

* Update full snapshot test to check nodes being indexed

* Refactor code to simplify snapshot logic

* Remove unnecessary function argument

* Use ctx cancellation for handling signals

* Add descriptive comments

Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 17:05:04 +05:30
prathamesh0
b241bf05eb
Add command to take an in-place snapshot (#42)
* Add command to take an in-place snapshot

* Add test data for in place snapshot unit test

* Implement unit test for inplace snapshot

* Add check for storage IPLD

* Run unit tests sequentially

* Add github workflow for unit test

* Add missing checks for state and storage cid fields

* Add more storage nodes to test

* Update ipld-eth-db version for tests

* Add comments for inplace snapshot test data

* Add in-place snapshot cmd in readme

* Implement defer pattern for db transaction

* Log transaction commit or rollback error

Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-06-15 12:51:26 +05:30
baef795834 Merge changes to fix iterator and update 2022-06-10 14:12:34 +05:30
8371344756 clean up logging 2022-06-08 20:12:22 +08:00
ef391d8d31 simplify tracker, prevent send to closed chan 2022-06-08 20:12:22 +08:00
Ashwin Phatak
7129044eae Log progress info 2022-06-08 18:32:10 +08:00
dc27487b66 Merge changes to add env bindings and prometheus metrics 2022-05-23 17:11:51 +05:30
prathamesh0
cce45bda57
Add prometheus metrics collection (#33)
* Upgrade geth

* Add prometheus metrics collection

* Update README
2022-05-23 16:56:48 +05:30
fc9e82f717 Index block number using string in file mode 2022-05-13 14:00:40 +05:30
44d48b8644 Change ON CONFLICT clauses to do nothing and upgrade geth 2022-05-13 09:28:36 +05:30
26ae3ab008 Updates to use v4 schema 2022-04-19 18:14:25 +05:30
i-norden
31105a009c add env bindings 2022-04-04 14:35:18 -05:00
i-norden
060eb5cf67 add some logs and guards, update module name, update readme 2022-03-30 18:57:30 -05:00
2481d0a28d recovery - handle signals 2022-03-14 23:34:26 +08:00
02488e2b79 logging tweaks 2022-03-14 23:32:00 +08:00
b5d0010581 implement job recovery; fix traversal & iterator 2022-03-03 17:12:29 +08:00
c4b4f232e9 create csv file publisher; update geth and schema 2022-02-14 07:22:31 +08:00
159f93902e tests - mock publisher with fixtures
* go structs for header, etc.
* refactor: pkg/types

* add fixture chaindata
2022-01-14 21:43:00 -06:00
2bb06e0a8d split ethdb out of service 2022-01-10 23:37:27 -06:00
1fab8ac1c8 rf config, mock publisher 2022-01-10 19:08:45 -06:00
1f21a65478 fixes 2022-01-10 18:06:29 -06:00
317b277939 Merge branch 'master' into async-traversal 2022-01-10 17:47:27 -06:00
f7b44173f8 get it to compile 2022-01-07 21:32:45 -06:00
i-norden
e50bd160fe log final counts when process completes 2021-12-28 23:47:33 -06:00
Arijit Das
50edaff3ce Improve logging. 2021-12-23 19:56:24 +05:30
Arijit Das
112cdc9670 Address comments. 2021-12-15 12:53:18 +05:30
Arijit Das
e4de336833 Fix linting errors. 2021-12-14 12:59:34 +05:30
Arijit Das
20d6b1dc64 Small changes. 2021-12-13 20:42:02 +05:30
Arijit Das
ba8e32ce97 Update geth and implement transaction batching. 2021-12-13 20:31:32 +05:30
cd3d9f27cf cleanup 2020-09-07 21:02:30 -05:00
66d6acc1fe update iterator 2020-09-06 02:36:36 -05:00
169887f39f number of workers instead of divide depth
update w/ iterator change
2020-08-31 11:09:56 -05:00
96ad583b06 cleanup 2020-08-24 00:13:21 -05:00
f651864ff8 [wip] use new state iterator 2020-08-23 21:48:24 -05:00
49bd60d9f1 [wip] async traversal 2020-08-20 15:29:10 -05:00
Ian Norden
5ed3e92469 use freezer db 2020-08-03 10:46:35 -05:00
Ian Norden
b8a0923250 store code bytes in ipfs blockstore table 2020-07-31 14:18:30 -05:00
Ian Norden
3bf9ab2d1b option for defering to the latest height 2020-07-16 10:31:37 -05:00
Ian Norden
367a9b4577 add guard 2020-07-16 10:02:30 -05:00
Ian Norden
a96e1df58d use only blocknumber; minor changes 2020-07-15 22:09:16 -05:00
Ian Norden
5cb67910bd finish snapshot extractor 2020-07-01 18:07:56 -05:00
Ian Norden
78adba02ea state snapshot extractor 2020-07-01 14:10:41 -05:00