Ian Norden
e63ffbf0ad
Merge pull request #69 from cerc-io/ian/v4_dev
...
Update to use v4 vdb geth v1.11.5
2023-03-31 12:50:23 -05:00
i-norden
6ede522ae0
missing .meta files in fixtures
2023-03-31 12:36:30 -05:00
i-norden
5c13d59515
refactor to work with v4 vdb geth v1.11.5
2023-03-31 10:39:27 -05:00
i-norden
6fe54aa8b4
use go1.19
2023-03-31 10:39:13 -05:00
i-norden
c9dd85488b
bump vdb v4 geth to v1.11.5
2023-03-31 10:39:00 -05:00
Michael
ac85fe29eb
dependency updates for geth 1.10.26 ( #67 )
2022-11-08 14:11:11 -05:00
c270f39de9
Update schema ( #61 )
...
Updates table to add header_cids.uncles_hash
2022-10-13 13:19:47 -05:00
Michael
9ea78c4ecf
dependency updates for geth 1.10.25 ( #65 )
2022-09-26 09:14:15 -04:00
Michael
989bd1c0f1
Cerc refactor ( #64 )
...
* cerc refactor waiting on unpublished dependencies
* more cerc_refactor with published ipld-eth-db unstable
* TearDownDB refactor
* missed second publisher_test TearDownDB
2022-09-20 13:47:34 -04:00
Michael
3f93a989dc
Merge pull request #63 from vulcanize/rebase-1.10.23-wip
...
updates for geth rebase on 1.10.23
2022-09-02 15:39:23 -04:00
Michael Shaw
321e37584d
updates for geth rebase on 1.10.23
2022-09-02 15:21:29 -04:00
prathamesh0
be544a3424
Add helper scripts for data dump correction ( #57 )
...
* Add a script to find bad data in CSV file dumps
* Add a script to delete bad rows from CSV file dumps
* Add instructions to run the scripts
* Reorganize instructions
2022-08-17 15:14:14 +05:30
Ian Norden
05aeeab581
Account selective snapshot ( #46 )
...
* snapshotter ignores nodes not along a path along those derived from a list of account addresses if one is provided
* config and env updates
* cmd update
* Encode watched address path bytes to hex for comparison
* actually ignore the subtries that are not along the paths of interest
* Fixes for account selective snapshot
* Use non-concurrent iterator when having a single worker
* Only index root node when starting path of an iterator is nil
* Upgrade deps
* Avoid tracking iterators and skip recovery test
* Fix recovery mechanism, use sync Map instead of buffered channels
* Add test for account selective snapshot
* Continue traversal with concurrent iterators with starting path nil
* Use errgroup to simplify error handling with concurrent iterators
* Check if all the nodes are indexed in the recovery test
* Use concurrency safe sync Map in account selective snapshot test
* Only track concurrent iterators and refactor code
* Fix node and recovered path comparison
* Revert back to using buffered channels for tracking iterators
* Add a metric to monitor number of active iterators
* Update docs
* Update seeked path after node is processed
* Return error on context cancellation from subtrie iteration
* Add tests for account selective snapshot recovery
* Explicity enforce concurrent iterator bounds to avoid duplicate nodes
* Update full snapshot test to check nodes being indexed
* Refactor code to simplify snapshot logic
* Remove unnecessary function argument
* Use ctx cancellation for handling signals
* Add descriptive comments
Co-authored-by: prathamesh0 <prathamesh.musale0@gmail.com>
2022-08-03 17:05:04 +05:30
prathamesh0
5002c82038
Upgrade dependencies ( #56 )
2022-08-03 16:11:38 +05:30
Michael
289fb63568
Merge pull request #54 from vulcanize/release-v4.1.1-alpha
...
update for go-ethereum 1.10.20
2022-07-20 17:18:58 -04:00
Michael Shaw
257fac1a0c
update for go-ethereum 1.10.20
2022-07-19 15:12:44 -04:00
prathamesh0
c0a7fdf9e8
Fix typo in config format in README ( #53 )
2022-07-18 16:48:58 +05:30
prathamesh0
2af26ad583
Add instructions to import snapshot data into database ( #52 )
...
* Add instructions to import snapshot data into database
* Add monitoring and update data processing in README
* Update instructions to import snapshot
2022-07-18 15:30:23 +05:30
prathamesh0
ac00ee280f
Upgrade geth ( #51 )
...
* Fix Makefile for installing mockgen tool
* Upgrade geth
* Add sleep in the failing recovery test
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-07-13 11:39:36 +05:30
prathamesh0
b241bf05eb
Add command to take an in-place snapshot ( #42 )
...
* Add command to take an in-place snapshot
* Add test data for in place snapshot unit test
* Implement unit test for inplace snapshot
* Add check for storage IPLD
* Run unit tests sequentially
* Add github workflow for unit test
* Add missing checks for state and storage cid fields
* Add more storage nodes to test
* Update ipld-eth-db version for tests
* Add comments for inplace snapshot test data
* Add in-place snapshot cmd in readme
* Implement defer pattern for db transaction
* Log transaction commit or rollback error
Co-authored-by: nabarun <nabarun@deepstacksoft.com>
2022-06-15 12:51:26 +05:30
Ashwin Phatak
c2ecea6d61
Merge pull request #45 from vulcanize/ng-merge-fix-iterator
...
Merge changes to fix iterator and update
2022-06-10 14:24:33 +05:30
baef795834
Merge changes to fix iterator and update
2022-06-10 14:12:34 +05:30
4979e4090d
Merge pull request #43 from roysc/fix-iterator-and-update
...
Fix iterator and update
2022-06-10 14:03:43 +05:30
59011cba3c
Update go-eth-state-node-iterator version
2022-06-10 14:02:13 +05:30
8371344756
clean up logging
2022-06-08 20:12:22 +08:00
ef391d8d31
simplify tracker, prevent send to closed chan
2022-06-08 20:12:22 +08:00
838dabcba6
update iterator mod
2022-06-08 20:12:22 +08:00
8572c36ee7
decrement path from restored iterator
...
to cover node gaps
2022-06-08 20:12:22 +08:00
Ashwin Phatak
74f82a641d
Fix dependency
2022-06-08 20:05:38 +08:00
3e24972a89
try to fix hanging iterator
2022-06-08 20:05:38 +08:00
Ashwin Phatak
7129044eae
Log progress info
2022-06-08 18:32:10 +08:00
Abdul Rabbani
c9d398b5af
Merge pull request #40 from vulcanize/feature/update-go-geth-sharding
...
Update go and geth versions
2022-06-01 09:22:04 -04:00
Abdul Rabbani
8aab6a5137
Update geth versions
2022-06-01 09:20:45 -04:00
Abdul Rabbani
cd2559e434
Merge branch 'master' into feature/update-go-geth-sharding
2022-06-01 09:19:18 -04:00
Abdul Rabbani
d213644ff4
Merge pull request #39 from vulcanize/feature/update-go-geth
...
Updating Go and Geth Version
2022-06-01 09:10:35 -04:00
Abdul Rabbani
a04777cb78
Updating Go and Geth Version
...
Ran Tests:
```
09:07:50:~/GitHub/cerc/ipld-eth-state-snapshot go test ./... -v
# github.com/vulcanize/ipld-eth-state-snapshot/cmd
cmd/stateSnapshot.go:47:3: (*github.com/sirupsen/logrus.Entry).Fatal call has possible formatting directive %v
? github.com/vulcanize/ipld-eth-state-snapshot [no test files]
=== RUN TestCreateSnapshot
=== RUN TestCreateSnapshot/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN TestCreateSnapshot/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN TestCreateSnapshot/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN TestCreateSnapshot/case#03
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestCreateSnapshot (0.20s)
--- PASS: TestCreateSnapshot/case (0.15s)
--- PASS: TestCreateSnapshot/case#01 (0.01s)
--- PASS: TestCreateSnapshot/case#02 (0.02s)
--- PASS: TestCreateSnapshot/case#03 (0.03s)
=== RUN TestRecovery
=== RUN TestRecovery/case
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase3574218035/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN TestRecovery/case#01
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#011814952517/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
=== RUN TestRecovery/case#02
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
time="2022-06-01T09:08:03-04:00" level=info msg="Dumping recovery state to: /var/folders/xf/cg688d9j7f39ktmd_4ttwxqh0000gn/T/TestRecoverycase#021650290358/001/recover.csv"
time="2022-06-01T09:08:03-04:00" level=info msg="Creating snapshot at height 1"
time="2022-06-01T09:08:03-04:00" level=info msg="head hash: 0xa7684ac44d48494670b2e0d9085b7750e7341620f0a271db146ed5e70c1db854 head height: 1"
--- PASS: TestRecovery (0.40s)
--- PASS: TestRecovery/case (0.12s)
--- PASS: TestRecovery/case#01 (0.19s)
--- PASS: TestRecovery/case#02 (0.09s)
PASS
ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot 1.109s
=== RUN TestWriting
--- PASS: TestWriting (0.00s)
=== RUN TestPgCopy
publisher_test.go:84: set TEST_WITH_DB to enable test
--- SKIP: TestPgCopy (0.00s)
PASS
ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/file 0.279s
=== RUN TestBasic
publisher_test.go:47: set TEST_WITH_DB to enable test
--- SKIP: TestBasic (0.00s)
PASS
ok github.com/vulcanize/ipld-eth-state-snapshot/pkg/snapshot/pg 0.647s
```
2022-06-01 09:09:19 -04:00
Ashwin Phatak
c785c12274
Merge pull request #35 from vulcanize/ng-file-ethereum-nodeId
...
Add ethereum config to fix db import in file mode
2022-05-25 18:38:09 +05:30
23c4d22885
Add ethereum config to fix db import in file mode
2022-05-25 18:15:08 +05:30
Ashwin Phatak
d3a149899b
Merge pull request #34 from deep-stack/pm-merge-metrics
...
Merge changes to add env bindings and prometheus metrics
2022-05-23 18:14:06 +05:30
dc27487b66
Merge changes to add env bindings and prometheus metrics
2022-05-23 17:11:51 +05:30
prathamesh0
cce45bda57
Add prometheus metrics collection ( #33 )
...
* Upgrade geth
* Add prometheus metrics collection
* Update README
2022-05-23 16:56:48 +05:30
Ian Norden
751e064f8f
Merge pull request #28 from vulcanize/release-v3.0.2
...
add env bindings
2022-05-17 22:36:08 -05:00
i-norden
004f3e6665
constants for CLI binding strings
2022-05-17 22:33:20 -05:00
prathamesh0
0a1ea3ae4a
Fix insert statements and upgrade dependencies ( #31 )
2022-05-17 19:19:52 +05:30
Ashwin Phatak
b3af4e9e75
Merge pull request #30 from deep-stack/pm-upgrade-geth
...
Upgrade vulcanize-geth to v4
2022-05-16 14:05:28 +05:30
67ec456e88
Upgrade vulcanize-geth to v4
2022-05-16 13:48:26 +05:30
Ashwin Phatak
04b57ca186
Merge pull request #29 from deep-stack/pm-db-sharding
...
Updates to use v4 schema
2022-05-13 16:06:09 +05:30
fc9e82f717
Index block number using string in file mode
2022-05-13 14:00:40 +05:30
44d48b8644
Change ON CONFLICT clauses to do nothing and upgrade geth
2022-05-13 09:28:36 +05:30
26ae3ab008
Updates to use v4 schema
2022-04-19 18:14:25 +05:30
