2019-11-18 12:22:36 +00:00
|
|
|
module github.com/ethereum/go-ethereum
|
|
|
|
|
|
2021-03-19 12:15:57 +00:00
|
|
|
go 1.15
|
2019-11-18 12:22:36 +00:00
|
|
|
|
|
|
|
|
require (
|
2021-04-28 10:04:25 +00:00
|
|
|
github.com/Azure/azure-pipeline-go v0.2.2 // indirect
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/Azure/azure-storage-blob-go v0.7.0
|
2021-04-28 10:04:25 +00:00
|
|
|
github.com/Azure/go-autorest/autorest/adal v0.8.0 // indirect
|
|
|
|
|
github.com/StackExchange/wmi v0.0.0-20180116203802-5d049714c4a6 // indirect
|
2021-06-08 08:39:05 +00:00
|
|
|
github.com/VictoriaMetrics/fastcache v1.6.0
|
2021-03-19 12:15:57 +00:00
|
|
|
github.com/aws/aws-sdk-go-v2 v1.2.0
|
|
|
|
|
github.com/aws/aws-sdk-go-v2/config v1.1.1
|
|
|
|
|
github.com/aws/aws-sdk-go-v2/credentials v1.1.1
|
|
|
|
|
github.com/aws/aws-sdk-go-v2/service/route53 v1.1.1
|
2021-01-25 09:42:07 +00:00
|
|
|
github.com/btcsuite/btcd v0.20.1-beta
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/cespare/cp v0.1.0
|
2021-03-26 21:15:20 +00:00
|
|
|
github.com/cloudflare/cloudflare-go v0.14.0
|
2021-04-28 10:04:25 +00:00
|
|
|
github.com/consensys/gnark-crypto v0.4.1-0.20210426202927-39ac3d4b3f1f
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/davecgh/go-spew v1.1.1
|
|
|
|
|
github.com/deckarep/golang-set v0.0.0-20180603214616-504e848d77ea
|
2021-08-17 16:40:14 +00:00
|
|
|
github.com/deepmap/oapi-codegen v1.8.2 // indirect
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/docker/docker v1.4.2-0.20180625184442-8e610b2b55bf
|
2021-10-13 12:42:00 +00:00
|
|
|
github.com/dop251/goja v0.0.0-20211011172007-d99e4b8cbf48
|
2021-01-25 09:42:07 +00:00
|
|
|
github.com/edsrzf/mmap-go v1.0.0
|
all: bloom-filter based pruning mechanism (#21724)
* cmd, core, tests: initial state pruner
core: fix db inspector
cmd/geth: add verify-state
cmd/geth: add verification tool
core/rawdb: implement flatdb
cmd, core: fix rebase
core/state: use new contract code layout
core/state/pruner: avoid deleting genesis state
cmd/geth: add helper function
core, cmd: fix extract genesis
core: minor fixes
contracts: remove useless
core/state/snapshot: plugin stacktrie
core: polish
core/state/snapshot: iterate storage concurrently
core/state/snapshot: fix iteration
core: add comments
core/state/snapshot: polish code
core/state: polish
core/state/snapshot: rebase
core/rawdb: add comments
core/rawdb: fix tests
core/rawdb: improve tests
core/state/snapshot: fix concurrent iteration
core/state: run pruning during the recovery
core, trie: implement martin's idea
core, eth: delete flatdb and polish pruner
trie: fix import
core/state/pruner: add log
core/state/pruner: fix issues
core/state/pruner: don't read back
core/state/pruner: fix contract code write
core/state/pruner: check root node presence
cmd, core: polish log
core/state: use HEAD-127 as the target
core/state/snapshot: improve tests
cmd/geth: fix verification tool
cmd/geth: use HEAD as the verification default target
all: replace the bloomfilter with martin's fork
cmd, core: polish code
core, cmd: forcibly delete state root
core/state/pruner: add hash64
core/state/pruner: fix blacklist
core/state: remove blacklist
cmd, core: delete trie clean cache before pruning
cmd, core: fix lint
cmd, core: fix rebase
core/state: fix the special case for clique networks
core/state/snapshot: remove useless code
core/state/pruner: capping the snapshot after pruning
cmd, core, eth: fixes
core/rawdb: update db inspector
cmd/geth: polish code
core/state/pruner: fsync bloom filter
cmd, core: print warning log
core/state/pruner: adjust the parameters for bloom filter
cmd, core: create the bloom filter by size
core: polish
core/state/pruner: sanitize invalid bloomfilter size
cmd: address comments
cmd/geth: address comments
cmd/geth: address comment
core/state/pruner: address comments
core/state/pruner: rename homedir to datadir
cmd, core: address comments
core/state/pruner: address comment
core/state: address comments
core, cmd, tests: address comments
core: address comments
core/state/pruner: release the iterator after each commit
core/state/pruner: improve pruner
cmd, core: adjust bloom paramters
core/state/pruner: fix lint
core/state/pruner: fix tests
core: fix rebase
core/state/pruner: remove atomic rename
core/state/pruner: address comments
all: run go mod tidy
core/state/pruner: avoid false-positive for the middle state roots
core/state/pruner: add checks for middle roots
cmd/geth: replace crit with error
* core/state/pruner: fix lint
* core: drop legacy bloom filter
* core/state/snapshot: improve pruner
* core/state/snapshot: polish concurrent logs to report ETA vs. hashes
* core/state/pruner: add progress report for pruning and compaction too
* core: fix snapshot test API
* core/state: fix some pruning logs
* core/state/pruner: support recovering from bloom flush fail
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-02-08 11:16:30 +00:00
|
|
|
github.com/fatih/color v1.7.0
|
2021-01-25 09:42:07 +00:00
|
|
|
github.com/fjl/memsize v0.0.0-20190710130421-bcb5799ab5e5
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/gballet/go-libpcsclite v0.0.0-20190607065134-2772fd86a8ff
|
2021-11-10 23:02:26 +00:00
|
|
|
github.com/georgysavva/scany v0.2.9
|
2021-04-28 10:04:25 +00:00
|
|
|
github.com/go-ole/go-ole v1.2.1 // indirect
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/go-stack/stack v1.8.0
|
2021-01-25 09:42:07 +00:00
|
|
|
github.com/golang/protobuf v1.4.3
|
2021-08-24 19:02:46 +00:00
|
|
|
github.com/golang/snappy v0.0.4
|
2020-09-16 11:15:22 +00:00
|
|
|
github.com/google/gofuzz v1.1.1-0.20200604201612-c04b05f3adfa
|
2021-02-26 14:28:34 +00:00
|
|
|
github.com/google/uuid v1.1.5
|
2021-01-25 09:42:07 +00:00
|
|
|
github.com/gorilla/websocket v1.4.2
|
|
|
|
|
github.com/graph-gophers/graphql-go v0.0.0-20201113091052-beb923fada29
|
2021-09-20 16:12:20 +00:00
|
|
|
github.com/hashicorp/go-bexpr v0.1.10
|
2021-01-20 18:34:21 +00:00
|
|
|
github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d
|
2021-01-12 16:39:31 +00:00
|
|
|
github.com/holiman/bloomfilter/v2 v2.0.3
|
2021-05-10 10:29:33 +00:00
|
|
|
github.com/holiman/uint256 v1.2.0
|
2021-07-29 13:20:15 +00:00
|
|
|
github.com/huin/goupnp v1.0.2
|
2021-01-25 09:42:07 +00:00
|
|
|
github.com/influxdata/influxdb v1.8.3
|
2021-08-17 16:40:14 +00:00
|
|
|
github.com/influxdata/influxdb-client-go/v2 v2.4.0
|
|
|
|
|
github.com/influxdata/line-protocol v0.0.0-20210311194329-9aa0e372d097 // indirect
|
Statediffing geth
* Write state diff to CSV (#2)
* port statediff from https://github.com/jpmorganchase/quorum/blob/9b7fd9af8082795eeeb6863d9746f12b82dd5078/statediff/statediff.go; minor fixes
* integrating state diff extracting, building, and persisting into geth processes
* work towards persisting created statediffs in ipfs; based off github.com/vulcanize/eth-block-extractor
* Add a state diff service
* Remove diff extractor from blockchain
* Update imports
* Move statediff on/off check to geth cmd config
* Update starting state diff service
* Add debugging logs for creating diff
* Add statediff extractor and builder tests and small refactoring
* Start to write statediff to a CSV
* Restructure statediff directory
* Pull CSV publishing methods into their own file
* Reformatting due to go fmt
* Add gomega to vendor dir
* Remove testing focuses
* Update statediff tests to use golang test pkg
instead of ginkgo
- builder_test
- extractor_test
- publisher_test
* Use hexutil.Encode instead of deprecated common.ToHex
* Remove OldValue from DiffBigInt and DiffUint64 fields
* Update builder test
* Remove old storage value from updated accounts
* Remove old values from created/deleted accounts
* Update publisher to account for only storing current account values
* Update service loop and fetching previous block
* Update testing
- remove statediff ginkgo test suite file
- move mocks to their own dir
* Updates per go fmt
* Updates to tests
* Pass statediff mode and path in through cli
* Return filename from publisher
* Remove some duplication in builder
* Remove code field from state diff output
this is the contract byte code, and it can still be obtained by querying
the db by the codeHash
* Consolidate acct diff structs for updated & updated/deleted accts
* Include block number in csv filename
* Clean up error logging
* Cleanup formatting, spelling, etc
* Address PR comments
* Add contract address and storage value to csv
* Refactor accumulating account row in csv publisher
* Add DiffStorage struct
* Add storage key to csv
* Address PR comments
* Fix publisher to include rows for accounts that don't have store updates
* Update builder test after merging in release/1.8
* Update test contract to include storage on contract intialization
- so that we're able to test that storage diffing works for created and
deleted accounts (not just updated accounts).
* Factor out a common trie iterator method in builder
* Apply goimports to statediff
* Apply gosimple changes to statediff
* Gracefully exit geth command(#4)
* Statediff for full node (#6)
* Open a trie from the in-memory database
* Use a node's LeafKey as an identifier instead of the address
It was proving difficult to find look the address up from a given path
with a full node (sometimes the value wouldn't exist in the disk db).
So, instead, for now we are using the node's LeafKey with is a Keccak256
hash of the address, so if we know the address we can figure out which
LeafKey it matches up to.
* Make sure that statediff has been processed before pruning
* Use blockchain stateCache.OpenTrie for storage diffs
* Clean up log lines and remove unnecessary fields from builder
* Apply go fmt changes
* Add a sleep to the blockchain test
* refactoring/reorganizing packages
* refactoring statediff builder and types and adjusted to relay proofs and paths (still need to make this optional)
* refactoring state diff service and adding api which allows for streaming state diff payloads over an rpc websocket subscription
* make proofs and paths optional + compress service loop into single for loop (may be missing something here)
* option to process intermediate nodes
* make state diff rlp serializable
* cli parameter to limit statediffing to select account addresses + test
* review fixes and fixes for issues ran into in integration
* review fixes; proper method signature for api; adjust service so that statediff processing is halted/paused until there is at least one subscriber listening for the results
* adjust buffering to improve stability; doc.go; fix notifier
err handling
* relay receipts with the rest of the data + review fixes/changes
* rpc method to get statediff at specific block; requires archival node or the block be within the pruning range
* fix linter issues
* include total difficulty to the payload
* fix state diff builder: emit actual leaf nodes instead of value nodes; diff on the leaf not on the value; emit correct path for intermediate nodes
* adjust statediff builder tests to changes and extend to test intermediate nodes; golint
* add genesis block to test; handle block 0 in StateDiffAt
* rlp files for mainnet blocks 0-3, for tests
* builder test on mainnet blocks
* common.BytesToHash(path) => crypto.Keaccak256(hash) in builder; BytesToHash produces same hash for e.g. []byte{} and []byte{\x00} - prefix \x00 steps are inconsequential to the hash result
* complete tests for early mainnet blocks
* diff type for representing deleted accounts
* fix builder so that we handle account deletions properly and properly diff storage when an account is moved to a new path; update params
* remove cli params; moving them to subscriber defined
* remove unneeded bc methods
* update service and api; statediffing params are now defined by user through api rather than by service provider by cli
* update top level tests
* add ability to watch specific storage slots (leaf keys) only
* comments; explain logic
* update mainnet blocks test
* update api_test.go
* storage leafkey filter test
* cleanup chain maker
* adjust chain maker for tests to add an empty account in block1 and switch to EIP-158 afterwards (now we just need to generate enough accounts until one causes the empty account to be touched and removed post-EIP-158 so we can simulate and test that process...); also added 2 new blocks where more contract storage is set and old slots are set to zero so they are removed so we can test that
* found an account whose creation causes the empty account to be moved to a new path; this should count as 'touching; the empty account and cause it to be removed according to eip-158... but it doesn't
* use new contract in unit tests that has self-destruct ability, so we can test eip-158 since simply moving an account to new path doesn't count as 'touchin' it
* handle storage deletions
* tests for eip-158 account removal and storage value deletions; there is one edge case left to test where we remove 1 account when only two exist such that the remaining account is moved up and replaces the root branch node
* finish testing known edge cases
* add endpoint to fetch all state and storage nodes at a given blockheight; useful for generating a recent atate cache/snapshot that we can diff forward from rather than needing to collect all diffs from genesis
* test for state trie builder
* if statediffing is on, lock tries in triedb until the statediffing service signals they are done using them
* fix mock blockchain; golint; bump patch
* increase maxRequestContentLength; bump patch
* log the sizes of the state objects we are sending
* CI build (#20)
* CI: run build on PR and on push to master
* CI: debug building geth
* CI: fix coping file
* CI: fix coping file v2
* CI: temporary upload file to release asset
* CI: get release upload_url by tag, upload asset to current relase
* CI: fix tag name
* fix ci build on statediff_at_anyblock-1.9.11 branch
* fix publishing assets in release
* use context deadline for timeout in eth_call
* collect and emit codehash=>code mappings for state objects
* subscription endpoint for retrieving all the codehash=>code mappings that exist at provided height
* Implement WriteStateDiffAt
* Writes state diffs directly to postgres
* Adds CLI flags to configure PG
* Refactors builder output with callbacks
* Copies refactored postgres handling code from ipld-eth-indexer
* rename PostgresCIDWriter.{index->upsert}*
* go.mod update
* rm unused
* cleanup
* output code & codehash iteratively
* had to rf some types for this
* prometheus metrics output
* duplicate recent eth-indexer changes
* migrations and metrics...
* [wip] prom.Init() here? another CLI flag?
* tidy & DRY
* statediff WriteLoop service + CLI flag
* [wip] update test mocks
* todo - do something meaningful to test write loop
* logging
* use geth log
* port tests to go testing
* drop ginkgo/gomega
* fix and cleanup tests
* fail before defer statement
* delete vendor/ dir
* fixes after rebase onto 1.9.23
* fix API registration
* use golang 1.15.5 version (#34)
* bump version meta; add 0.0.11 branch to actions
* bump version meta; update github actions workflows
* statediff: refactor metrics
* Remove redundant statediff/indexer/prom tooling and use existing
prometheus integration.
* "indexer" namespace for metrics
* add reporting loop for db metrics
* doc
* metrics for statediff stats
* metrics namespace/subsystem = statediff/{indexer,service}
* statediff: use a worker pool (for direct writes)
* fix test
* fix chain event subscription
* log tweaks
* func name
* unused import
* intermediate chain event channel for metrics
* update github actions; linting
* add poststate and status to receipt ipld indexes
* stateDiffFor endpoints for fetching or writing statediff object by blockhash; bump statediff version
* fixes after rebase on to v1.10.1
* update github actions and version meta; go fmt
* add leaf key to removed 'nodes'
* include Postgres migrations and schema
* service documentation
* touching up
update github actions after rebase
fix connection leak (misplaced defer) and perform proper rollback on errs
improve error logging; handle PushBlock internal err
* build docker image and publish it to Docker Hub on release
* add access list tx to unit tests
* MarshalBinary and UnmarshalBinary methods for receipt
* fix error caused by 2718 by using MarshalBinary instead of EncodeRLP methods
* ipld encoding/decoding tests
* update TxModel; add AccessListElementModel
* index tx type and access lists
* add access list metrics
* unit tests for tx_type and access list table
* unit tests for receipt marshal/unmarshal binary methods
* improve documentation of the encoding methods
* fix issue identified in linting
update github actions and version meta after rebase
unit test that fails undeterministically on eip2930 txs, giving same error we are seeing in prod
fix bug
Include genesis block state diff.
Fix linting issue.
documentation on versioning, rebasing, releasing; bump version meta
Add geth and statediff unit test to CI.
Set pgpassword in env.
Added comments.
Add new major branch to github action.
Fix failing test.
Fix lint errors.
Add support for Dynamic txn(EIP-1559).
Update version meta to 0.0.24
Verify block base fee in test.
Fix base_fee type and add backward compatible test.
Remove type definition for AccessListElementModel
Change basefee to int64/bigint.
block and uncle reward in PoA network = 0 (#87)
* in PoA networks there is no block and uncle rewards
* bump meta version
(cherry picked from commit b64ca14689689178b78f915c855fe75010d886c7)
Use Ropsten to test block reward.
Add Makefile target to build static linux binaries.
Strip symbol tables from static binaries.
Fix block_fee to support NULL values.
bump version meta.
Add new major branch to github action.
Add new major branch to github action.
Add new major branch to github action.
Add new major branch to github action.
rename doc.go to README.md
Create a seperate table for storing logs
Self review
Bump statediff version to 0.0.26.
add btree index to state/storage_cids.node_type; updated schema
Dedup receipt data.
Fix linter errors.
Address comments.
Bump statediff version to 0.0.27.
new cli flag for initializing db first time service is ran
only write Removed node ipld block (on db init) and reuse constant cid and mhkey
linting
test new handling of Removed nodes; don't require init flag
log metrics
Add new major branch to github action.
Fix build.
Update golang version in CI.
Use ipld-eth-db in testing.
Remove migration from repo.
Add new major branch to github action.
Use `GetTd` instead of `GetTdByHash`
https://github.com/ethereum/go-ethereum/commit/62891378274a58c84b9fa35f3c9b99155644b5eb
2019-01-28 21:31:01 +00:00
|
|
|
github.com/ipfs/go-block-format v0.0.2
|
|
|
|
|
github.com/ipfs/go-cid v0.0.7
|
|
|
|
|
github.com/ipfs/go-ipfs-blockstore v1.0.1
|
|
|
|
|
github.com/ipfs/go-ipfs-ds-help v1.0.0
|
|
|
|
|
github.com/ipfs/go-ipld-format v0.2.0
|
2021-11-10 23:02:26 +00:00
|
|
|
github.com/jackc/fake v0.0.0-20150926172116-812a484cc733 // indirect
|
|
|
|
|
github.com/jackc/pgconn v1.10.0
|
|
|
|
|
github.com/jackc/pgx v3.6.2+incompatible
|
|
|
|
|
github.com/jackc/pgx/v4 v4.13.0
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/jackpal/go-nat-pmp v1.0.2-0.20160603034137-1fa385a6f458
|
2020-11-27 11:13:54 +00:00
|
|
|
github.com/jedisct1/go-minisign v0.0.0-20190909160543-45766022959e
|
Statediffing geth
* Write state diff to CSV (#2)
* port statediff from https://github.com/jpmorganchase/quorum/blob/9b7fd9af8082795eeeb6863d9746f12b82dd5078/statediff/statediff.go; minor fixes
* integrating state diff extracting, building, and persisting into geth processes
* work towards persisting created statediffs in ipfs; based off github.com/vulcanize/eth-block-extractor
* Add a state diff service
* Remove diff extractor from blockchain
* Update imports
* Move statediff on/off check to geth cmd config
* Update starting state diff service
* Add debugging logs for creating diff
* Add statediff extractor and builder tests and small refactoring
* Start to write statediff to a CSV
* Restructure statediff directory
* Pull CSV publishing methods into their own file
* Reformatting due to go fmt
* Add gomega to vendor dir
* Remove testing focuses
* Update statediff tests to use golang test pkg
instead of ginkgo
- builder_test
- extractor_test
- publisher_test
* Use hexutil.Encode instead of deprecated common.ToHex
* Remove OldValue from DiffBigInt and DiffUint64 fields
* Update builder test
* Remove old storage value from updated accounts
* Remove old values from created/deleted accounts
* Update publisher to account for only storing current account values
* Update service loop and fetching previous block
* Update testing
- remove statediff ginkgo test suite file
- move mocks to their own dir
* Updates per go fmt
* Updates to tests
* Pass statediff mode and path in through cli
* Return filename from publisher
* Remove some duplication in builder
* Remove code field from state diff output
this is the contract byte code, and it can still be obtained by querying
the db by the codeHash
* Consolidate acct diff structs for updated & updated/deleted accts
* Include block number in csv filename
* Clean up error logging
* Cleanup formatting, spelling, etc
* Address PR comments
* Add contract address and storage value to csv
* Refactor accumulating account row in csv publisher
* Add DiffStorage struct
* Add storage key to csv
* Address PR comments
* Fix publisher to include rows for accounts that don't have store updates
* Update builder test after merging in release/1.8
* Update test contract to include storage on contract intialization
- so that we're able to test that storage diffing works for created and
deleted accounts (not just updated accounts).
* Factor out a common trie iterator method in builder
* Apply goimports to statediff
* Apply gosimple changes to statediff
* Gracefully exit geth command(#4)
* Statediff for full node (#6)
* Open a trie from the in-memory database
* Use a node's LeafKey as an identifier instead of the address
It was proving difficult to find look the address up from a given path
with a full node (sometimes the value wouldn't exist in the disk db).
So, instead, for now we are using the node's LeafKey with is a Keccak256
hash of the address, so if we know the address we can figure out which
LeafKey it matches up to.
* Make sure that statediff has been processed before pruning
* Use blockchain stateCache.OpenTrie for storage diffs
* Clean up log lines and remove unnecessary fields from builder
* Apply go fmt changes
* Add a sleep to the blockchain test
* refactoring/reorganizing packages
* refactoring statediff builder and types and adjusted to relay proofs and paths (still need to make this optional)
* refactoring state diff service and adding api which allows for streaming state diff payloads over an rpc websocket subscription
* make proofs and paths optional + compress service loop into single for loop (may be missing something here)
* option to process intermediate nodes
* make state diff rlp serializable
* cli parameter to limit statediffing to select account addresses + test
* review fixes and fixes for issues ran into in integration
* review fixes; proper method signature for api; adjust service so that statediff processing is halted/paused until there is at least one subscriber listening for the results
* adjust buffering to improve stability; doc.go; fix notifier
err handling
* relay receipts with the rest of the data + review fixes/changes
* rpc method to get statediff at specific block; requires archival node or the block be within the pruning range
* fix linter issues
* include total difficulty to the payload
* fix state diff builder: emit actual leaf nodes instead of value nodes; diff on the leaf not on the value; emit correct path for intermediate nodes
* adjust statediff builder tests to changes and extend to test intermediate nodes; golint
* add genesis block to test; handle block 0 in StateDiffAt
* rlp files for mainnet blocks 0-3, for tests
* builder test on mainnet blocks
* common.BytesToHash(path) => crypto.Keaccak256(hash) in builder; BytesToHash produces same hash for e.g. []byte{} and []byte{\x00} - prefix \x00 steps are inconsequential to the hash result
* complete tests for early mainnet blocks
* diff type for representing deleted accounts
* fix builder so that we handle account deletions properly and properly diff storage when an account is moved to a new path; update params
* remove cli params; moving them to subscriber defined
* remove unneeded bc methods
* update service and api; statediffing params are now defined by user through api rather than by service provider by cli
* update top level tests
* add ability to watch specific storage slots (leaf keys) only
* comments; explain logic
* update mainnet blocks test
* update api_test.go
* storage leafkey filter test
* cleanup chain maker
* adjust chain maker for tests to add an empty account in block1 and switch to EIP-158 afterwards (now we just need to generate enough accounts until one causes the empty account to be touched and removed post-EIP-158 so we can simulate and test that process...); also added 2 new blocks where more contract storage is set and old slots are set to zero so they are removed so we can test that
* found an account whose creation causes the empty account to be moved to a new path; this should count as 'touching; the empty account and cause it to be removed according to eip-158... but it doesn't
* use new contract in unit tests that has self-destruct ability, so we can test eip-158 since simply moving an account to new path doesn't count as 'touchin' it
* handle storage deletions
* tests for eip-158 account removal and storage value deletions; there is one edge case left to test where we remove 1 account when only two exist such that the remaining account is moved up and replaces the root branch node
* finish testing known edge cases
* add endpoint to fetch all state and storage nodes at a given blockheight; useful for generating a recent atate cache/snapshot that we can diff forward from rather than needing to collect all diffs from genesis
* test for state trie builder
* if statediffing is on, lock tries in triedb until the statediffing service signals they are done using them
* fix mock blockchain; golint; bump patch
* increase maxRequestContentLength; bump patch
* log the sizes of the state objects we are sending
* CI build (#20)
* CI: run build on PR and on push to master
* CI: debug building geth
* CI: fix coping file
* CI: fix coping file v2
* CI: temporary upload file to release asset
* CI: get release upload_url by tag, upload asset to current relase
* CI: fix tag name
* fix ci build on statediff_at_anyblock-1.9.11 branch
* fix publishing assets in release
* use context deadline for timeout in eth_call
* collect and emit codehash=>code mappings for state objects
* subscription endpoint for retrieving all the codehash=>code mappings that exist at provided height
* Implement WriteStateDiffAt
* Writes state diffs directly to postgres
* Adds CLI flags to configure PG
* Refactors builder output with callbacks
* Copies refactored postgres handling code from ipld-eth-indexer
* rename PostgresCIDWriter.{index->upsert}*
* go.mod update
* rm unused
* cleanup
* output code & codehash iteratively
* had to rf some types for this
* prometheus metrics output
* duplicate recent eth-indexer changes
* migrations and metrics...
* [wip] prom.Init() here? another CLI flag?
* tidy & DRY
* statediff WriteLoop service + CLI flag
* [wip] update test mocks
* todo - do something meaningful to test write loop
* logging
* use geth log
* port tests to go testing
* drop ginkgo/gomega
* fix and cleanup tests
* fail before defer statement
* delete vendor/ dir
* fixes after rebase onto 1.9.23
* fix API registration
* use golang 1.15.5 version (#34)
* bump version meta; add 0.0.11 branch to actions
* bump version meta; update github actions workflows
* statediff: refactor metrics
* Remove redundant statediff/indexer/prom tooling and use existing
prometheus integration.
* "indexer" namespace for metrics
* add reporting loop for db metrics
* doc
* metrics for statediff stats
* metrics namespace/subsystem = statediff/{indexer,service}
* statediff: use a worker pool (for direct writes)
* fix test
* fix chain event subscription
* log tweaks
* func name
* unused import
* intermediate chain event channel for metrics
* update github actions; linting
* add poststate and status to receipt ipld indexes
* stateDiffFor endpoints for fetching or writing statediff object by blockhash; bump statediff version
* fixes after rebase on to v1.10.1
* update github actions and version meta; go fmt
* add leaf key to removed 'nodes'
* include Postgres migrations and schema
* service documentation
* touching up
update github actions after rebase
fix connection leak (misplaced defer) and perform proper rollback on errs
improve error logging; handle PushBlock internal err
* build docker image and publish it to Docker Hub on release
* add access list tx to unit tests
* MarshalBinary and UnmarshalBinary methods for receipt
* fix error caused by 2718 by using MarshalBinary instead of EncodeRLP methods
* ipld encoding/decoding tests
* update TxModel; add AccessListElementModel
* index tx type and access lists
* add access list metrics
* unit tests for tx_type and access list table
* unit tests for receipt marshal/unmarshal binary methods
* improve documentation of the encoding methods
* fix issue identified in linting
update github actions and version meta after rebase
unit test that fails undeterministically on eip2930 txs, giving same error we are seeing in prod
fix bug
Include genesis block state diff.
Fix linting issue.
documentation on versioning, rebasing, releasing; bump version meta
Add geth and statediff unit test to CI.
Set pgpassword in env.
Added comments.
Add new major branch to github action.
Fix failing test.
Fix lint errors.
Add support for Dynamic txn(EIP-1559).
Update version meta to 0.0.24
Verify block base fee in test.
Fix base_fee type and add backward compatible test.
Remove type definition for AccessListElementModel
Change basefee to int64/bigint.
block and uncle reward in PoA network = 0 (#87)
* in PoA networks there is no block and uncle rewards
* bump meta version
(cherry picked from commit b64ca14689689178b78f915c855fe75010d886c7)
Use Ropsten to test block reward.
Add Makefile target to build static linux binaries.
Strip symbol tables from static binaries.
Fix block_fee to support NULL values.
bump version meta.
Add new major branch to github action.
Add new major branch to github action.
Add new major branch to github action.
Add new major branch to github action.
rename doc.go to README.md
Create a seperate table for storing logs
Self review
Bump statediff version to 0.0.26.
add btree index to state/storage_cids.node_type; updated schema
Dedup receipt data.
Fix linter errors.
Address comments.
Bump statediff version to 0.0.27.
new cli flag for initializing db first time service is ran
only write Removed node ipld block (on db init) and reuse constant cid and mhkey
linting
test new handling of Removed nodes; don't require init flag
log metrics
Add new major branch to github action.
Fix build.
Update golang version in CI.
Use ipld-eth-db in testing.
Remove migration from repo.
Add new major branch to github action.
Use `GetTd` instead of `GetTdByHash`
https://github.com/ethereum/go-ethereum/commit/62891378274a58c84b9fa35f3c9b99155644b5eb
2019-01-28 21:31:01 +00:00
|
|
|
github.com/jmoiron/sqlx v1.2.0
|
2021-01-25 09:42:07 +00:00
|
|
|
github.com/julienschmidt/httprouter v1.2.0
|
2021-10-05 16:33:53 +00:00
|
|
|
github.com/karalabe/usb v0.0.0-20211005121534-4c5740d64559
|
2021-04-28 10:04:25 +00:00
|
|
|
github.com/kylelemons/godebug v1.1.0 // indirect
|
Statediffing geth
* Write state diff to CSV (#2)
* port statediff from https://github.com/jpmorganchase/quorum/blob/9b7fd9af8082795eeeb6863d9746f12b82dd5078/statediff/statediff.go; minor fixes
* integrating state diff extracting, building, and persisting into geth processes
* work towards persisting created statediffs in ipfs; based off github.com/vulcanize/eth-block-extractor
* Add a state diff service
* Remove diff extractor from blockchain
* Update imports
* Move statediff on/off check to geth cmd config
* Update starting state diff service
* Add debugging logs for creating diff
* Add statediff extractor and builder tests and small refactoring
* Start to write statediff to a CSV
* Restructure statediff directory
* Pull CSV publishing methods into their own file
* Reformatting due to go fmt
* Add gomega to vendor dir
* Remove testing focuses
* Update statediff tests to use golang test pkg
instead of ginkgo
- builder_test
- extractor_test
- publisher_test
* Use hexutil.Encode instead of deprecated common.ToHex
* Remove OldValue from DiffBigInt and DiffUint64 fields
* Update builder test
* Remove old storage value from updated accounts
* Remove old values from created/deleted accounts
* Update publisher to account for only storing current account values
* Update service loop and fetching previous block
* Update testing
- remove statediff ginkgo test suite file
- move mocks to their own dir
* Updates per go fmt
* Updates to tests
* Pass statediff mode and path in through cli
* Return filename from publisher
* Remove some duplication in builder
* Remove code field from state diff output
this is the contract byte code, and it can still be obtained by querying
the db by the codeHash
* Consolidate acct diff structs for updated & updated/deleted accts
* Include block number in csv filename
* Clean up error logging
* Cleanup formatting, spelling, etc
* Address PR comments
* Add contract address and storage value to csv
* Refactor accumulating account row in csv publisher
* Add DiffStorage struct
* Add storage key to csv
* Address PR comments
* Fix publisher to include rows for accounts that don't have store updates
* Update builder test after merging in release/1.8
* Update test contract to include storage on contract intialization
- so that we're able to test that storage diffing works for created and
deleted accounts (not just updated accounts).
* Factor out a common trie iterator method in builder
* Apply goimports to statediff
* Apply gosimple changes to statediff
* Gracefully exit geth command(#4)
* Statediff for full node (#6)
* Open a trie from the in-memory database
* Use a node's LeafKey as an identifier instead of the address
It was proving difficult to find look the address up from a given path
with a full node (sometimes the value wouldn't exist in the disk db).
So, instead, for now we are using the node's LeafKey with is a Keccak256
hash of the address, so if we know the address we can figure out which
LeafKey it matches up to.
* Make sure that statediff has been processed before pruning
* Use blockchain stateCache.OpenTrie for storage diffs
* Clean up log lines and remove unnecessary fields from builder
* Apply go fmt changes
* Add a sleep to the blockchain test
* refactoring/reorganizing packages
* refactoring statediff builder and types and adjusted to relay proofs and paths (still need to make this optional)
* refactoring state diff service and adding api which allows for streaming state diff payloads over an rpc websocket subscription
* make proofs and paths optional + compress service loop into single for loop (may be missing something here)
* option to process intermediate nodes
* make state diff rlp serializable
* cli parameter to limit statediffing to select account addresses + test
* review fixes and fixes for issues ran into in integration
* review fixes; proper method signature for api; adjust service so that statediff processing is halted/paused until there is at least one subscriber listening for the results
* adjust buffering to improve stability; doc.go; fix notifier
err handling
* relay receipts with the rest of the data + review fixes/changes
* rpc method to get statediff at specific block; requires archival node or the block be within the pruning range
* fix linter issues
* include total difficulty to the payload
* fix state diff builder: emit actual leaf nodes instead of value nodes; diff on the leaf not on the value; emit correct path for intermediate nodes
* adjust statediff builder tests to changes and extend to test intermediate nodes; golint
* add genesis block to test; handle block 0 in StateDiffAt
* rlp files for mainnet blocks 0-3, for tests
* builder test on mainnet blocks
* common.BytesToHash(path) => crypto.Keaccak256(hash) in builder; BytesToHash produces same hash for e.g. []byte{} and []byte{\x00} - prefix \x00 steps are inconsequential to the hash result
* complete tests for early mainnet blocks
* diff type for representing deleted accounts
* fix builder so that we handle account deletions properly and properly diff storage when an account is moved to a new path; update params
* remove cli params; moving them to subscriber defined
* remove unneeded bc methods
* update service and api; statediffing params are now defined by user through api rather than by service provider by cli
* update top level tests
* add ability to watch specific storage slots (leaf keys) only
* comments; explain logic
* update mainnet blocks test
* update api_test.go
* storage leafkey filter test
* cleanup chain maker
* adjust chain maker for tests to add an empty account in block1 and switch to EIP-158 afterwards (now we just need to generate enough accounts until one causes the empty account to be touched and removed post-EIP-158 so we can simulate and test that process...); also added 2 new blocks where more contract storage is set and old slots are set to zero so they are removed so we can test that
* found an account whose creation causes the empty account to be moved to a new path; this should count as 'touching; the empty account and cause it to be removed according to eip-158... but it doesn't
* use new contract in unit tests that has self-destruct ability, so we can test eip-158 since simply moving an account to new path doesn't count as 'touchin' it
* handle storage deletions
* tests for eip-158 account removal and storage value deletions; there is one edge case left to test where we remove 1 account when only two exist such that the remaining account is moved up and replaces the root branch node
* finish testing known edge cases
* add endpoint to fetch all state and storage nodes at a given blockheight; useful for generating a recent atate cache/snapshot that we can diff forward from rather than needing to collect all diffs from genesis
* test for state trie builder
* if statediffing is on, lock tries in triedb until the statediffing service signals they are done using them
* fix mock blockchain; golint; bump patch
* increase maxRequestContentLength; bump patch
* log the sizes of the state objects we are sending
* CI build (#20)
* CI: run build on PR and on push to master
* CI: debug building geth
* CI: fix coping file
* CI: fix coping file v2
* CI: temporary upload file to release asset
* CI: get release upload_url by tag, upload asset to current relase
* CI: fix tag name
* fix ci build on statediff_at_anyblock-1.9.11 branch
* fix publishing assets in release
* use context deadline for timeout in eth_call
* collect and emit codehash=>code mappings for state objects
* subscription endpoint for retrieving all the codehash=>code mappings that exist at provided height
* Implement WriteStateDiffAt
* Writes state diffs directly to postgres
* Adds CLI flags to configure PG
* Refactors builder output with callbacks
* Copies refactored postgres handling code from ipld-eth-indexer
* rename PostgresCIDWriter.{index->upsert}*
* go.mod update
* rm unused
* cleanup
* output code & codehash iteratively
* had to rf some types for this
* prometheus metrics output
* duplicate recent eth-indexer changes
* migrations and metrics...
* [wip] prom.Init() here? another CLI flag?
* tidy & DRY
* statediff WriteLoop service + CLI flag
* [wip] update test mocks
* todo - do something meaningful to test write loop
* logging
* use geth log
* port tests to go testing
* drop ginkgo/gomega
* fix and cleanup tests
* fail before defer statement
* delete vendor/ dir
* fixes after rebase onto 1.9.23
* fix API registration
* use golang 1.15.5 version (#34)
* bump version meta; add 0.0.11 branch to actions
* bump version meta; update github actions workflows
* statediff: refactor metrics
* Remove redundant statediff/indexer/prom tooling and use existing
prometheus integration.
* "indexer" namespace for metrics
* add reporting loop for db metrics
* doc
* metrics for statediff stats
* metrics namespace/subsystem = statediff/{indexer,service}
* statediff: use a worker pool (for direct writes)
* fix test
* fix chain event subscription
* log tweaks
* func name
* unused import
* intermediate chain event channel for metrics
* update github actions; linting
* add poststate and status to receipt ipld indexes
* stateDiffFor endpoints for fetching or writing statediff object by blockhash; bump statediff version
* fixes after rebase on to v1.10.1
* update github actions and version meta; go fmt
* add leaf key to removed 'nodes'
* include Postgres migrations and schema
* service documentation
* touching up
update github actions after rebase
fix connection leak (misplaced defer) and perform proper rollback on errs
improve error logging; handle PushBlock internal err
* build docker image and publish it to Docker Hub on release
* add access list tx to unit tests
* MarshalBinary and UnmarshalBinary methods for receipt
* fix error caused by 2718 by using MarshalBinary instead of EncodeRLP methods
* ipld encoding/decoding tests
* update TxModel; add AccessListElementModel
* index tx type and access lists
* add access list metrics
* unit tests for tx_type and access list table
* unit tests for receipt marshal/unmarshal binary methods
* improve documentation of the encoding methods
* fix issue identified in linting
update github actions and version meta after rebase
unit test that fails undeterministically on eip2930 txs, giving same error we are seeing in prod
fix bug
Include genesis block state diff.
Fix linting issue.
documentation on versioning, rebasing, releasing; bump version meta
Add geth and statediff unit test to CI.
Set pgpassword in env.
Added comments.
Add new major branch to github action.
Fix failing test.
Fix lint errors.
Add support for Dynamic txn(EIP-1559).
Update version meta to 0.0.24
Verify block base fee in test.
Fix base_fee type and add backward compatible test.
Remove type definition for AccessListElementModel
Change basefee to int64/bigint.
block and uncle reward in PoA network = 0 (#87)
* in PoA networks there is no block and uncle rewards
* bump meta version
(cherry picked from commit b64ca14689689178b78f915c855fe75010d886c7)
Use Ropsten to test block reward.
Add Makefile target to build static linux binaries.
Strip symbol tables from static binaries.
Fix block_fee to support NULL values.
bump version meta.
Add new major branch to github action.
Add new major branch to github action.
Add new major branch to github action.
Add new major branch to github action.
rename doc.go to README.md
Create a seperate table for storing logs
Self review
Bump statediff version to 0.0.26.
add btree index to state/storage_cids.node_type; updated schema
Dedup receipt data.
Fix linter errors.
Address comments.
Bump statediff version to 0.0.27.
new cli flag for initializing db first time service is ran
only write Removed node ipld block (on db init) and reuse constant cid and mhkey
linting
test new handling of Removed nodes; don't require init flag
log metrics
Add new major branch to github action.
Fix build.
Update golang version in CI.
Use ipld-eth-db in testing.
Remove migration from repo.
Add new major branch to github action.
Use `GetTd` instead of `GetTdByHash`
https://github.com/ethereum/go-ethereum/commit/62891378274a58c84b9fa35f3c9b99155644b5eb
2019-01-28 21:31:01 +00:00
|
|
|
github.com/lib/pq v1.10.2
|
2021-08-17 16:40:14 +00:00
|
|
|
github.com/mattn/go-colorable v0.1.8
|
|
|
|
|
github.com/mattn/go-isatty v0.0.12
|
Statediffing geth
* Write state diff to CSV (#2)
* port statediff from https://github.com/jpmorganchase/quorum/blob/9b7fd9af8082795eeeb6863d9746f12b82dd5078/statediff/statediff.go; minor fixes
* integrating state diff extracting, building, and persisting into geth processes
* work towards persisting created statediffs in ipfs; based off github.com/vulcanize/eth-block-extractor
* Add a state diff service
* Remove diff extractor from blockchain
* Update imports
* Move statediff on/off check to geth cmd config
* Update starting state diff service
* Add debugging logs for creating diff
* Add statediff extractor and builder tests and small refactoring
* Start to write statediff to a CSV
* Restructure statediff directory
* Pull CSV publishing methods into their own file
* Reformatting due to go fmt
* Add gomega to vendor dir
* Remove testing focuses
* Update statediff tests to use golang test pkg
instead of ginkgo
- builder_test
- extractor_test
- publisher_test
* Use hexutil.Encode instead of deprecated common.ToHex
* Remove OldValue from DiffBigInt and DiffUint64 fields
* Update builder test
* Remove old storage value from updated accounts
* Remove old values from created/deleted accounts
* Update publisher to account for only storing current account values
* Update service loop and fetching previous block
* Update testing
- remove statediff ginkgo test suite file
- move mocks to their own dir
* Updates per go fmt
* Updates to tests
* Pass statediff mode and path in through cli
* Return filename from publisher
* Remove some duplication in builder
* Remove code field from state diff output
this is the contract byte code, and it can still be obtained by querying
the db by the codeHash
* Consolidate acct diff structs for updated & updated/deleted accts
* Include block number in csv filename
* Clean up error logging
* Cleanup formatting, spelling, etc
* Address PR comments
* Add contract address and storage value to csv
* Refactor accumulating account row in csv publisher
* Add DiffStorage struct
* Add storage key to csv
* Address PR comments
* Fix publisher to include rows for accounts that don't have store updates
* Update builder test after merging in release/1.8
* Update test contract to include storage on contract intialization
- so that we're able to test that storage diffing works for created and
deleted accounts (not just updated accounts).
* Factor out a common trie iterator method in builder
* Apply goimports to statediff
* Apply gosimple changes to statediff
* Gracefully exit geth command(#4)
* Statediff for full node (#6)
* Open a trie from the in-memory database
* Use a node's LeafKey as an identifier instead of the address
It was proving difficult to find look the address up from a given path
with a full node (sometimes the value wouldn't exist in the disk db).
So, instead, for now we are using the node's LeafKey with is a Keccak256
hash of the address, so if we know the address we can figure out which
LeafKey it matches up to.
* Make sure that statediff has been processed before pruning
* Use blockchain stateCache.OpenTrie for storage diffs
* Clean up log lines and remove unnecessary fields from builder
* Apply go fmt changes
* Add a sleep to the blockchain test
* refactoring/reorganizing packages
* refactoring statediff builder and types and adjusted to relay proofs and paths (still need to make this optional)
* refactoring state diff service and adding api which allows for streaming state diff payloads over an rpc websocket subscription
* make proofs and paths optional + compress service loop into single for loop (may be missing something here)
* option to process intermediate nodes
* make state diff rlp serializable
* cli parameter to limit statediffing to select account addresses + test
* review fixes and fixes for issues ran into in integration
* review fixes; proper method signature for api; adjust service so that statediff processing is halted/paused until there is at least one subscriber listening for the results
* adjust buffering to improve stability; doc.go; fix notifier
err handling
* relay receipts with the rest of the data + review fixes/changes
* rpc method to get statediff at specific block; requires archival node or the block be within the pruning range
* fix linter issues
* include total difficulty to the payload
* fix state diff builder: emit actual leaf nodes instead of value nodes; diff on the leaf not on the value; emit correct path for intermediate nodes
* adjust statediff builder tests to changes and extend to test intermediate nodes; golint
* add genesis block to test; handle block 0 in StateDiffAt
* rlp files for mainnet blocks 0-3, for tests
* builder test on mainnet blocks
* common.BytesToHash(path) => crypto.Keaccak256(hash) in builder; BytesToHash produces same hash for e.g. []byte{} and []byte{\x00} - prefix \x00 steps are inconsequential to the hash result
* complete tests for early mainnet blocks
* diff type for representing deleted accounts
* fix builder so that we handle account deletions properly and properly diff storage when an account is moved to a new path; update params
* remove cli params; moving them to subscriber defined
* remove unneeded bc methods
* update service and api; statediffing params are now defined by user through api rather than by service provider by cli
* update top level tests
* add ability to watch specific storage slots (leaf keys) only
* comments; explain logic
* update mainnet blocks test
* update api_test.go
* storage leafkey filter test
* cleanup chain maker
* adjust chain maker for tests to add an empty account in block1 and switch to EIP-158 afterwards (now we just need to generate enough accounts until one causes the empty account to be touched and removed post-EIP-158 so we can simulate and test that process...); also added 2 new blocks where more contract storage is set and old slots are set to zero so they are removed so we can test that
* found an account whose creation causes the empty account to be moved to a new path; this should count as 'touching; the empty account and cause it to be removed according to eip-158... but it doesn't
* use new contract in unit tests that has self-destruct ability, so we can test eip-158 since simply moving an account to new path doesn't count as 'touchin' it
* handle storage deletions
* tests for eip-158 account removal and storage value deletions; there is one edge case left to test where we remove 1 account when only two exist such that the remaining account is moved up and replaces the root branch node
* finish testing known edge cases
* add endpoint to fetch all state and storage nodes at a given blockheight; useful for generating a recent atate cache/snapshot that we can diff forward from rather than needing to collect all diffs from genesis
* test for state trie builder
* if statediffing is on, lock tries in triedb until the statediffing service signals they are done using them
* fix mock blockchain; golint; bump patch
* increase maxRequestContentLength; bump patch
* log the sizes of the state objects we are sending
* CI build (#20)
* CI: run build on PR and on push to master
* CI: debug building geth
* CI: fix coping file
* CI: fix coping file v2
* CI: temporary upload file to release asset
* CI: get release upload_url by tag, upload asset to current relase
* CI: fix tag name
* fix ci build on statediff_at_anyblock-1.9.11 branch
* fix publishing assets in release
* use context deadline for timeout in eth_call
* collect and emit codehash=>code mappings for state objects
* subscription endpoint for retrieving all the codehash=>code mappings that exist at provided height
* Implement WriteStateDiffAt
* Writes state diffs directly to postgres
* Adds CLI flags to configure PG
* Refactors builder output with callbacks
* Copies refactored postgres handling code from ipld-eth-indexer
* rename PostgresCIDWriter.{index->upsert}*
* go.mod update
* rm unused
* cleanup
* output code & codehash iteratively
* had to rf some types for this
* prometheus metrics output
* duplicate recent eth-indexer changes
* migrations and metrics...
* [wip] prom.Init() here? another CLI flag?
* tidy & DRY
* statediff WriteLoop service + CLI flag
* [wip] update test mocks
* todo - do something meaningful to test write loop
* logging
* use geth log
* port tests to go testing
* drop ginkgo/gomega
* fix and cleanup tests
* fail before defer statement
* delete vendor/ dir
* fixes after rebase onto 1.9.23
* fix API registration
* use golang 1.15.5 version (#34)
* bump version meta; add 0.0.11 branch to actions
* bump version meta; update github actions workflows
* statediff: refactor metrics
* Remove redundant statediff/indexer/prom tooling and use existing
prometheus integration.
* "indexer" namespace for metrics
* add reporting loop for db metrics
* doc
* metrics for statediff stats
* metrics namespace/subsystem = statediff/{indexer,service}
* statediff: use a worker pool (for direct writes)
* fix test
* fix chain event subscription
* log tweaks
* func name
* unused import
* intermediate chain event channel for metrics
* update github actions; linting
* add poststate and status to receipt ipld indexes
* stateDiffFor endpoints for fetching or writing statediff object by blockhash; bump statediff version
* fixes after rebase on to v1.10.1
* update github actions and version meta; go fmt
* add leaf key to removed 'nodes'
* include Postgres migrations and schema
* service documentation
* touching up
update github actions after rebase
fix connection leak (misplaced defer) and perform proper rollback on errs
improve error logging; handle PushBlock internal err
* build docker image and publish it to Docker Hub on release
* add access list tx to unit tests
* MarshalBinary and UnmarshalBinary methods for receipt
* fix error caused by 2718 by using MarshalBinary instead of EncodeRLP methods
* ipld encoding/decoding tests
* update TxModel; add AccessListElementModel
* index tx type and access lists
* add access list metrics
* unit tests for tx_type and access list table
* unit tests for receipt marshal/unmarshal binary methods
* improve documentation of the encoding methods
* fix issue identified in linting
update github actions and version meta after rebase
unit test that fails undeterministically on eip2930 txs, giving same error we are seeing in prod
fix bug
Include genesis block state diff.
Fix linting issue.
documentation on versioning, rebasing, releasing; bump version meta
Add geth and statediff unit test to CI.
Set pgpassword in env.
Added comments.
Add new major branch to github action.
Fix failing test.
Fix lint errors.
Add support for Dynamic txn(EIP-1559).
Update version meta to 0.0.24
Verify block base fee in test.
Fix base_fee type and add backward compatible test.
Remove type definition for AccessListElementModel
Change basefee to int64/bigint.
block and uncle reward in PoA network = 0 (#87)
* in PoA networks there is no block and uncle rewards
* bump meta version
(cherry picked from commit b64ca14689689178b78f915c855fe75010d886c7)
Use Ropsten to test block reward.
Add Makefile target to build static linux binaries.
Strip symbol tables from static binaries.
Fix block_fee to support NULL values.
bump version meta.
Add new major branch to github action.
Add new major branch to github action.
Add new major branch to github action.
Add new major branch to github action.
rename doc.go to README.md
Create a seperate table for storing logs
Self review
Bump statediff version to 0.0.26.
add btree index to state/storage_cids.node_type; updated schema
Dedup receipt data.
Fix linter errors.
Address comments.
Bump statediff version to 0.0.27.
new cli flag for initializing db first time service is ran
only write Removed node ipld block (on db init) and reuse constant cid and mhkey
linting
test new handling of Removed nodes; don't require init flag
log metrics
Add new major branch to github action.
Fix build.
Update golang version in CI.
Use ipld-eth-db in testing.
Remove migration from repo.
Add new major branch to github action.
Use `GetTd` instead of `GetTdByHash`
https://github.com/ethereum/go-ethereum/commit/62891378274a58c84b9fa35f3c9b99155644b5eb
2019-01-28 21:31:01 +00:00
|
|
|
github.com/multiformats/go-multihash v0.0.14
|
2021-04-28 10:04:25 +00:00
|
|
|
github.com/naoina/go-stringutil v0.1.0 // indirect
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/naoina/toml v0.1.2-0.20170918210437-9fafd6967416
|
2021-03-26 21:15:20 +00:00
|
|
|
github.com/olekukonko/tablewriter v0.0.5
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/peterh/liner v1.1.1-0.20190123174540-a2c9a5303de7
|
all: bloom-filter based pruning mechanism (#21724)
* cmd, core, tests: initial state pruner
core: fix db inspector
cmd/geth: add verify-state
cmd/geth: add verification tool
core/rawdb: implement flatdb
cmd, core: fix rebase
core/state: use new contract code layout
core/state/pruner: avoid deleting genesis state
cmd/geth: add helper function
core, cmd: fix extract genesis
core: minor fixes
contracts: remove useless
core/state/snapshot: plugin stacktrie
core: polish
core/state/snapshot: iterate storage concurrently
core/state/snapshot: fix iteration
core: add comments
core/state/snapshot: polish code
core/state: polish
core/state/snapshot: rebase
core/rawdb: add comments
core/rawdb: fix tests
core/rawdb: improve tests
core/state/snapshot: fix concurrent iteration
core/state: run pruning during the recovery
core, trie: implement martin's idea
core, eth: delete flatdb and polish pruner
trie: fix import
core/state/pruner: add log
core/state/pruner: fix issues
core/state/pruner: don't read back
core/state/pruner: fix contract code write
core/state/pruner: check root node presence
cmd, core: polish log
core/state: use HEAD-127 as the target
core/state/snapshot: improve tests
cmd/geth: fix verification tool
cmd/geth: use HEAD as the verification default target
all: replace the bloomfilter with martin's fork
cmd, core: polish code
core, cmd: forcibly delete state root
core/state/pruner: add hash64
core/state/pruner: fix blacklist
core/state: remove blacklist
cmd, core: delete trie clean cache before pruning
cmd, core: fix lint
cmd, core: fix rebase
core/state: fix the special case for clique networks
core/state/snapshot: remove useless code
core/state/pruner: capping the snapshot after pruning
cmd, core, eth: fixes
core/rawdb: update db inspector
cmd/geth: polish code
core/state/pruner: fsync bloom filter
cmd, core: print warning log
core/state/pruner: adjust the parameters for bloom filter
cmd, core: create the bloom filter by size
core: polish
core/state/pruner: sanitize invalid bloomfilter size
cmd: address comments
cmd/geth: address comments
cmd/geth: address comment
core/state/pruner: address comments
core/state/pruner: rename homedir to datadir
cmd, core: address comments
core/state/pruner: address comment
core/state: address comments
core, cmd, tests: address comments
core: address comments
core/state/pruner: release the iterator after each commit
core/state/pruner: improve pruner
cmd, core: adjust bloom paramters
core/state/pruner: fix lint
core/state/pruner: fix tests
core: fix rebase
core/state/pruner: remove atomic rename
core/state/pruner: address comments
all: run go mod tidy
core/state/pruner: avoid false-positive for the middle state roots
core/state/pruner: add checks for middle roots
cmd/geth: replace crit with error
* core/state/pruner: fix lint
* core: drop legacy bloom filter
* core/state/snapshot: improve pruner
* core/state/snapshot: polish concurrent logs to report ETA vs. hashes
* core/state/pruner: add progress report for pruning and compaction too
* core: fix snapshot test API
* core/state: fix some pruning logs
* core/state/pruner: support recovering from bloom flush fail
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-02-08 11:16:30 +00:00
|
|
|
github.com/prometheus/tsdb v0.7.1
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/rjeczalik/notify v0.9.1
|
2021-01-25 09:42:07 +00:00
|
|
|
github.com/rs/cors v1.7.0
|
2021-04-20 08:54:41 +00:00
|
|
|
github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/status-im/keycard-go v0.0.0-20190316090335-8537d3370df4
|
2021-01-25 09:42:07 +00:00
|
|
|
github.com/stretchr/testify v1.7.0
|
2021-08-24 19:02:46 +00:00
|
|
|
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7
|
2021-04-20 08:54:41 +00:00
|
|
|
github.com/tklauser/go-sysconf v0.3.5 // indirect
|
2019-11-18 12:22:36 +00:00
|
|
|
github.com/tyler-smith/go-bip39 v1.0.1-0.20181017060643-dbb3b84ba2ef
|
2021-11-10 23:02:26 +00:00
|
|
|
golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97
|
2021-05-04 19:45:21 +00:00
|
|
|
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
|
2021-08-23 11:45:16 +00:00
|
|
|
golang.org/x/sys v0.0.0-20210816183151-1e6c022a8912
|
2021-07-29 13:20:45 +00:00
|
|
|
golang.org/x/text v0.3.6
|
2021-08-17 16:40:14 +00:00
|
|
|
golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba
|
Statediffing geth
* Write state diff to CSV (#2)
* port statediff from https://github.com/jpmorganchase/quorum/blob/9b7fd9af8082795eeeb6863d9746f12b82dd5078/statediff/statediff.go; minor fixes
* integrating state diff extracting, building, and persisting into geth processes
* work towards persisting created statediffs in ipfs; based off github.com/vulcanize/eth-block-extractor
* Add a state diff service
* Remove diff extractor from blockchain
* Update imports
* Move statediff on/off check to geth cmd config
* Update starting state diff service
* Add debugging logs for creating diff
* Add statediff extractor and builder tests and small refactoring
* Start to write statediff to a CSV
* Restructure statediff directory
* Pull CSV publishing methods into their own file
* Reformatting due to go fmt
* Add gomega to vendor dir
* Remove testing focuses
* Update statediff tests to use golang test pkg
instead of ginkgo
- builder_test
- extractor_test
- publisher_test
* Use hexutil.Encode instead of deprecated common.ToHex
* Remove OldValue from DiffBigInt and DiffUint64 fields
* Update builder test
* Remove old storage value from updated accounts
* Remove old values from created/deleted accounts
* Update publisher to account for only storing current account values
* Update service loop and fetching previous block
* Update testing
- remove statediff ginkgo test suite file
- move mocks to their own dir
* Updates per go fmt
* Updates to tests
* Pass statediff mode and path in through cli
* Return filename from publisher
* Remove some duplication in builder
* Remove code field from state diff output
this is the contract byte code, and it can still be obtained by querying
the db by the codeHash
* Consolidate acct diff structs for updated & updated/deleted accts
* Include block number in csv filename
* Clean up error logging
* Cleanup formatting, spelling, etc
* Address PR comments
* Add contract address and storage value to csv
* Refactor accumulating account row in csv publisher
* Add DiffStorage struct
* Add storage key to csv
* Address PR comments
* Fix publisher to include rows for accounts that don't have store updates
* Update builder test after merging in release/1.8
* Update test contract to include storage on contract intialization
- so that we're able to test that storage diffing works for created and
deleted accounts (not just updated accounts).
* Factor out a common trie iterator method in builder
* Apply goimports to statediff
* Apply gosimple changes to statediff
* Gracefully exit geth command(#4)
* Statediff for full node (#6)
* Open a trie from the in-memory database
* Use a node's LeafKey as an identifier instead of the address
It was proving difficult to find look the address up from a given path
with a full node (sometimes the value wouldn't exist in the disk db).
So, instead, for now we are using the node's LeafKey with is a Keccak256
hash of the address, so if we know the address we can figure out which
LeafKey it matches up to.
* Make sure that statediff has been processed before pruning
* Use blockchain stateCache.OpenTrie for storage diffs
* Clean up log lines and remove unnecessary fields from builder
* Apply go fmt changes
* Add a sleep to the blockchain test
* refactoring/reorganizing packages
* refactoring statediff builder and types and adjusted to relay proofs and paths (still need to make this optional)
* refactoring state diff service and adding api which allows for streaming state diff payloads over an rpc websocket subscription
* make proofs and paths optional + compress service loop into single for loop (may be missing something here)
* option to process intermediate nodes
* make state diff rlp serializable
* cli parameter to limit statediffing to select account addresses + test
* review fixes and fixes for issues ran into in integration
* review fixes; proper method signature for api; adjust service so that statediff processing is halted/paused until there is at least one subscriber listening for the results
* adjust buffering to improve stability; doc.go; fix notifier
err handling
* relay receipts with the rest of the data + review fixes/changes
* rpc method to get statediff at specific block; requires archival node or the block be within the pruning range
* fix linter issues
* include total difficulty to the payload
* fix state diff builder: emit actual leaf nodes instead of value nodes; diff on the leaf not on the value; emit correct path for intermediate nodes
* adjust statediff builder tests to changes and extend to test intermediate nodes; golint
* add genesis block to test; handle block 0 in StateDiffAt
* rlp files for mainnet blocks 0-3, for tests
* builder test on mainnet blocks
* common.BytesToHash(path) => crypto.Keaccak256(hash) in builder; BytesToHash produces same hash for e.g. []byte{} and []byte{\x00} - prefix \x00 steps are inconsequential to the hash result
* complete tests for early mainnet blocks
* diff type for representing deleted accounts
* fix builder so that we handle account deletions properly and properly diff storage when an account is moved to a new path; update params
* remove cli params; moving them to subscriber defined
* remove unneeded bc methods
* update service and api; statediffing params are now defined by user through api rather than by service provider by cli
* update top level tests
* add ability to watch specific storage slots (leaf keys) only
* comments; explain logic
* update mainnet blocks test
* update api_test.go
* storage leafkey filter test
* cleanup chain maker
* adjust chain maker for tests to add an empty account in block1 and switch to EIP-158 afterwards (now we just need to generate enough accounts until one causes the empty account to be touched and removed post-EIP-158 so we can simulate and test that process...); also added 2 new blocks where more contract storage is set and old slots are set to zero so they are removed so we can test that
* found an account whose creation causes the empty account to be moved to a new path; this should count as 'touching; the empty account and cause it to be removed according to eip-158... but it doesn't
* use new contract in unit tests that has self-destruct ability, so we can test eip-158 since simply moving an account to new path doesn't count as 'touchin' it
* handle storage deletions
* tests for eip-158 account removal and storage value deletions; there is one edge case left to test where we remove 1 account when only two exist such that the remaining account is moved up and replaces the root branch node
* finish testing known edge cases
* add endpoint to fetch all state and storage nodes at a given blockheight; useful for generating a recent atate cache/snapshot that we can diff forward from rather than needing to collect all diffs from genesis
* test for state trie builder
* if statediffing is on, lock tries in triedb until the statediffing service signals they are done using them
* fix mock blockchain; golint; bump patch
* increase maxRequestContentLength; bump patch
* log the sizes of the state objects we are sending
* CI build (#20)
* CI: run build on PR and on push to master
* CI: debug building geth
* CI: fix coping file
* CI: fix coping file v2
* CI: temporary upload file to release asset
* CI: get release upload_url by tag, upload asset to current relase
* CI: fix tag name
* fix ci build on statediff_at_anyblock-1.9.11 branch
* fix publishing assets in release
* use context deadline for timeout in eth_call
* collect and emit codehash=>code mappings for state objects
* subscription endpoint for retrieving all the codehash=>code mappings that exist at provided height
* Implement WriteStateDiffAt
* Writes state diffs directly to postgres
* Adds CLI flags to configure PG
* Refactors builder output with callbacks
* Copies refactored postgres handling code from ipld-eth-indexer
* rename PostgresCIDWriter.{index->upsert}*
* go.mod update
* rm unused
* cleanup
* output code & codehash iteratively
* had to rf some types for this
* prometheus metrics output
* duplicate recent eth-indexer changes
* migrations and metrics...
* [wip] prom.Init() here? another CLI flag?
* tidy & DRY
* statediff WriteLoop service + CLI flag
* [wip] update test mocks
* todo - do something meaningful to test write loop
* logging
* use geth log
* port tests to go testing
* drop ginkgo/gomega
* fix and cleanup tests
* fail before defer statement
* delete vendor/ dir
* fixes after rebase onto 1.9.23
* fix API registration
* use golang 1.15.5 version (#34)
* bump version meta; add 0.0.11 branch to actions
* bump version meta; update github actions workflows
* statediff: refactor metrics
* Remove redundant statediff/indexer/prom tooling and use existing
prometheus integration.
* "indexer" namespace for metrics
* add reporting loop for db metrics
* doc
* metrics for statediff stats
* metrics namespace/subsystem = statediff/{indexer,service}
* statediff: use a worker pool (for direct writes)
* fix test
* fix chain event subscription
* log tweaks
* func name
* unused import
* intermediate chain event channel for metrics
* update github actions; linting
* add poststate and status to receipt ipld indexes
* stateDiffFor endpoints for fetching or writing statediff object by blockhash; bump statediff version
* fixes after rebase on to v1.10.1
* update github actions and version meta; go fmt
* add leaf key to removed 'nodes'
* include Postgres migrations and schema
* service documentation
* touching up
update github actions after rebase
fix connection leak (misplaced defer) and perform proper rollback on errs
improve error logging; handle PushBlock internal err
* build docker image and publish it to Docker Hub on release
* add access list tx to unit tests
* MarshalBinary and UnmarshalBinary methods for receipt
* fix error caused by 2718 by using MarshalBinary instead of EncodeRLP methods
* ipld encoding/decoding tests
* update TxModel; add AccessListElementModel
* index tx type and access lists
* add access list metrics
* unit tests for tx_type and access list table
* unit tests for receipt marshal/unmarshal binary methods
* improve documentation of the encoding methods
* fix issue identified in linting
update github actions and version meta after rebase
unit test that fails undeterministically on eip2930 txs, giving same error we are seeing in prod
fix bug
Include genesis block state diff.
Fix linting issue.
documentation on versioning, rebasing, releasing; bump version meta
Add geth and statediff unit test to CI.
Set pgpassword in env.
Added comments.
Add new major branch to github action.
Fix failing test.
Fix lint errors.
Add support for Dynamic txn(EIP-1559).
Update version meta to 0.0.24
Verify block base fee in test.
Fix base_fee type and add backward compatible test.
Remove type definition for AccessListElementModel
Change basefee to int64/bigint.
block and uncle reward in PoA network = 0 (#87)
* in PoA networks there is no block and uncle rewards
* bump meta version
(cherry picked from commit b64ca14689689178b78f915c855fe75010d886c7)
Use Ropsten to test block reward.
Add Makefile target to build static linux binaries.
Strip symbol tables from static binaries.
Fix block_fee to support NULL values.
bump version meta.
Add new major branch to github action.
Add new major branch to github action.
Add new major branch to github action.
Add new major branch to github action.
rename doc.go to README.md
Create a seperate table for storing logs
Self review
Bump statediff version to 0.0.26.
add btree index to state/storage_cids.node_type; updated schema
Dedup receipt data.
Fix linter errors.
Address comments.
Bump statediff version to 0.0.27.
new cli flag for initializing db first time service is ran
only write Removed node ipld block (on db init) and reuse constant cid and mhkey
linting
test new handling of Removed nodes; don't require init flag
log metrics
Add new major branch to github action.
Fix build.
Update golang version in CI.
Use ipld-eth-db in testing.
Remove migration from repo.
Add new major branch to github action.
Use `GetTd` instead of `GetTdByHash`
https://github.com/ethereum/go-ethereum/commit/62891378274a58c84b9fa35f3c9b99155644b5eb
2019-01-28 21:31:01 +00:00
|
|
|
golang.org/x/tools v0.1.2 // indirect
|
2019-11-18 12:22:36 +00:00
|
|
|
gopkg.in/natefinch/npipe.v2 v2.0.0-20160621034901-c1b8fa8bdcce
|
2020-06-19 13:43:52 +00:00
|
|
|
gopkg.in/olebedev/go-duktape.v3 v3.0.0-20200619000410-60c24ae608a6
|
2019-11-18 12:22:36 +00:00
|
|
|
gopkg.in/urfave/cli.v1 v1.20.0
|
2021-04-28 10:04:25 +00:00
|
|
|
gotest.tools v2.2.0+incompatible // indirect
|
2019-11-18 12:22:36 +00:00
|
|
|
)
|
