* Statediff Geth
Handle conflicts (#244)
* Handle conflicts
* Update go mod file versions
* Make lint changes
Disassociate block number from the indexer object
Update ipld-eth-db ref
Refactor builder code to make it reusable
Use prefix comparison for account selective statediffing
Update builder unit tests
Add mode to write to CSV files in statediff file writer (#249)
* Change file writing mode to csv files
* Implement writer interface for file indexer
* Implement option for csv or sql in file mode
* Close files in CSV writer
* Add tests for CSV file mode
* Implement CSV file for watched addresses
* Separate test configs for CSV and SQL
* Refactor common code for file indexer tests
Update indexer to include block hash in receipts and logs (#256)
* Update indexer to include block hash in receipts and logs
* Upgrade ipld-eth-db image in docker-compose to run tests
Use watched addresses from direct indexing params by default while serving statediff APIs (#262)
* Use watched addresses from direct indexing params in statediff APIs by default
* Avoid using indexer object when direct indexing is off
* Add nil check before accessing watched addresses from direct indexing params
Rebase missed these changes needed at 1.10.20
Flags cleanup for CLI changes and linter complaints
Linter appeasements to achieve perfection
enforce go 1.18 for check (#267)
* enforce go 1.18 for check
* tests on 1.18 as well
* adding db yml for possible change in docker-compose behavior in yml parsing
Add indexer tests for handling non canonical blocks (#254)
* Add indexer tests for header and transactions in a non canonical block
* Add indexer tests for receipts in a non-canonical block and refactor
* Add indexer tests for logs in a non-canonical block
* Add indexer tests for state and storage nodes in a non-canonical block
* Add indexer tests for non-canonical block at another height
* Avoid passing address of a pointer
* Update refs in GitHub workflow
* Add genesis file path to stack-orchestrator config in GitHub workflow
* Add descriptive comments
fix non-deterministic ordering in unit tests
Refactor indexer tests to avoid duplicate code (#270)
* Refactor indexer tests to avoid duplicate code
* Refactor file mode indexer tests
* Fix expected db stats for sqlx after tx closure
* Refactor indexer tests for legacy block
* Refactor mainnet indexer tests
* Refactor tests for watched addressess methods
* Fix query in legacy indexer test
rebase and resolve onto 1.10.23... still error out of index related to GetLeafKeys
changed trie.Commit behavior was subtle about not not flushing to disk without an Update
* no merge nodeset throws nil
* linter appeasement
Co-authored-by: Abdul Rabbani <abdulrabbani00@gmail.com>
Cerc refactor (#281)
* first pass cerc refactor in cicd
* 1st attempt to publish binary to git.vdb.to from github release
* docker build step mangled
* docker build step mangled
* wrong username for docker login... which still succeeded
* circcicd is not cerccicd
* bad hostname
* core, trie: flush preimages to db on database close
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* rename Close to CommitPreimages for clarity
* core, trie: nitpick fixes
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* core: use TryGetAccount to read where TryUpdateAccount has been used to write
* Gary's review feedback
* implement Gary's suggestion
* fix bug + rename NewSecure into NewStateTrie
* trie: add backwards-compatibility aliases for SecureTrie
* Update database.go
* make the linter happy
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* Remove locking in (*BlockChain).ExportN
Since ExportN is read-only, it shouldn't need the lock. (?)
* Add hash check to detect reorgs during export.
* fix check order
* Update blockchain.go
* Update blockchain.go
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* core: recover the state in SetChainHead if the head state is missing
* core: disable test logging
* core: address comment from martin
* core: improve log level in case state is recovered
* core, eth, les, light: rename SetChainHead to SetCanonical
* eth/downloader: implement beacon sync
* eth/downloader: fix a crash if the beacon chain is reduced in length
* eth/downloader: fix beacon sync start/stop thrashing data race
* eth/downloader: use a non-nil pivot even in degenerate sync requests
* eth/downloader: don't touch internal state on beacon Head retrieval
* eth/downloader: fix spelling mistakes
* eth/downloader: fix some typos
* eth: integrate legacy/beacon sync switchover and UX
* eth: handle UX wise being stuck on post-merge TTD
* core, eth: integrate the beacon client with the beacon sync
* eth/catalyst: make some warning messages nicer
* eth/downloader: remove Ethereum 1&2 notions in favor of merge
* core/beacon, eth: clean up engine API returns a bit
* eth/downloader: add skeleton extension tests
* eth/catalyst: keep non-kiln spec, handle mining on ttd
* eth/downloader: add beacon header retrieval tests
* eth: fixed spelling, commented failing tests out
* eth/downloader: review fixes
* eth/downloader: drop peers failing to deliver beacon headers
* core/rawdb: track beacon sync data in db inspect
* eth: fix review concerns
* internal/web3ext: nit
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* core/rawdb, cmd, ethdb, eth: implement freezer tail deletion
* core/rawdb: address comments from martin and sina
* core/rawdb: fixes cornercase in tail deletion
* core/rawdb: separate metadata into a standalone file
* core/rawdb: remove unused code
* core/rawdb: add random test
* core/rawdb: polish code
* core/rawdb: fsync meta file before manipulating the index
* core/rawdb: fix typo
* core/rawdb: address comments
This PR fixes a special corner case in transaction indexing.
When the chain is rewound by SetHead to a historical point which is even lower than the transaction indexes tail, then system will report Failed to decode block body error all the time, because the relevant blocks are already deleted.
In order to avoid this "non-critical-but-annoying" issue, we can recap the indexing target to head+1(to is excluded, so it means indexing transactions from 0 to head).
* all: work for eth1/2 transtition
* consensus/beacon, eth: change beacon difficulty to 0
* eth: updates
* all: add terminalBlockDifficulty config, fix rebasing issues
* eth: implemented merge interop spec
* internal/ethapi: update to v1.0.0.alpha.2
This commit updates the code to the new spec, moving payloadId into
it's own object. It also fixes an issue with finalizing an empty blockhash.
It also properly sets the basefee
* all: sync polishes, other fixes + refactors
* core, eth: correct semantics for LeavePoW, EnterPoS
* core: fixed rebasing artifacts
* core: light: performance improvements
* core: use keyed field (f)
* core: eth: fix compilation issues + tests
* eth/catalyst: dbetter error codes
* all: move Merger to consensus/, remove reliance on it in bc
* all: renamed EnterPoS and LeavePoW to ReachTDD and FinalizePoS
* core: make mergelogs a function
* core: use InsertChain instead of InsertBlock
* les: drop merger from lightchain object
* consensus: add merger
* core: recoverAncestors in catalyst mode
* core: fix nitpick
* all: removed merger from beacon, use TTD, nitpicks
* consensus: eth: add docstring, removed unnecessary code duplication
* consensus/beacon: better comment
* all: easy to fix nitpicks by karalabe
* consensus/beacon: verify known headers to be sure
* core: comments
* core: eth: don't drop peers who advertise blocks, nitpicks
* core: never add beacon blocks to the future queue
* core: fixed nitpicks
* consensus/beacon: simplify IsTTDReached check
* consensus/beacon: correct IsTTDReached check
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
It is because write known block only checks block and state without snapshot, which could lead to gap between newest snapshot and newest block state. However, new blocks which would cause snapshot to become fixed were ignored, since state was already known.
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
This removes some code:
- The clique engine calculated the snapshot twice when verifying headers/blocks.
- The method GetBlockHashesFromHash in Header/Block/Lightchain was only used by tests. It
is now removed from the API.
- The method GetTdByHash internally looked up the number before calling GetTd(hash, num).
In many cases, callers already had the number, and used this method just because it has a
shorter name. I have removed the method to make the API surface smaller.
This change removes misuses of sync.WaitGroup in BlockChain. Before this change,
block insertion modified the WaitGroup counter in order to ensure that Stop would wait
for pending operations to complete. This was racy and could even lead to crashes
if Stop was called at an unfortunate time. The issue is resolved by adding a specialized
'closable' mutex, which prevents chain modifications after stopping while also
synchronizing writers with each other.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change is a rewrite of the freezer code.
When writing ancient chain data to the freezer, the previous version first encoded each
individual item to a temporary buffer, then wrote the buffer. For small item sizes (for
example, in the block hash freezer table), this strategy causes a lot of system calls for
writing tiny chunks of data. It also allocated a lot of temporary []byte buffers.
In the new version, we instead encode multiple items into a re-useable batch buffer, which
is then written to the file all at once. This avoids performing a system call for every
inserted item.
To make the internal batching work, the ancient database API had to be changed. While
integrating this new API in BlockChain.InsertReceiptChain, additional optimizations were
also added there.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change adds the --catalyst flag, enabling an RPC API for eth2 integration.
In this initial version, catalyst mode also disables all peer-to-peer networking.
Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
* all: add thousandths separators for big numbers on log messages
* p2p/sentry: drop accidental file
* common, log: add fast number formatter
* common, eth/protocols/snap: simplifty fancy num types
* log: handle nil big ints
Squashed from the following commits:
core/state: lazily init snapshot storage map
core/state: fix flawed meter on storage reads
core/state: make statedb/stateobjects reuse a hasher
core/blockchain, core/state: implement new trie prefetcher
core: make trie prefetcher deliver tries to statedb
core/state: refactor trie_prefetcher, export storage tries
blockchain: re-enable the next-block-prefetcher
state: remove panics in trie prefetcher
core/state/trie_prefetcher: address some review concerns
sq
