* Statediff Geth
Handle conflicts (#244)
* Handle conflicts
* Update go mod file versions
* Make lint changes
Disassociate block number from the indexer object
Update ipld-eth-db ref
Refactor builder code to make it reusable
Use prefix comparison for account selective statediffing
Update builder unit tests
Add mode to write to CSV files in statediff file writer (#249)
* Change file writing mode to csv files
* Implement writer interface for file indexer
* Implement option for csv or sql in file mode
* Close files in CSV writer
* Add tests for CSV file mode
* Implement CSV file for watched addresses
* Separate test configs for CSV and SQL
* Refactor common code for file indexer tests
Update indexer to include block hash in receipts and logs (#256)
* Update indexer to include block hash in receipts and logs
* Upgrade ipld-eth-db image in docker-compose to run tests
Use watched addresses from direct indexing params by default while serving statediff APIs (#262)
* Use watched addresses from direct indexing params in statediff APIs by default
* Avoid using indexer object when direct indexing is off
* Add nil check before accessing watched addresses from direct indexing params
Rebase missed these changes needed at 1.10.20
Flags cleanup for CLI changes and linter complaints
Linter appeasements to achieve perfection
enforce go 1.18 for check (#267)
* enforce go 1.18 for check
* tests on 1.18 as well
* adding db yml for possible change in docker-compose behavior in yml parsing
Add indexer tests for handling non canonical blocks (#254)
* Add indexer tests for header and transactions in a non canonical block
* Add indexer tests for receipts in a non-canonical block and refactor
* Add indexer tests for logs in a non-canonical block
* Add indexer tests for state and storage nodes in a non-canonical block
* Add indexer tests for non-canonical block at another height
* Avoid passing address of a pointer
* Update refs in GitHub workflow
* Add genesis file path to stack-orchestrator config in GitHub workflow
* Add descriptive comments
fix non-deterministic ordering in unit tests
Refactor indexer tests to avoid duplicate code (#270)
* Refactor indexer tests to avoid duplicate code
* Refactor file mode indexer tests
* Fix expected db stats for sqlx after tx closure
* Refactor indexer tests for legacy block
* Refactor mainnet indexer tests
* Refactor tests for watched addressess methods
* Fix query in legacy indexer test
rebase and resolve onto 1.10.23... still error out of index related to GetLeafKeys
changed trie.Commit behavior was subtle about not not flushing to disk without an Update
* no merge nodeset throws nil
* linter appeasement
Co-authored-by: Abdul Rabbani <abdulrabbani00@gmail.com>
Cerc refactor (#281)
* first pass cerc refactor in cicd
* 1st attempt to publish binary to git.vdb.to from github release
* docker build step mangled
* docker build step mangled
* wrong username for docker login... which still succeeded
* circcicd is not cerccicd
* bad hostname
This adds a cache for block logs which is shared by all filters. The cache
size of is configurable using the `--cache.blocklogs` flag.
Co-authored-by: Felix Lange <fjl@twurst.com>
* core: use TryGetAccount to read where TryUpdateAccount has been used to write
* Gary's review feedback
* implement Gary's suggestion
* fix bug + rename NewSecure into NewStateTrie
* trie: add backwards-compatibility aliases for SecureTrie
* Update database.go
* make the linter happy
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
This change updates our urfave/cli dependency to the v2 branch of the library.
There are some Go API changes in cli v2:
- Flag values can now be accessed using the methods ctx.Bool,
ctx.Int, ctx.String, ... regardless of whether the flag is 'local' or
'global'.
- v2 has built-in support for flag categories. Our home-grown category
system is removed and the categories of flags are assigned as part of
the flag definition.
For users, there is only one observable difference with cli v2: flags must now
strictly appear before regular arguments. For example, the following command is
now invalid:
geth account import mykey.json --password file.txt
Instead, the command must be invoked as follows:
geth account import --password file.txt mykey.json
This enables the following linters
- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec
WIth a few exceptions.
- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
In #24028 we flagged a warning when finding legacy receipts in the freezer. This PR nudges users a bit more strongly by preventing geth from starting in this case until receipts have been migrated.
It also adds a flag --ignore-legacy-receipts which when present allows geth to start normally.
This PR adds db tooling (geth db check-state-content) to verify the integrity of trie nodes. It iterates through the 32-byte key space in the database, which is expected to contain RLP-encoded trie nodes, addressed by hash.
Previously freezer has only been used for storing ancient chain data, while obviously it can be used more. This PR unties the chain data and freezer, keep the minimal freezer structure and move all other logic (like incrementally freezing block data) into a separate structure called ChainFreezer.
This PR also extends the database interface by adding a new ancient store function AncientDatadir which can return the root directory of ancient store. The ancient root directory can be used when we want to open some other ancient-stores (e.g. reverse diff freezer).
This PR groups all built-in network flags together and list them in the command as a whole.
And all database path flags(datadir, ancient) are also grouped, since usually these two are
used together.
This PR fixes the flaw that @rjl493456442 found in https://github.com/ethereum/go-ethereum/pull/#issuecomment-1093817551 , namely, that the snapshot iterator uses the combined (disk + difflayers) 'view', wheres the raw iterator uses only the disk 'view'.
This PR instead splits up the work: one phase is iterating the disk layer data, another phase is loading the journalled difflayers and performing the same check there.
This commit replaces ioutil.TempDir with t.TempDir in tests. The
directory created by t.TempDir is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using ioutil.TempDir
had to be removed manually by calling os.RemoveAll, which is omitted in
some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but t.TempDir handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* eth/catalyst: only apply block if we actually have the state
* add header to payload queue
* Update cmd/geth/dbcmd.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cmd/geth: only check for presence of legacy receipts if developer mode is not enabled
* cmd/geth: degrade log level
* cmd/geth: fix format
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
* cmd,core: add simple legacy receipt converter
core/rawdb: use forEach in migrate
core/rawdb: batch reads in forEach
core/rawdb: make forEach anonymous fn
cmd/geth: check for legacy receipts on node startup
fix err msg
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
fix log
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
fix some review comments
add warning to cmd
drop isLegacy fn from migrateTable params
add test for windows rename
test replacing in windows case
* minor fix
* sanity check for tail-deletion
* add log before moving files around
* speed-up hack for mainnet
* fix mainnet check, use networkid instead
* check mainnet genesis
* review fixes
* resume previous migration attempt
* core/rawdb: lint fix
Co-authored-by: Martin Holst Swende <martin@swende.se>
* core/beacon: eth/catalyst: updated engine api to new version
* core: implement exchangeTransitionConfig
* core/beacon: prevRandao instead of Random
* eth/catalyst: Fix ExchangeTransitionConfig, add test
* eth/catalyst: stop external miners on TTD reached
* node: implement --authrpc.vhosts flag
* core: allow for config override on non-mainnet networks
* eth/catalyst: fix peters comments
* eth/catalyst: make stop remote sealer more explicit
* eth/catalyst: add log output
* cmd/utils: rename authrpc.host to authrpc.addr
* eth/catalyst: disable the disabling of the miner
* eth: core: remove notion of terminal pow block
* eth: les: more of peters nitpicks
* cmd, eth: Rename whitelist argument to peer.requiredblocks
* eth/ethconfig: document PeerRequiredBlocks better
* cmd/utils: rename new flag to --eth.requiredblocks
Co-authored-by: Felix Lange <fjl@twurst.com>
