## Issue Addressed
Closes#1557
## Proposed Changes
Modify the pruning algorithm so that it mutates the head-tracker _before_ committing the database transaction to disk, and _only if_ all the heads to be removed are still present in the head-tracker (i.e. no concurrent mutations).
In the process of writing and testing this I also had to make a few other changes:
* Use internal mutability for all `BeaconChainHarness` functions (namely the RNG and the graffiti), in order to enable parallel calls (see testing section below).
* Disable logging in harness tests unless the `test_logger` feature is turned on
And chose to make some clean-ups:
* Delete the `NullMigrator`
* Remove type-based configuration for the migrator in favour of runtime config (simpler, less duplicated code)
* Use the non-blocking migrator unless the blocking migrator is required. In the store tests we need the blocking migrator because some tests make asserts about the state of the DB after the migration has run.
* Rename `validators_keypairs` -> `validator_keypairs` in the `BeaconChainHarness`
## Testing
To confirm that the fix worked, I wrote a test using [Hiatus](https://crates.io/crates/hiatus), which can be found here:
https://github.com/michaelsproul/lighthouse/tree/hiatus-issue-1557
That test can't be merged because it inserts random breakpoints everywhere, but if you check out that branch you can run the test with:
```
$ cd beacon_node/beacon_chain
$ cargo test --release --test parallel_tests --features test_logger
```
It should pass, and the log output should show:
```
WARN Pruning deferred because of a concurrent mutation, message: this is expected only very rarely!
```
## Additional Info
This is a backwards-compatible change with no impact on consensus.
## Issue Addressed
Closes#673
## Proposed Changes
Store a schema version in the database so that future releases can check they're running against a compatible database version. This would also enable automatic migration on breaking database changes, but that's left as future work.
The database config is also stored in the database so that the `slots_per_restore_point` value can be checked for consistency, which closes#673
The PR:
* Adds the ability to generate a crucial test scenario that isn't possible with `BeaconChainHarness` (i.e. two blocks occupying the same slot; previously forks necessitated skipping slots):
* New testing API: Instead of repeatedly calling add_block(), you generate a sorted `Vec<Slot>` and leave it up to the framework to generate blocks at those slots.
* Jumping backwards to an earlier epoch is a hard error, so that tests necessarily generate blocks in a epoch-by-epoch manner.
* Configures the test logger so that output is printed on the console in case a test fails. The logger also plays well with `--nocapture`, contrary to the existing testing framework
* Rewrites existing fork pruning tests to use the new API
* Adds a tests that triggers finalization at a non epoch boundary slot
* Renamed `BeaconChainYoke` to `BeaconChainTestingRig` because the former has been too confusing
* Fixed multiple tests (e.g. `block_production_different_shuffling_long`, `delete_blocks_and_states`, `shuffling_compatible_simple_fork`) that relied on a weird (and accidental) feature of the old `BeaconChainHarness` that attestations aren't produced for epochs earlier than the current one, thus masking potential bugs in test cases.
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
## Issue Addressed
Closes#1488
## Proposed Changes
* Prevent the pruning algorithm from over-eagerly deleting states at skipped slots when they are shared with the canonical chain.
* Add `debug` logging to the pruning algorithm so we have so better chance of debugging future issues from logs.
* Modify the handling of the "finalized state" in the beacon chain, so that it's always the state at the first slot of the finalized epoch (previously it was the state at the finalized block). This gives database pruning a clearer and cleaner view of things, and will marginally impact the pruning of the op pool, observed proposers, etc (in ways that are safe as far as I can tell).
* Remove duplicated `RevertedFinalizedEpoch` check from `after_finalization`
* Delete useless and unused `max_finality_distance`
* Add tests that exercise pruning with shared states at skip slots
* Delete unnecessary `block_strategy` argument from `add_blocks` and friends in the test harness (will likely conflict with #1380 slightly, sorry @adaszko -- but we can fix that)
* Bonus: add a `BeaconChain::with_head` method. I didn't end up needing it, but it turned out quite nice, so I figured we could keep it?
## Additional Info
Any users who have experienced pruning errors on Medalla will need to resync after upgrading to a release including this change. This should end unbounded `chain_db` growth! 🎉
## Issue Addressed
NA
## Proposed Changes
- Adds a new function to allow getting a state with a bad state root history for attestation verification. This reduces unnecessary tree hashing during attestation processing, which accounted for 23% of memory allocations (by bytes) in a recent `heaptrack` observation.
- Don't clone caches on intermediate epoch-boundary states during block processing.
- Reject blocks that are known to fork choice earlier during gossip processing, instead of waiting until after state has been loaded (this only happens in edge-case).
- Avoid multiple re-allocations by creating a "forced" exact size iterator.
## Additional Info
NA
* Layer do_atomically() abstractions properly
* Reduce allocs and DRY get_key_for_col()
* Parameterize HotColdDB with hot and cold item stores
* -impl Store for MemoryStore
* Replace Store uses with HotColdDB
* Ditch Store trait
* cargo fmt
* Style fix
* Readd missing dep that broke the build
* Remove redundant method
* Pull out a method out of a struct
* More precise db access abstractions
* Move fake trait method out of it
* cargo fmt
* Fix compilation error after refactoring
* Move another fake method out the Store trait
* Get rid of superfluous method
* Fix refactoring bug
* Rename: SimpleStoreItem -> StoreItem
* Get rid of the confusing DiskStore type alias
* Get rid of SimpleDiskStore type alias
* Correction: A method took both self and a ref to Self
* Improve error handling in block iteration
* Introduce atomic DB operations
* Fix race condition
An invariant was violated: For every block hash in head_tracker, that
block is accessible from the store.
* Address compiler warning
* Prune abandoned fork choice forks
* New approach to pruning
* Wrap some block hashes in a newtype pattern
For increased type safety.
* Add Graphviz chain dump emitter for debugging
* Fix broken test case
* Make prunes_abandoned_forks use real DiskStore
* Mark finalized blocks in the GraphViz output
* Refine debug stringification of Slot and Epoch
Before this commit: print!("{:?}", Slot(123)) == "Slot(\n123\n)".
After this commit: print!("{:?", Slot(123)) == "Slot(123)".
* Simplify build_block()
* Rewrite test case using more composable test primitives
* Working rewritten test case
* Tighten fork prunning test checks
* Add another pruning test case
* Bugfix: Finalized blocks weren't always properly detected
* Pruning: Add pruning_does_not_touch_blocks_prior_to_finalization test case
* Tighten pruning tests: check if heads are tracked properly
* Add a failing test case for a buggy scenario
* Change name of function to a more accurate one
* Fix failing test case
* Test case: Were skipped slots' states pruned?
* Style fix: Simplify dereferencing
* Tighten pruning tests: check if abandoned states are deleted
* Towards atomicity of db ops
* Correct typo
* Prune also skipped slots' states
* New logic for handling skipped states
* Make skipped slots test pass
* Post conflict resolution fixes
* Formatting fixes
* Tests passing
* Block hashes in Graphviz node labels
* Removed unused changes
* Fix bug with states having < SlotsPerHistoricalRoot roots
* Consolidate State/BlockRootsIterator for pruning
* Address review feedback
* Fix a bug in pruning tests
* Detach prune_abandoned_forks() from its object
* Move migrate.rs from store to beacon_chain
* Move forks pruning onto a background thread
* Bugfix: Heads weren't pruned when prune set contained only the head
* Rename: freeze_to_state() -> process_finalization()
* Eliminate redundant function parameter
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
* Unfinished progress
* Update more persistence code
* Start fixing tests
* Combine persist head and fork choice
* Persist head on reorg
* Gracefully handle op pool and eth1 cache missing
* Fix test failure
* Address Michael's comments
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Disable/delete two outdated tests
* Bump eth1 default vote warn to error
* Delete outdated eth1 test
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
* Add LRU caches to store
* Improvements to LRU caches
* Take state by value in `Store::put_state`
* Store blocks by value, configurable cache sizes
* Use a StateBatch to efficiently store skip states
* Fix store tests
* Add CloneConfig test, remove unused metrics
* Use Mutexes instead of RwLocks for LRU caches
* Store dht enrs on shutdown
* Load enrs on startup and add tests
* Remove enr_entries from behavior
* Move all dht persisting logic to `NetworkService`
* Move `PersistedDht` from eth2-libp2p to network crate
* Add test to confirm dht persistence
* Add logging
* Remove extra call to beacon_chain persist
* Expose only mutable `add_enr` method from behaviour
* Fix tests
* Fix merge errors
This should reduce disk usage by 32x while keeping historical state queries to
less than 10s. If historical states are required quickly, the minimum SPRP of 32
can be set on the CLI.
* Sparse hot DB and block root tree
* Fix store_tests
* Ensure loads of hot states on boundaries are fast
* Milder error for unaligned finalized blocks
* Apply clippy lints to beacon node
* Remove unnecessary logging and correct formatting
* Initial bones of load-balanced range-sync
* Port bump meshsup tests
* Further structure and network handling logic added
* Basic structure, ignoring error handling
* Correct max peers delay bug
* Clean up and re-write message processor and sync manager
* Restructure directory, correct type issues
* Fix compiler issues
* Completed first testing of new sync
* Correct merge issues
* Clean up warnings
* Push attestation processed log down to dbg
* Add state enc/dec benches
* Correct math error, downgraded logs
* Add example for flamegraph
* Use `PublicKeyBytes` for `Validator`
* Ripple PublicKeyBytes change through codebase
* Add RPC error handling and improved syncing code
* Add benches, optimizations to store BeaconState
* Store BeaconState in StorageContainer too
* Optimize StorageContainer with std::mem magic
* Add libp2p stream error handling and dropping of invalid peers
* Lower logs
* Update lcli to parse spec at boot, remove pycli
* Fix issues when starting with mainnet spec
* Set default spec to mainnet
* Fix lcli --spec param
* Add discovery tweak
* Ensure ETH1_FOLLOW_DISTANCE is in YamlConfig
* Set testnet ETH1_FOLLOW_DISTANCE to 16
* Fix rest_api tests
* Set testnet min validator count
* Update with new testnet dir
* Remove some dbg, println
* Add timeout when notifier waits for libp2p lock
* Add validator count CLI flag to lcli contract deploy
* Extend genesis delay time
* Correct libp2p service locking
* Update testnet dir
* Add basic block/state caching on beacon chain
* Add decimals display to notifier sync speed
* Try merge in change to reduce fork choice calls
* Remove fork choice from process block
* Minor log fix
* Check successes > 0
* Adds checkpoint cache
* Stop storing the tree hash cache in the db
* Handles peer disconnects for sync
* Fix failing beacon chain tests
* Change eth2_testnet_config tests to Mainnet
* Add logs downgrade discovery log
* Remove dedunant beacon state write
* Fix re-org warnings
* Use caching get methods in fork choice
* Fix mistake in prev commit
* Use caching state getting in state_by_slot
* Add state.cacheless_clone
* Less fork choice (#679)
* Try merge in change to reduce fork choice calls
* Remove fork choice from process block
* Minor log fix
* Check successes > 0
* Fix failing beacon chain tests
* Fix re-org warnings
* Fix mistake in prev commit
* Attempt to improve attestation processing times
* Introduce HeadInfo struct
* Used cache tree hash for block processing
* Use cached tree hash for block production too
* Range sync refactor
- Introduces `ChainCollection`
- Correct Disconnect node handling
- Removes duplicate code
* Add more logging for DB
* Various bug fixes
* Remove unnecessary logs
* Maintain syncing state in the transition from finalied to head
* Improved disconnect handling
* Add `Speedo` struct
* Fix bugs in speedo
* Fix bug in speedo
* Fix rounding bug in speedo
* Move code around, reduce speedo observation count
* Adds forwards block interator
* Fix inf NaN
* Add first draft of validator onboarding
* Update docs
* Add documentation link to main README
* Continue docs development
* Update book readme
* Update docs
* Allow vc to run without testnet subcommand
* Small change to onboarding docs
* Tidy CLI help messages
* Update docs
* Add check to val client see if beacon node is synced
* Attempt to fix NaN bug
* Fix compile bug
* Add notifier service to validator client
* Re-order onboarding steps
* Update deposit contract address
* Update testnet dir
* Add note about public eth1 node
* Fix installation link
* Set default eth1 endpoint to sigp
* Fix broken test
* Try fix eth1 cache locking
* Be more specific about eth1 endpoint
* Increase gas limit for deposit
* Fix default deposit amount
* Fix re-org log
* Add basic block/state caching on beacon chain
* Adds checkpoint cache
* Stop storing the tree hash cache in the db
* Remove dedunant beacon state write
* Use caching get methods in fork choice
* Use caching state getting in state_by_slot
* Add state.cacheless_clone
* Attempt to improve attestation processing times
* Introduce HeadInfo struct
* Used cache tree hash for block processing
* Use cached tree hash for block production too
* Add state enc/dec benches
* Add example for flamegraph
* Use `PublicKeyBytes` for `Validator`
* Ripple PublicKeyBytes change through codebase
* Add benches, optimizations to store BeaconState
* Store BeaconState in StorageContainer too
* Optimize StorageContainer with std::mem magic
* Fix rest_api tests
* Draft of checkpoint freezer DB
* Fix bugs
* Adjust root iterators for checkpoint database
* Fix freezer state lookups with no slot hint
* Fix split comment
* Use "restore point" to refer to frozen states
* Resolve some FIXMEs
* Configurable slots per restore point
* Document new freezer DB functions
* Fix up StoreConfig
* Fix new test for merge
* Document SPRP default CLI flag, clarify tests
* Implement freezer database for state vectors
* Improve BeaconState safe accessors
And fix a bug in the compact committees accessor.
* Banish dodgy type bounds back to gRPC
* Clean up
* Switch to exclusive end points in chunked vec
* Cleaning up and start of tests
* Randao fix, more tests
* Fix unsightly hack
* Resolve test FIXMEs
* Config file support
* More clean-ups, migrator beginnings
* Finish migrator, integrate into BeaconChain
* Fixups
* Fix store tests
* Fix BeaconChain tests
* Fix LMD GHOST tests
* Address review comments, delete 'static bounds
* Cargo format
* Address review comments
* Fix LMD ghost tests
* Update to spec v0.9.0
* Update to v0.9.1
* Bump spec tags for v0.9.1
* Formatting, fix CI failures
* Resolve accidental KeyPair merge conflict
* Document new BeaconState functions
* Fix incorrect cache drops in `advance_caches`
* Update fork choice for v0.9.1
* Clean up some FIXMEs
* Fix a few docs/logs
* Update for new builder paradigm, spec changes
* Freezer DB integration into BeaconNode
* Cleaning up
* This works, clean it up
* Cleanups
* Fix and improve store tests
* Refine store test
* Delete unused beacon_chain_builder.rs
* Fix CLI
* Store state at split slot in hot database
* Make fork choice lookup fast again
* Store freezer DB split slot in the database
* Handle potential div by 0 in chunked_vector
* Exclude committee caches from freezer DB
* Remove FIXME about long-running test
* Implement basic tree hash caching
* Use spaces to indent top-level Cargo.toml
* Optimize BLS tree hash by hashing bytes directly
* Implement tree hash caching for validator registry
* Persist BeaconState tree hash cache to disk
* Address Paul's review comments
* Change into_iter to iter
* Fix clippy 'easy' warnings
* Clippy eth2/utils
* Add struct NetworkInfo
* Clippy for types, utils, and beacon_node/store/src/iters.rs
* Cargo fmt
* Change foo to my_foo
* Remove complex signature
* suppress clippy warning for unit_value in benches
* Use enumerate instead of iterating over range
* Allow trivially_copy_pass_by_ref in serde_utils
* Renamed fork_choice::process_attestation_from_block
* Processing attestation in fork choice
* Retrieving state from store and checking signature
* Looser check on beacon state validity.
* Cleaned up get_attestation_state
* Expanded fork choice api to provide latest validator message.
* Checking if the an attestation contains a latest message
* Correct process_attestation error handling.
* Copy paste error in comment fixed.
* Tidy ancestor iterators
* Getting attestation slot via helper method
* Refactored attestation creation in test utils
* Revert "Refactored attestation creation in test utils"
This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306.
* Integration tests for free attestation processing
* Implicit conflicts resolved.
* formatting
* Do first pass on Grants code
* Add another attestation processing test
* Tidy attestation processing
* Remove old code fragment
* Add non-compiling half finished changes
* Simplify, fix bugs, add tests for chain iters
* Remove attestation processing from op pool
* Fix bug with fork choice, tidy
* Fix overly restrictive check in fork choice.
* Ensure committee cache is build during attn proc
* Ignore unknown blocks at fork choice
* Various minor fixes
* Make fork choice write lock in to read lock
* Remove unused method
* Tidy comments
* Fix attestation prod. target roots change
* Fix compile error in store iters
* Reject any attestation prior to finalization
* Begin metrics refactor
* Move beacon_chain to new metrics structure.
* Make metrics not panic if already defined
* Use global prometheus gather at rest api
* Unify common metric fns into a crate
* Add heavy metering to block processing
* Remove hypen from prometheus metric name
* Add more beacon chain metrics
* Add beacon chain persistence metric
* Prune op pool on finalization
* Add extra prom beacon chain metrics
* Prefix BeaconChain metrics with "beacon_"
* Add more store metrics
* Add basic metrics to libp2p
* Add metrics to HTTP server
* Remove old `http_server` crate
* Update metrics names to be more like standard
* Fix broken beacon chain metrics, add slot clock metrics
* Add lighthouse_metrics gather fn
* Remove http args
* Fix wrong state given to op pool prune
* Make prom metric names more consistent
* Add more metrics, tidy existing metrics
* Fix store block read metrics
* Tidy attestation metrics
* Fix minor PR comments
* Allow travis failures on beta (see desc)
There's a non-backward compatible change in `cargo fmt`. Stable and beta
do not agree.
* Tidy `lighthouse_metrics` docs
* Fix typo
* Renamed fork_choice::process_attestation_from_block
* Processing attestation in fork choice
* Retrieving state from store and checking signature
* Looser check on beacon state validity.
* Cleaned up get_attestation_state
* Expanded fork choice api to provide latest validator message.
* Checking if the an attestation contains a latest message
* Correct process_attestation error handling.
* Copy paste error in comment fixed.
* Tidy ancestor iterators
* Getting attestation slot via helper method
* Refactored attestation creation in test utils
* Revert "Refactored attestation creation in test utils"
This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306.
* Integration tests for free attestation processing
* Implicit conflicts resolved.
* formatting
* Do first pass on Grants code
* Add another attestation processing test
* Tidy attestation processing
* Remove old code fragment
* Add non-compiling half finished changes
* Simplify, fix bugs, add tests for chain iters
* Remove attestation processing from op pool
* Fix bug with fork choice, tidy
* Fix overly restrictive check in fork choice.
* Ensure committee cache is build during attn proc
* Ignore unknown blocks at fork choice
* Various minor fixes
* Make fork choice write lock in to read lock
* Remove unused method
* Tidy comments
* Fix attestation prod. target roots change
* Fix compile error in store iters
* Reject any attestation prior to finalization
* Fix minor PR comments
* Remove duplicated attestation finalization check
* Remove awkward `let` statement
* Add half-finished rest api changes
* Add basic, messy changes to rest api
* Fix expect() in ApiRequest
* Remove ApiRequest, add route for beacon state
* Tidy rest api, add get state from root
* Add api method for getting state roots by slot
* Add test for URL helper
* Simplify state_at_slot fn
* Add tests for rest api helper fns
* Add extra tests for parse root
* Fix clippy lints
* Fix compile error in rest api
* Update test to new ethereum-types
