lighthouse/consensus/proto_array/src/fork_choice_test_definition.rs

208 lines
7.1 KiB
Rust
Raw Normal View History

Add proto_array fork choice (#804) * Start implementing proto_array * Add progress * Add unfinished progress * Add further progress * Add progress * Add tree filtering * Add half-finished modifications * Add refactored version * Tidy, add incomplete LmdGhost impl * Move impls in LmdGhost trait def * Remove old reduced_tree fork choice * Combine two functions in to `compute_deltas` * Start testing * Add more compute_deltas tests * Add fork choice testing * Add more fork choice testing * Add more fork choice tests * Add more testing to proto-array * Remove old tests * Modify tests * Add more tests * Add more testing * Add comments and fixes * Re-organise crate * Tidy, finish pruning tests * Add ssz encoding, other pub fns * Rename lmd_ghost > proto_array_fork_choice * Integrate proto_array into lighthouse * Add first pass at fixing filter * Clean out old comments * Add more comments * Attempt to fix prune error * Adjust TODO * Fix test compile errors * Add extra justification change check * Update cargo.lock * Fix fork choice test compile errors * Most remove ffg_update_required * Fix bug with epoch of attestation votes * Start adding new test format * Make fork choice tests declarative * Create test def concept * Move test defs into crate * Add binary, re-org crate * Shuffle files * Start adding ffg tests * Add more fork choice tests * Add fork choice JSON dumping * Add more detail to best node error * Ensure fin+just checkpoints from from same block * Rename JustificationManager * Move checkpoint manager into own file * Tidy * Add targetted logging for sneaky sync bug * Fix justified balances bug * Add cache metrics * Add metrics for log levels * Fix bug in checkpoint manager * Fix compile error in fork choice tests * Ignore duplicate blocks in fork choice * Add block to fock choice before db * Rename on_new_block fn * Fix spec inconsistency in `CheckpointManager` * Remove BlockRootTree * Remove old reduced_tree code fragment * Add API endpoint for fork choice * Add more ffg tests * Remove block_root_tree reminents * Ensure effective balances are used * Remove old debugging code, fix API fault * Add check to ensure parent block is in fork choice * Update readme dates * Fix readme * Tidy checkpoint manager * Remove fork choice yaml files from repo * Remove fork choice yaml from repo * General tidy * Address majority of Michael's comments * Tidy bin/lib business * Remove dangling file * Undo changes for rpc/handler from master * Revert "Undo changes for rpc/handler from master" This reverts commit 876edff0e4a501aafbb47113454852826dcc24e8. Co-authored-by: Age Manning <Age@AgeManning.com>
2020-01-29 04:05:00 +00:00
mod ffg_updates;
mod no_votes;
mod votes;
use crate::proto_array_fork_choice::{Block, ExecutionStatus, ProtoArrayForkChoice};
Add proto_array fork choice (#804) * Start implementing proto_array * Add progress * Add unfinished progress * Add further progress * Add progress * Add tree filtering * Add half-finished modifications * Add refactored version * Tidy, add incomplete LmdGhost impl * Move impls in LmdGhost trait def * Remove old reduced_tree fork choice * Combine two functions in to `compute_deltas` * Start testing * Add more compute_deltas tests * Add fork choice testing * Add more fork choice testing * Add more fork choice tests * Add more testing to proto-array * Remove old tests * Modify tests * Add more tests * Add more testing * Add comments and fixes * Re-organise crate * Tidy, finish pruning tests * Add ssz encoding, other pub fns * Rename lmd_ghost > proto_array_fork_choice * Integrate proto_array into lighthouse * Add first pass at fixing filter * Clean out old comments * Add more comments * Attempt to fix prune error * Adjust TODO * Fix test compile errors * Add extra justification change check * Update cargo.lock * Fix fork choice test compile errors * Most remove ffg_update_required * Fix bug with epoch of attestation votes * Start adding new test format * Make fork choice tests declarative * Create test def concept * Move test defs into crate * Add binary, re-org crate * Shuffle files * Start adding ffg tests * Add more fork choice tests * Add fork choice JSON dumping * Add more detail to best node error * Ensure fin+just checkpoints from from same block * Rename JustificationManager * Move checkpoint manager into own file * Tidy * Add targetted logging for sneaky sync bug * Fix justified balances bug * Add cache metrics * Add metrics for log levels * Fix bug in checkpoint manager * Fix compile error in fork choice tests * Ignore duplicate blocks in fork choice * Add block to fock choice before db * Rename on_new_block fn * Fix spec inconsistency in `CheckpointManager` * Remove BlockRootTree * Remove old reduced_tree code fragment * Add API endpoint for fork choice * Add more ffg tests * Remove block_root_tree reminents * Ensure effective balances are used * Remove old debugging code, fix API fault * Add check to ensure parent block is in fork choice * Update readme dates * Fix readme * Tidy checkpoint manager * Remove fork choice yaml files from repo * Remove fork choice yaml from repo * General tidy * Address majority of Michael's comments * Tidy bin/lib business * Remove dangling file * Undo changes for rpc/handler from master * Revert "Undo changes for rpc/handler from master" This reverts commit 876edff0e4a501aafbb47113454852826dcc24e8. Co-authored-by: Age Manning <Age@AgeManning.com>
2020-01-29 04:05:00 +00:00
use serde_derive::{Deserialize, Serialize};
Advance state to next slot after importing block (#2174) ## Issue Addressed NA ## Proposed Changes Add an optimization to perform `per_slot_processing` from the *leading-edge* of block processing to the *trailing-edge*. Ultimately, this allows us to import the block at slot `n` faster because we used the tail-end of slot `n - 1` to perform `per_slot_processing`. Additionally, add a "block proposer cache" which allows us to cache the block proposer for some epoch. Since we're now doing trailing-edge `per_slot_processing`, we can prime this cache with the values for the next epoch before those blocks arrive (assuming those blocks don't have some weird forking). There were several ancillary changes required to achieve this: - Remove the `state_root` field of `BeaconSnapshot`, since there's no need to know it on a `pre_state` and in all other cases we can just read it from `block.state_root()`. - This caused some "dust" changes of `snapshot.beacon_state_root` to `snapshot.beacon_state_root()`, where the `BeaconSnapshot::beacon_state_root()` func just reads the state root from the block. - Rename `types::ShuffingId` to `AttestationShufflingId`. I originally did this because I added a `ProposerShufflingId` struct which turned out to be not so useful. I thought this new name was more descriptive so I kept it. - Address https://github.com/ethereum/eth2.0-specs/pull/2196 - Add a debug log when we get a block with an unknown parent. There was previously no logging around this case. - Add a function to `BeaconState` to compute all proposers for an epoch without re-computing the active indices for each slot. ## Additional Info - ~~Blocked on #2173~~ - ~~Blocked on #2179~~ That PR was wrapped into this PR. - There's potentially some places where we could avoid computing the proposer indices in `per_block_processing` but I haven't done this here. These would be an optimization beyond the issue at hand (improving block propagation times) and I think this PR is already doing enough. We can come back for that later. ## TODO - [x] Tidy, improve comments. - [x] ~~Try avoid computing proposer index in `per_block_processing`?~~
2021-02-15 07:17:52 +00:00
use types::{AttestationShufflingId, Epoch, Hash256, Slot};
Add proto_array fork choice (#804) * Start implementing proto_array * Add progress * Add unfinished progress * Add further progress * Add progress * Add tree filtering * Add half-finished modifications * Add refactored version * Tidy, add incomplete LmdGhost impl * Move impls in LmdGhost trait def * Remove old reduced_tree fork choice * Combine two functions in to `compute_deltas` * Start testing * Add more compute_deltas tests * Add fork choice testing * Add more fork choice testing * Add more fork choice tests * Add more testing to proto-array * Remove old tests * Modify tests * Add more tests * Add more testing * Add comments and fixes * Re-organise crate * Tidy, finish pruning tests * Add ssz encoding, other pub fns * Rename lmd_ghost > proto_array_fork_choice * Integrate proto_array into lighthouse * Add first pass at fixing filter * Clean out old comments * Add more comments * Attempt to fix prune error * Adjust TODO * Fix test compile errors * Add extra justification change check * Update cargo.lock * Fix fork choice test compile errors * Most remove ffg_update_required * Fix bug with epoch of attestation votes * Start adding new test format * Make fork choice tests declarative * Create test def concept * Move test defs into crate * Add binary, re-org crate * Shuffle files * Start adding ffg tests * Add more fork choice tests * Add fork choice JSON dumping * Add more detail to best node error * Ensure fin+just checkpoints from from same block * Rename JustificationManager * Move checkpoint manager into own file * Tidy * Add targetted logging for sneaky sync bug * Fix justified balances bug * Add cache metrics * Add metrics for log levels * Fix bug in checkpoint manager * Fix compile error in fork choice tests * Ignore duplicate blocks in fork choice * Add block to fock choice before db * Rename on_new_block fn * Fix spec inconsistency in `CheckpointManager` * Remove BlockRootTree * Remove old reduced_tree code fragment * Add API endpoint for fork choice * Add more ffg tests * Remove block_root_tree reminents * Ensure effective balances are used * Remove old debugging code, fix API fault * Add check to ensure parent block is in fork choice * Update readme dates * Fix readme * Tidy checkpoint manager * Remove fork choice yaml files from repo * Remove fork choice yaml from repo * General tidy * Address majority of Michael's comments * Tidy bin/lib business * Remove dangling file * Undo changes for rpc/handler from master * Revert "Undo changes for rpc/handler from master" This reverts commit 876edff0e4a501aafbb47113454852826dcc24e8. Co-authored-by: Age Manning <Age@AgeManning.com>
2020-01-29 04:05:00 +00:00
pub use ffg_updates::*;
pub use no_votes::*;
pub use votes::*;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum Operation {
FindHead {
justified_epoch: Epoch,
justified_root: Hash256,
finalized_epoch: Epoch,
justified_state_balances: Vec<u64>,
expected_head: Hash256,
},
InvalidFindHead {
justified_epoch: Epoch,
justified_root: Hash256,
finalized_epoch: Epoch,
justified_state_balances: Vec<u64>,
},
ProcessBlock {
slot: Slot,
root: Hash256,
parent_root: Hash256,
justified_epoch: Epoch,
finalized_epoch: Epoch,
},
ProcessAttestation {
validator_index: usize,
block_root: Hash256,
target_epoch: Epoch,
},
Prune {
finalized_root: Hash256,
prune_threshold: usize,
expected_len: usize,
},
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ForkChoiceTestDefinition {
pub finalized_block_slot: Slot,
pub justified_epoch: Epoch,
pub finalized_epoch: Epoch,
pub finalized_root: Hash256,
pub operations: Vec<Operation>,
}
impl ForkChoiceTestDefinition {
pub fn run(self) {
Advance state to next slot after importing block (#2174) ## Issue Addressed NA ## Proposed Changes Add an optimization to perform `per_slot_processing` from the *leading-edge* of block processing to the *trailing-edge*. Ultimately, this allows us to import the block at slot `n` faster because we used the tail-end of slot `n - 1` to perform `per_slot_processing`. Additionally, add a "block proposer cache" which allows us to cache the block proposer for some epoch. Since we're now doing trailing-edge `per_slot_processing`, we can prime this cache with the values for the next epoch before those blocks arrive (assuming those blocks don't have some weird forking). There were several ancillary changes required to achieve this: - Remove the `state_root` field of `BeaconSnapshot`, since there's no need to know it on a `pre_state` and in all other cases we can just read it from `block.state_root()`. - This caused some "dust" changes of `snapshot.beacon_state_root` to `snapshot.beacon_state_root()`, where the `BeaconSnapshot::beacon_state_root()` func just reads the state root from the block. - Rename `types::ShuffingId` to `AttestationShufflingId`. I originally did this because I added a `ProposerShufflingId` struct which turned out to be not so useful. I thought this new name was more descriptive so I kept it. - Address https://github.com/ethereum/eth2.0-specs/pull/2196 - Add a debug log when we get a block with an unknown parent. There was previously no logging around this case. - Add a function to `BeaconState` to compute all proposers for an epoch without re-computing the active indices for each slot. ## Additional Info - ~~Blocked on #2173~~ - ~~Blocked on #2179~~ That PR was wrapped into this PR. - There's potentially some places where we could avoid computing the proposer indices in `per_block_processing` but I haven't done this here. These would be an optimization beyond the issue at hand (improving block propagation times) and I think this PR is already doing enough. We can come back for that later. ## TODO - [x] Tidy, improve comments. - [x] ~~Try avoid computing proposer index in `per_block_processing`?~~
2021-02-15 07:17:52 +00:00
let junk_shuffling_id =
AttestationShufflingId::from_components(Epoch::new(0), Hash256::zero());
let execution_status = ExecutionStatus::irrelevant();
v0.12 fork choice update (#1229) * Incomplete scraps * Add progress on new fork choice impl * Further progress * First complete compiling version * Remove chain reference * Add new lmd_ghost crate * Start integrating into beacon chain * Update `milagro_bls` to new release (#1183) * Update milagro_bls to new release Signed-off-by: Kirk Baird <baird.k@outlook.com> * Tidy up fake cryptos Signed-off-by: Kirk Baird <baird.k@outlook.com> * move SecretHash to bls and put plaintext back Signed-off-by: Kirk Baird <baird.k@outlook.com> * Update state processing for v0.12 * Fix EF test runners for v0.12 * Fix some tests * Fix broken attestation verification test * More test fixes * Rough beacon chain impl working * Remove fork_choice_2 * Remove checkpoint manager * Half finished ssz impl * Add missed file * Add persistence * Tidy, fix some compile errors * Remove RwLock from ProtoArrayForkChoice * Fix store-based compile errors * Add comments, tidy * Move function out of ForkChoice struct * Start testing * More testing * Fix compile error * Tidy beacon_chain::fork_choice * Queue attestations from the current slot * Allow fork choice to handle prior-to-genesis start * Improve error granularity * Test attestation dequeuing * Process attestations during block * Store target root in fork choice * Move fork choice verification into new crate * Update tests * Consensus updates for v0.12 (#1228) * Update state processing for v0.12 * Fix EF test runners for v0.12 * Fix some tests * Fix broken attestation verification test * More test fixes * Fix typo found in review * Add `Block` struct to ProtoArray * Start fixing get_ancestor * Add rough progress on testing * Get fork choice tests working * Progress with testing * Fix partialeq impl * Move slot clock from fc_store * Improve testing * Add testing for best justified * Add clone back to SystemTimeSlotClock * Add balances test * Start adding balances cache again * Wire-in balances cache * Improve tests * Remove commented-out tests * Remove beacon_chain::ForkChoice * Rename crates * Update wider codebase to new fork_choice layout * Move advance_slot in test harness * Tidy ForkChoice::update_time * Fix verification tests * Fix compile error with iter::once * Fix fork choice tests * Ensure block attestations are processed * Fix failing beacon_chain tests * Add first invalid block check * Add finalized block check * Progress with testing, new store builder * Add fixes to get_ancestor * Fix old genesis justification test * Fix remaining fork choice tests * Change root iteration method * Move on_verified_block * Remove unused method * Start adding attestation verification tests * Add invalid ffg target test * Add target epoch test * Add queued attestation test * Remove old fork choice verification tests * Tidy, add test * Move fork choice lock drop * Rename BeaconForkChoiceStore * Add comments, tidy BeaconForkChoiceStore * Update metrics, rename fork_choice_store.rs * Remove genesis_block_root from ForkChoice * Tidy * Update fork_choice comments * Tidy, add comments * Tidy, simplify ForkChoice, fix compile issue * Tidy, removed dead file * Increase http request timeout * Fix failing rest_api test * Set HTTP timeout back to 5s * Apply fix to get_ancestor * Address Michael's comments * Fix typo * Revert "Fix broken attestation verification test" This reverts commit 722cdc903b12611de27916a57eeecfa3224f2279. Co-authored-by: Kirk Baird <baird.k@outlook.com> Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-06-17 01:10:22 +00:00
let mut fork_choice = ProtoArrayForkChoice::new(
Add proto_array fork choice (#804) * Start implementing proto_array * Add progress * Add unfinished progress * Add further progress * Add progress * Add tree filtering * Add half-finished modifications * Add refactored version * Tidy, add incomplete LmdGhost impl * Move impls in LmdGhost trait def * Remove old reduced_tree fork choice * Combine two functions in to `compute_deltas` * Start testing * Add more compute_deltas tests * Add fork choice testing * Add more fork choice testing * Add more fork choice tests * Add more testing to proto-array * Remove old tests * Modify tests * Add more tests * Add more testing * Add comments and fixes * Re-organise crate * Tidy, finish pruning tests * Add ssz encoding, other pub fns * Rename lmd_ghost > proto_array_fork_choice * Integrate proto_array into lighthouse * Add first pass at fixing filter * Clean out old comments * Add more comments * Attempt to fix prune error * Adjust TODO * Fix test compile errors * Add extra justification change check * Update cargo.lock * Fix fork choice test compile errors * Most remove ffg_update_required * Fix bug with epoch of attestation votes * Start adding new test format * Make fork choice tests declarative * Create test def concept * Move test defs into crate * Add binary, re-org crate * Shuffle files * Start adding ffg tests * Add more fork choice tests * Add fork choice JSON dumping * Add more detail to best node error * Ensure fin+just checkpoints from from same block * Rename JustificationManager * Move checkpoint manager into own file * Tidy * Add targetted logging for sneaky sync bug * Fix justified balances bug * Add cache metrics * Add metrics for log levels * Fix bug in checkpoint manager * Fix compile error in fork choice tests * Ignore duplicate blocks in fork choice * Add block to fock choice before db * Rename on_new_block fn * Fix spec inconsistency in `CheckpointManager` * Remove BlockRootTree * Remove old reduced_tree code fragment * Add API endpoint for fork choice * Add more ffg tests * Remove block_root_tree reminents * Ensure effective balances are used * Remove old debugging code, fix API fault * Add check to ensure parent block is in fork choice * Update readme dates * Fix readme * Tidy checkpoint manager * Remove fork choice yaml files from repo * Remove fork choice yaml from repo * General tidy * Address majority of Michael's comments * Tidy bin/lib business * Remove dangling file * Undo changes for rpc/handler from master * Revert "Undo changes for rpc/handler from master" This reverts commit 876edff0e4a501aafbb47113454852826dcc24e8. Co-authored-by: Age Manning <Age@AgeManning.com>
2020-01-29 04:05:00 +00:00
self.finalized_block_slot,
Optimize attestation processing (#841) * Start updating types * WIP * Signature hacking * Existing EF tests passing with fake_crypto * Updates * Delete outdated API spec * The refactor continues * It compiles * WIP test fixes * All release tests passing bar genesis state parsing * Update and test YamlConfig * Update to spec v0.10 compatible BLS * Updates to BLS EF tests * Add EF test for AggregateVerify And delete unused hash2curve tests for uncompressed points * Update EF tests to v0.10.1 * Use optional block root correctly in block proc * Use genesis fork in deposit domain. All tests pass * Cargo fmt * Fast aggregate verify test * Update REST API docs * Cargo fmt * Fix unused import * Bump spec tags to v0.10.1 * Add `seconds_per_eth1_block` to chainspec * Update to timestamp based eth1 voting scheme * Return None from `get_votes_to_consider` if block cache is empty * Handle overflows in `is_candidate_block` * Revert to failing tests * Fix eth1 data sets test * Choose default vote according to spec * Fix collect_valid_votes tests * Fix `get_votes_to_consider` to choose all eligible blocks * Uncomment winning_vote tests * Add comments; remove unused code * Reduce seconds_per_eth1_block for simulation * Addressed review comments * Add test for default vote case * Fix logs * Remove unused functions * Meter default eth1 votes * Fix comments * Address review comments; remove unused dependency * Add first attempt at attestation proc. re-write * Add version 2 of attestation processing * Minor fixes * Add validator pubkey cache * Make get_indexed_attestation take a committee * Link signature processing into new attn verification * First working version * Ensure pubkey cache is updated * Add more metrics, slight optimizations * Clone committee cache during attestation processing * Update shuffling cache during block processing * Remove old commented-out code * Fix shuffling cache insert bug * Used indexed attestation in fork choice * Restructure attn processing, add metrics * Add more detailed metrics * Tidy, fix failing tests * Fix failing tests, tidy * Disable/delete two outdated tests * Tidy * Add pubkey cache persistence file * Add more comments * Integrate persistence file into builder * Add pubkey cache tests * Add data_dir to beacon chain builder * Remove Option in pubkey cache persistence file * Ensure consistency between datadir/data_dir * Fix failing network test * Tidy * Fix todos * Add attestation processing tests * Add another test * Only run attestation tests in release * Make attestation tests MainnetEthSpec * Address Michael's comments * Remove redundant check * Fix warning * Fix failing test Co-authored-by: Michael Sproul <micsproul@gmail.com> Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
2020-03-05 06:19:35 +00:00
Hash256::zero(),
Add proto_array fork choice (#804) * Start implementing proto_array * Add progress * Add unfinished progress * Add further progress * Add progress * Add tree filtering * Add half-finished modifications * Add refactored version * Tidy, add incomplete LmdGhost impl * Move impls in LmdGhost trait def * Remove old reduced_tree fork choice * Combine two functions in to `compute_deltas` * Start testing * Add more compute_deltas tests * Add fork choice testing * Add more fork choice testing * Add more fork choice tests * Add more testing to proto-array * Remove old tests * Modify tests * Add more tests * Add more testing * Add comments and fixes * Re-organise crate * Tidy, finish pruning tests * Add ssz encoding, other pub fns * Rename lmd_ghost > proto_array_fork_choice * Integrate proto_array into lighthouse * Add first pass at fixing filter * Clean out old comments * Add more comments * Attempt to fix prune error * Adjust TODO * Fix test compile errors * Add extra justification change check * Update cargo.lock * Fix fork choice test compile errors * Most remove ffg_update_required * Fix bug with epoch of attestation votes * Start adding new test format * Make fork choice tests declarative * Create test def concept * Move test defs into crate * Add binary, re-org crate * Shuffle files * Start adding ffg tests * Add more fork choice tests * Add fork choice JSON dumping * Add more detail to best node error * Ensure fin+just checkpoints from from same block * Rename JustificationManager * Move checkpoint manager into own file * Tidy * Add targetted logging for sneaky sync bug * Fix justified balances bug * Add cache metrics * Add metrics for log levels * Fix bug in checkpoint manager * Fix compile error in fork choice tests * Ignore duplicate blocks in fork choice * Add block to fock choice before db * Rename on_new_block fn * Fix spec inconsistency in `CheckpointManager` * Remove BlockRootTree * Remove old reduced_tree code fragment * Add API endpoint for fork choice * Add more ffg tests * Remove block_root_tree reminents * Ensure effective balances are used * Remove old debugging code, fix API fault * Add check to ensure parent block is in fork choice * Update readme dates * Fix readme * Tidy checkpoint manager * Remove fork choice yaml files from repo * Remove fork choice yaml from repo * General tidy * Address majority of Michael's comments * Tidy bin/lib business * Remove dangling file * Undo changes for rpc/handler from master * Revert "Undo changes for rpc/handler from master" This reverts commit 876edff0e4a501aafbb47113454852826dcc24e8. Co-authored-by: Age Manning <Age@AgeManning.com>
2020-01-29 04:05:00 +00:00
self.justified_epoch,
self.finalized_epoch,
self.finalized_root,
Implement standard eth2.0 API (#1569) - Resolves #1550 - Resolves #824 - Resolves #825 - Resolves #1131 - Resolves #1411 - Resolves #1256 - Resolve #1177 - Includes the `ShufflingId` struct initially defined in #1492. That PR is now closed and the changes are included here, with significant bug fixes. - Implement the https://github.com/ethereum/eth2.0-APIs in a new `http_api` crate using `warp`. This replaces the `rest_api` crate. - Add a new `common/eth2` crate which provides a wrapper around `reqwest`, providing the HTTP client that is used by the validator client and for testing. This replaces the `common/remote_beacon_node` crate. - Create a `http_metrics` crate which is a dedicated server for Prometheus metrics (they are no longer served on the same port as the REST API). We now have flags for `--metrics`, `--metrics-address`, etc. - Allow the `subnet_id` to be an optional parameter for `VerifiedUnaggregatedAttestation::verify`. This means it does not need to be provided unnecessarily by the validator client. - Move `fn map_attestation_committee` in `mod beacon_chain::attestation_verification` to a new `fn with_committee_cache` on the `BeaconChain` so the same cache can be used for obtaining validator duties. - Add some other helpers to `BeaconChain` to assist with common API duties (e.g., `block_root_at_slot`, `head_beacon_block_root`). - Change the `NaiveAggregationPool` so it can index attestations by `hash_tree_root(attestation.data)`. This is a requirement of the API. - Add functions to `BeaconChainHarness` to allow it to create slashings and exits. - Allow for `eth1::Eth1NetworkId` to go to/from a `String`. - Add functions to the `OperationPool` to allow getting all objects in the pool. - Add function to `BeaconState` to check if a committee cache is initialized. - Fix bug where `seconds_per_eth1_block` was not transferring over from `YamlConfig` to `ChainSpec`. - Add the `deposit_contract_address` to `YamlConfig` and `ChainSpec`. We needed to be able to return it in an API response. - Change some uses of serde `serialize_with` and `deserialize_with` to a single use of `with` (code quality). - Impl `Display` and `FromStr` for several BLS fields. - Check for clock discrepancy when VC polls BN for sync state (with +/- 1 slot tolerance). This is not intended to be comprehensive, it was just easy to do. - See #1434 for a per-endpoint overview. - Seeking clarity here: https://github.com/ethereum/eth2.0-APIs/issues/75 - [x] Add docs for prom port to close #1256 - [x] Follow up on this #1177 - [x] ~~Follow up with #1424~~ Will fix in future PR. - [x] Follow up with #1411 - [x] ~~Follow up with #1260~~ Will fix in future PR. - [x] Add quotes to all integers. - [x] Remove `rest_types` - [x] Address missing beacon block error. (#1629) - [x] ~~Add tests for lighthouse/peers endpoints~~ Wontfix - [x] ~~Follow up with validator status proposal~~ Tracked in #1434 - [x] Unify graffiti structs - [x] ~~Start server when waiting for genesis?~~ Will fix in future PR. - [x] TODO in http_api tests - [x] Move lighthouse endpoints off /eth/v1 - [x] Update docs to link to standard - ~~Blocked on #1586~~ Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-09-29 03:46:54 +00:00
junk_shuffling_id.clone(),
junk_shuffling_id,
execution_status,
Add proto_array fork choice (#804) * Start implementing proto_array * Add progress * Add unfinished progress * Add further progress * Add progress * Add tree filtering * Add half-finished modifications * Add refactored version * Tidy, add incomplete LmdGhost impl * Move impls in LmdGhost trait def * Remove old reduced_tree fork choice * Combine two functions in to `compute_deltas` * Start testing * Add more compute_deltas tests * Add fork choice testing * Add more fork choice testing * Add more fork choice tests * Add more testing to proto-array * Remove old tests * Modify tests * Add more tests * Add more testing * Add comments and fixes * Re-organise crate * Tidy, finish pruning tests * Add ssz encoding, other pub fns * Rename lmd_ghost > proto_array_fork_choice * Integrate proto_array into lighthouse * Add first pass at fixing filter * Clean out old comments * Add more comments * Attempt to fix prune error * Adjust TODO * Fix test compile errors * Add extra justification change check * Update cargo.lock * Fix fork choice test compile errors * Most remove ffg_update_required * Fix bug with epoch of attestation votes * Start adding new test format * Make fork choice tests declarative * Create test def concept * Move test defs into crate * Add binary, re-org crate * Shuffle files * Start adding ffg tests * Add more fork choice tests * Add fork choice JSON dumping * Add more detail to best node error * Ensure fin+just checkpoints from from same block * Rename JustificationManager * Move checkpoint manager into own file * Tidy * Add targetted logging for sneaky sync bug * Fix justified balances bug * Add cache metrics * Add metrics for log levels * Fix bug in checkpoint manager * Fix compile error in fork choice tests * Ignore duplicate blocks in fork choice * Add block to fock choice before db * Rename on_new_block fn * Fix spec inconsistency in `CheckpointManager` * Remove BlockRootTree * Remove old reduced_tree code fragment * Add API endpoint for fork choice * Add more ffg tests * Remove block_root_tree reminents * Ensure effective balances are used * Remove old debugging code, fix API fault * Add check to ensure parent block is in fork choice * Update readme dates * Fix readme * Tidy checkpoint manager * Remove fork choice yaml files from repo * Remove fork choice yaml from repo * General tidy * Address majority of Michael's comments * Tidy bin/lib business * Remove dangling file * Undo changes for rpc/handler from master * Revert "Undo changes for rpc/handler from master" This reverts commit 876edff0e4a501aafbb47113454852826dcc24e8. Co-authored-by: Age Manning <Age@AgeManning.com>
2020-01-29 04:05:00 +00:00
)
.expect("should create fork choice struct");
for (op_index, op) in self.operations.into_iter().enumerate() {
match op.clone() {
Operation::FindHead {
justified_epoch,
justified_root,
finalized_epoch,
justified_state_balances,
expected_head,
} => {
let head = fork_choice
.find_head(
justified_epoch,
justified_root,
finalized_epoch,
&justified_state_balances,
)
.unwrap_or_else(|_| {
panic!("find_head op at index {} returned error", op_index)
});
Add proto_array fork choice (#804) * Start implementing proto_array * Add progress * Add unfinished progress * Add further progress * Add progress * Add tree filtering * Add half-finished modifications * Add refactored version * Tidy, add incomplete LmdGhost impl * Move impls in LmdGhost trait def * Remove old reduced_tree fork choice * Combine two functions in to `compute_deltas` * Start testing * Add more compute_deltas tests * Add fork choice testing * Add more fork choice testing * Add more fork choice tests * Add more testing to proto-array * Remove old tests * Modify tests * Add more tests * Add more testing * Add comments and fixes * Re-organise crate * Tidy, finish pruning tests * Add ssz encoding, other pub fns * Rename lmd_ghost > proto_array_fork_choice * Integrate proto_array into lighthouse * Add first pass at fixing filter * Clean out old comments * Add more comments * Attempt to fix prune error * Adjust TODO * Fix test compile errors * Add extra justification change check * Update cargo.lock * Fix fork choice test compile errors * Most remove ffg_update_required * Fix bug with epoch of attestation votes * Start adding new test format * Make fork choice tests declarative * Create test def concept * Move test defs into crate * Add binary, re-org crate * Shuffle files * Start adding ffg tests * Add more fork choice tests * Add fork choice JSON dumping * Add more detail to best node error * Ensure fin+just checkpoints from from same block * Rename JustificationManager * Move checkpoint manager into own file * Tidy * Add targetted logging for sneaky sync bug * Fix justified balances bug * Add cache metrics * Add metrics for log levels * Fix bug in checkpoint manager * Fix compile error in fork choice tests * Ignore duplicate blocks in fork choice * Add block to fock choice before db * Rename on_new_block fn * Fix spec inconsistency in `CheckpointManager` * Remove BlockRootTree * Remove old reduced_tree code fragment * Add API endpoint for fork choice * Add more ffg tests * Remove block_root_tree reminents * Ensure effective balances are used * Remove old debugging code, fix API fault * Add check to ensure parent block is in fork choice * Update readme dates * Fix readme * Tidy checkpoint manager * Remove fork choice yaml files from repo * Remove fork choice yaml from repo * General tidy * Address majority of Michael's comments * Tidy bin/lib business * Remove dangling file * Undo changes for rpc/handler from master * Revert "Undo changes for rpc/handler from master" This reverts commit 876edff0e4a501aafbb47113454852826dcc24e8. Co-authored-by: Age Manning <Age@AgeManning.com>
2020-01-29 04:05:00 +00:00
assert_eq!(
head, expected_head,
"Operation at index {} failed checks. Operation: {:?}",
op_index, op
);
check_bytes_round_trip(&fork_choice);
}
Operation::InvalidFindHead {
justified_epoch,
justified_root,
finalized_epoch,
justified_state_balances,
} => {
let result = fork_choice.find_head(
justified_epoch,
justified_root,
finalized_epoch,
&justified_state_balances,
);
assert!(
result.is_err(),
"Operation at index {} . Operation: {:?}",
op_index,
op
);
check_bytes_round_trip(&fork_choice);
}
Operation::ProcessBlock {
slot,
root,
parent_root,
justified_epoch,
finalized_epoch,
} => {
v0.12 fork choice update (#1229) * Incomplete scraps * Add progress on new fork choice impl * Further progress * First complete compiling version * Remove chain reference * Add new lmd_ghost crate * Start integrating into beacon chain * Update `milagro_bls` to new release (#1183) * Update milagro_bls to new release Signed-off-by: Kirk Baird <baird.k@outlook.com> * Tidy up fake cryptos Signed-off-by: Kirk Baird <baird.k@outlook.com> * move SecretHash to bls and put plaintext back Signed-off-by: Kirk Baird <baird.k@outlook.com> * Update state processing for v0.12 * Fix EF test runners for v0.12 * Fix some tests * Fix broken attestation verification test * More test fixes * Rough beacon chain impl working * Remove fork_choice_2 * Remove checkpoint manager * Half finished ssz impl * Add missed file * Add persistence * Tidy, fix some compile errors * Remove RwLock from ProtoArrayForkChoice * Fix store-based compile errors * Add comments, tidy * Move function out of ForkChoice struct * Start testing * More testing * Fix compile error * Tidy beacon_chain::fork_choice * Queue attestations from the current slot * Allow fork choice to handle prior-to-genesis start * Improve error granularity * Test attestation dequeuing * Process attestations during block * Store target root in fork choice * Move fork choice verification into new crate * Update tests * Consensus updates for v0.12 (#1228) * Update state processing for v0.12 * Fix EF test runners for v0.12 * Fix some tests * Fix broken attestation verification test * More test fixes * Fix typo found in review * Add `Block` struct to ProtoArray * Start fixing get_ancestor * Add rough progress on testing * Get fork choice tests working * Progress with testing * Fix partialeq impl * Move slot clock from fc_store * Improve testing * Add testing for best justified * Add clone back to SystemTimeSlotClock * Add balances test * Start adding balances cache again * Wire-in balances cache * Improve tests * Remove commented-out tests * Remove beacon_chain::ForkChoice * Rename crates * Update wider codebase to new fork_choice layout * Move advance_slot in test harness * Tidy ForkChoice::update_time * Fix verification tests * Fix compile error with iter::once * Fix fork choice tests * Ensure block attestations are processed * Fix failing beacon_chain tests * Add first invalid block check * Add finalized block check * Progress with testing, new store builder * Add fixes to get_ancestor * Fix old genesis justification test * Fix remaining fork choice tests * Change root iteration method * Move on_verified_block * Remove unused method * Start adding attestation verification tests * Add invalid ffg target test * Add target epoch test * Add queued attestation test * Remove old fork choice verification tests * Tidy, add test * Move fork choice lock drop * Rename BeaconForkChoiceStore * Add comments, tidy BeaconForkChoiceStore * Update metrics, rename fork_choice_store.rs * Remove genesis_block_root from ForkChoice * Tidy * Update fork_choice comments * Tidy, add comments * Tidy, simplify ForkChoice, fix compile issue * Tidy, removed dead file * Increase http request timeout * Fix failing rest_api test * Set HTTP timeout back to 5s * Apply fix to get_ancestor * Address Michael's comments * Fix typo * Revert "Fix broken attestation verification test" This reverts commit 722cdc903b12611de27916a57eeecfa3224f2279. Co-authored-by: Kirk Baird <baird.k@outlook.com> Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-06-17 01:10:22 +00:00
let block = Block {
slot,
root,
parent_root: Some(parent_root),
state_root: Hash256::zero(),
target_root: Hash256::zero(),
Advance state to next slot after importing block (#2174) ## Issue Addressed NA ## Proposed Changes Add an optimization to perform `per_slot_processing` from the *leading-edge* of block processing to the *trailing-edge*. Ultimately, this allows us to import the block at slot `n` faster because we used the tail-end of slot `n - 1` to perform `per_slot_processing`. Additionally, add a "block proposer cache" which allows us to cache the block proposer for some epoch. Since we're now doing trailing-edge `per_slot_processing`, we can prime this cache with the values for the next epoch before those blocks arrive (assuming those blocks don't have some weird forking). There were several ancillary changes required to achieve this: - Remove the `state_root` field of `BeaconSnapshot`, since there's no need to know it on a `pre_state` and in all other cases we can just read it from `block.state_root()`. - This caused some "dust" changes of `snapshot.beacon_state_root` to `snapshot.beacon_state_root()`, where the `BeaconSnapshot::beacon_state_root()` func just reads the state root from the block. - Rename `types::ShuffingId` to `AttestationShufflingId`. I originally did this because I added a `ProposerShufflingId` struct which turned out to be not so useful. I thought this new name was more descriptive so I kept it. - Address https://github.com/ethereum/eth2.0-specs/pull/2196 - Add a debug log when we get a block with an unknown parent. There was previously no logging around this case. - Add a function to `BeaconState` to compute all proposers for an epoch without re-computing the active indices for each slot. ## Additional Info - ~~Blocked on #2173~~ - ~~Blocked on #2179~~ That PR was wrapped into this PR. - There's potentially some places where we could avoid computing the proposer indices in `per_block_processing` but I haven't done this here. These would be an optimization beyond the issue at hand (improving block propagation times) and I think this PR is already doing enough. We can come back for that later. ## TODO - [x] Tidy, improve comments. - [x] ~~Try avoid computing proposer index in `per_block_processing`?~~
2021-02-15 07:17:52 +00:00
current_epoch_shuffling_id: AttestationShufflingId::from_components(
Implement standard eth2.0 API (#1569) - Resolves #1550 - Resolves #824 - Resolves #825 - Resolves #1131 - Resolves #1411 - Resolves #1256 - Resolve #1177 - Includes the `ShufflingId` struct initially defined in #1492. That PR is now closed and the changes are included here, with significant bug fixes. - Implement the https://github.com/ethereum/eth2.0-APIs in a new `http_api` crate using `warp`. This replaces the `rest_api` crate. - Add a new `common/eth2` crate which provides a wrapper around `reqwest`, providing the HTTP client that is used by the validator client and for testing. This replaces the `common/remote_beacon_node` crate. - Create a `http_metrics` crate which is a dedicated server for Prometheus metrics (they are no longer served on the same port as the REST API). We now have flags for `--metrics`, `--metrics-address`, etc. - Allow the `subnet_id` to be an optional parameter for `VerifiedUnaggregatedAttestation::verify`. This means it does not need to be provided unnecessarily by the validator client. - Move `fn map_attestation_committee` in `mod beacon_chain::attestation_verification` to a new `fn with_committee_cache` on the `BeaconChain` so the same cache can be used for obtaining validator duties. - Add some other helpers to `BeaconChain` to assist with common API duties (e.g., `block_root_at_slot`, `head_beacon_block_root`). - Change the `NaiveAggregationPool` so it can index attestations by `hash_tree_root(attestation.data)`. This is a requirement of the API. - Add functions to `BeaconChainHarness` to allow it to create slashings and exits. - Allow for `eth1::Eth1NetworkId` to go to/from a `String`. - Add functions to the `OperationPool` to allow getting all objects in the pool. - Add function to `BeaconState` to check if a committee cache is initialized. - Fix bug where `seconds_per_eth1_block` was not transferring over from `YamlConfig` to `ChainSpec`. - Add the `deposit_contract_address` to `YamlConfig` and `ChainSpec`. We needed to be able to return it in an API response. - Change some uses of serde `serialize_with` and `deserialize_with` to a single use of `with` (code quality). - Impl `Display` and `FromStr` for several BLS fields. - Check for clock discrepancy when VC polls BN for sync state (with +/- 1 slot tolerance). This is not intended to be comprehensive, it was just easy to do. - See #1434 for a per-endpoint overview. - Seeking clarity here: https://github.com/ethereum/eth2.0-APIs/issues/75 - [x] Add docs for prom port to close #1256 - [x] Follow up on this #1177 - [x] ~~Follow up with #1424~~ Will fix in future PR. - [x] Follow up with #1411 - [x] ~~Follow up with #1260~~ Will fix in future PR. - [x] Add quotes to all integers. - [x] Remove `rest_types` - [x] Address missing beacon block error. (#1629) - [x] ~~Add tests for lighthouse/peers endpoints~~ Wontfix - [x] ~~Follow up with validator status proposal~~ Tracked in #1434 - [x] Unify graffiti structs - [x] ~~Start server when waiting for genesis?~~ Will fix in future PR. - [x] TODO in http_api tests - [x] Move lighthouse endpoints off /eth/v1 - [x] Update docs to link to standard - ~~Blocked on #1586~~ Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-09-29 03:46:54 +00:00
Epoch::new(0),
Hash256::zero(),
),
Advance state to next slot after importing block (#2174) ## Issue Addressed NA ## Proposed Changes Add an optimization to perform `per_slot_processing` from the *leading-edge* of block processing to the *trailing-edge*. Ultimately, this allows us to import the block at slot `n` faster because we used the tail-end of slot `n - 1` to perform `per_slot_processing`. Additionally, add a "block proposer cache" which allows us to cache the block proposer for some epoch. Since we're now doing trailing-edge `per_slot_processing`, we can prime this cache with the values for the next epoch before those blocks arrive (assuming those blocks don't have some weird forking). There were several ancillary changes required to achieve this: - Remove the `state_root` field of `BeaconSnapshot`, since there's no need to know it on a `pre_state` and in all other cases we can just read it from `block.state_root()`. - This caused some "dust" changes of `snapshot.beacon_state_root` to `snapshot.beacon_state_root()`, where the `BeaconSnapshot::beacon_state_root()` func just reads the state root from the block. - Rename `types::ShuffingId` to `AttestationShufflingId`. I originally did this because I added a `ProposerShufflingId` struct which turned out to be not so useful. I thought this new name was more descriptive so I kept it. - Address https://github.com/ethereum/eth2.0-specs/pull/2196 - Add a debug log when we get a block with an unknown parent. There was previously no logging around this case. - Add a function to `BeaconState` to compute all proposers for an epoch without re-computing the active indices for each slot. ## Additional Info - ~~Blocked on #2173~~ - ~~Blocked on #2179~~ That PR was wrapped into this PR. - There's potentially some places where we could avoid computing the proposer indices in `per_block_processing` but I haven't done this here. These would be an optimization beyond the issue at hand (improving block propagation times) and I think this PR is already doing enough. We can come back for that later. ## TODO - [x] Tidy, improve comments. - [x] ~~Try avoid computing proposer index in `per_block_processing`?~~
2021-02-15 07:17:52 +00:00
next_epoch_shuffling_id: AttestationShufflingId::from_components(
Implement standard eth2.0 API (#1569) - Resolves #1550 - Resolves #824 - Resolves #825 - Resolves #1131 - Resolves #1411 - Resolves #1256 - Resolve #1177 - Includes the `ShufflingId` struct initially defined in #1492. That PR is now closed and the changes are included here, with significant bug fixes. - Implement the https://github.com/ethereum/eth2.0-APIs in a new `http_api` crate using `warp`. This replaces the `rest_api` crate. - Add a new `common/eth2` crate which provides a wrapper around `reqwest`, providing the HTTP client that is used by the validator client and for testing. This replaces the `common/remote_beacon_node` crate. - Create a `http_metrics` crate which is a dedicated server for Prometheus metrics (they are no longer served on the same port as the REST API). We now have flags for `--metrics`, `--metrics-address`, etc. - Allow the `subnet_id` to be an optional parameter for `VerifiedUnaggregatedAttestation::verify`. This means it does not need to be provided unnecessarily by the validator client. - Move `fn map_attestation_committee` in `mod beacon_chain::attestation_verification` to a new `fn with_committee_cache` on the `BeaconChain` so the same cache can be used for obtaining validator duties. - Add some other helpers to `BeaconChain` to assist with common API duties (e.g., `block_root_at_slot`, `head_beacon_block_root`). - Change the `NaiveAggregationPool` so it can index attestations by `hash_tree_root(attestation.data)`. This is a requirement of the API. - Add functions to `BeaconChainHarness` to allow it to create slashings and exits. - Allow for `eth1::Eth1NetworkId` to go to/from a `String`. - Add functions to the `OperationPool` to allow getting all objects in the pool. - Add function to `BeaconState` to check if a committee cache is initialized. - Fix bug where `seconds_per_eth1_block` was not transferring over from `YamlConfig` to `ChainSpec`. - Add the `deposit_contract_address` to `YamlConfig` and `ChainSpec`. We needed to be able to return it in an API response. - Change some uses of serde `serialize_with` and `deserialize_with` to a single use of `with` (code quality). - Impl `Display` and `FromStr` for several BLS fields. - Check for clock discrepancy when VC polls BN for sync state (with +/- 1 slot tolerance). This is not intended to be comprehensive, it was just easy to do. - See #1434 for a per-endpoint overview. - Seeking clarity here: https://github.com/ethereum/eth2.0-APIs/issues/75 - [x] Add docs for prom port to close #1256 - [x] Follow up on this #1177 - [x] ~~Follow up with #1424~~ Will fix in future PR. - [x] Follow up with #1411 - [x] ~~Follow up with #1260~~ Will fix in future PR. - [x] Add quotes to all integers. - [x] Remove `rest_types` - [x] Address missing beacon block error. (#1629) - [x] ~~Add tests for lighthouse/peers endpoints~~ Wontfix - [x] ~~Follow up with validator status proposal~~ Tracked in #1434 - [x] Unify graffiti structs - [x] ~~Start server when waiting for genesis?~~ Will fix in future PR. - [x] TODO in http_api tests - [x] Move lighthouse endpoints off /eth/v1 - [x] Update docs to link to standard - ~~Blocked on #1586~~ Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-09-29 03:46:54 +00:00
Epoch::new(0),
Hash256::zero(),
),
v0.12 fork choice update (#1229) * Incomplete scraps * Add progress on new fork choice impl * Further progress * First complete compiling version * Remove chain reference * Add new lmd_ghost crate * Start integrating into beacon chain * Update `milagro_bls` to new release (#1183) * Update milagro_bls to new release Signed-off-by: Kirk Baird <baird.k@outlook.com> * Tidy up fake cryptos Signed-off-by: Kirk Baird <baird.k@outlook.com> * move SecretHash to bls and put plaintext back Signed-off-by: Kirk Baird <baird.k@outlook.com> * Update state processing for v0.12 * Fix EF test runners for v0.12 * Fix some tests * Fix broken attestation verification test * More test fixes * Rough beacon chain impl working * Remove fork_choice_2 * Remove checkpoint manager * Half finished ssz impl * Add missed file * Add persistence * Tidy, fix some compile errors * Remove RwLock from ProtoArrayForkChoice * Fix store-based compile errors * Add comments, tidy * Move function out of ForkChoice struct * Start testing * More testing * Fix compile error * Tidy beacon_chain::fork_choice * Queue attestations from the current slot * Allow fork choice to handle prior-to-genesis start * Improve error granularity * Test attestation dequeuing * Process attestations during block * Store target root in fork choice * Move fork choice verification into new crate * Update tests * Consensus updates for v0.12 (#1228) * Update state processing for v0.12 * Fix EF test runners for v0.12 * Fix some tests * Fix broken attestation verification test * More test fixes * Fix typo found in review * Add `Block` struct to ProtoArray * Start fixing get_ancestor * Add rough progress on testing * Get fork choice tests working * Progress with testing * Fix partialeq impl * Move slot clock from fc_store * Improve testing * Add testing for best justified * Add clone back to SystemTimeSlotClock * Add balances test * Start adding balances cache again * Wire-in balances cache * Improve tests * Remove commented-out tests * Remove beacon_chain::ForkChoice * Rename crates * Update wider codebase to new fork_choice layout * Move advance_slot in test harness * Tidy ForkChoice::update_time * Fix verification tests * Fix compile error with iter::once * Fix fork choice tests * Ensure block attestations are processed * Fix failing beacon_chain tests * Add first invalid block check * Add finalized block check * Progress with testing, new store builder * Add fixes to get_ancestor * Fix old genesis justification test * Fix remaining fork choice tests * Change root iteration method * Move on_verified_block * Remove unused method * Start adding attestation verification tests * Add invalid ffg target test * Add target epoch test * Add queued attestation test * Remove old fork choice verification tests * Tidy, add test * Move fork choice lock drop * Rename BeaconForkChoiceStore * Add comments, tidy BeaconForkChoiceStore * Update metrics, rename fork_choice_store.rs * Remove genesis_block_root from ForkChoice * Tidy * Update fork_choice comments * Tidy, add comments * Tidy, simplify ForkChoice, fix compile issue * Tidy, removed dead file * Increase http request timeout * Fix failing rest_api test * Set HTTP timeout back to 5s * Apply fix to get_ancestor * Address Michael's comments * Fix typo * Revert "Fix broken attestation verification test" This reverts commit 722cdc903b12611de27916a57eeecfa3224f2279. Co-authored-by: Kirk Baird <baird.k@outlook.com> Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-06-17 01:10:22 +00:00
justified_epoch,
finalized_epoch,
execution_status,
v0.12 fork choice update (#1229) * Incomplete scraps * Add progress on new fork choice impl * Further progress * First complete compiling version * Remove chain reference * Add new lmd_ghost crate * Start integrating into beacon chain * Update `milagro_bls` to new release (#1183) * Update milagro_bls to new release Signed-off-by: Kirk Baird <baird.k@outlook.com> * Tidy up fake cryptos Signed-off-by: Kirk Baird <baird.k@outlook.com> * move SecretHash to bls and put plaintext back Signed-off-by: Kirk Baird <baird.k@outlook.com> * Update state processing for v0.12 * Fix EF test runners for v0.12 * Fix some tests * Fix broken attestation verification test * More test fixes * Rough beacon chain impl working * Remove fork_choice_2 * Remove checkpoint manager * Half finished ssz impl * Add missed file * Add persistence * Tidy, fix some compile errors * Remove RwLock from ProtoArrayForkChoice * Fix store-based compile errors * Add comments, tidy * Move function out of ForkChoice struct * Start testing * More testing * Fix compile error * Tidy beacon_chain::fork_choice * Queue attestations from the current slot * Allow fork choice to handle prior-to-genesis start * Improve error granularity * Test attestation dequeuing * Process attestations during block * Store target root in fork choice * Move fork choice verification into new crate * Update tests * Consensus updates for v0.12 (#1228) * Update state processing for v0.12 * Fix EF test runners for v0.12 * Fix some tests * Fix broken attestation verification test * More test fixes * Fix typo found in review * Add `Block` struct to ProtoArray * Start fixing get_ancestor * Add rough progress on testing * Get fork choice tests working * Progress with testing * Fix partialeq impl * Move slot clock from fc_store * Improve testing * Add testing for best justified * Add clone back to SystemTimeSlotClock * Add balances test * Start adding balances cache again * Wire-in balances cache * Improve tests * Remove commented-out tests * Remove beacon_chain::ForkChoice * Rename crates * Update wider codebase to new fork_choice layout * Move advance_slot in test harness * Tidy ForkChoice::update_time * Fix verification tests * Fix compile error with iter::once * Fix fork choice tests * Ensure block attestations are processed * Fix failing beacon_chain tests * Add first invalid block check * Add finalized block check * Progress with testing, new store builder * Add fixes to get_ancestor * Fix old genesis justification test * Fix remaining fork choice tests * Change root iteration method * Move on_verified_block * Remove unused method * Start adding attestation verification tests * Add invalid ffg target test * Add target epoch test * Add queued attestation test * Remove old fork choice verification tests * Tidy, add test * Move fork choice lock drop * Rename BeaconForkChoiceStore * Add comments, tidy BeaconForkChoiceStore * Update metrics, rename fork_choice_store.rs * Remove genesis_block_root from ForkChoice * Tidy * Update fork_choice comments * Tidy, add comments * Tidy, simplify ForkChoice, fix compile issue * Tidy, removed dead file * Increase http request timeout * Fix failing rest_api test * Set HTTP timeout back to 5s * Apply fix to get_ancestor * Address Michael's comments * Fix typo * Revert "Fix broken attestation verification test" This reverts commit 722cdc903b12611de27916a57eeecfa3224f2279. Co-authored-by: Kirk Baird <baird.k@outlook.com> Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-06-17 01:10:22 +00:00
};
fork_choice.process_block(block).unwrap_or_else(|e| {
panic!(
"process_block op at index {} returned error: {:?}",
op_index, e
Optimize attestation processing (#841) * Start updating types * WIP * Signature hacking * Existing EF tests passing with fake_crypto * Updates * Delete outdated API spec * The refactor continues * It compiles * WIP test fixes * All release tests passing bar genesis state parsing * Update and test YamlConfig * Update to spec v0.10 compatible BLS * Updates to BLS EF tests * Add EF test for AggregateVerify And delete unused hash2curve tests for uncompressed points * Update EF tests to v0.10.1 * Use optional block root correctly in block proc * Use genesis fork in deposit domain. All tests pass * Cargo fmt * Fast aggregate verify test * Update REST API docs * Cargo fmt * Fix unused import * Bump spec tags to v0.10.1 * Add `seconds_per_eth1_block` to chainspec * Update to timestamp based eth1 voting scheme * Return None from `get_votes_to_consider` if block cache is empty * Handle overflows in `is_candidate_block` * Revert to failing tests * Fix eth1 data sets test * Choose default vote according to spec * Fix collect_valid_votes tests * Fix `get_votes_to_consider` to choose all eligible blocks * Uncomment winning_vote tests * Add comments; remove unused code * Reduce seconds_per_eth1_block for simulation * Addressed review comments * Add test for default vote case * Fix logs * Remove unused functions * Meter default eth1 votes * Fix comments * Address review comments; remove unused dependency * Add first attempt at attestation proc. re-write * Add version 2 of attestation processing * Minor fixes * Add validator pubkey cache * Make get_indexed_attestation take a committee * Link signature processing into new attn verification * First working version * Ensure pubkey cache is updated * Add more metrics, slight optimizations * Clone committee cache during attestation processing * Update shuffling cache during block processing * Remove old commented-out code * Fix shuffling cache insert bug * Used indexed attestation in fork choice * Restructure attn processing, add metrics * Add more detailed metrics * Tidy, fix failing tests * Fix failing tests, tidy * Disable/delete two outdated tests * Tidy * Add pubkey cache persistence file * Add more comments * Integrate persistence file into builder * Add pubkey cache tests * Add data_dir to beacon chain builder * Remove Option in pubkey cache persistence file * Ensure consistency between datadir/data_dir * Fix failing network test * Tidy * Fix todos * Add attestation processing tests * Add another test * Only run attestation tests in release * Make attestation tests MainnetEthSpec * Address Michael's comments * Remove redundant check * Fix warning * Fix failing test Co-authored-by: Michael Sproul <micsproul@gmail.com> Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
2020-03-05 06:19:35 +00:00
)
});
Add proto_array fork choice (#804) * Start implementing proto_array * Add progress * Add unfinished progress * Add further progress * Add progress * Add tree filtering * Add half-finished modifications * Add refactored version * Tidy, add incomplete LmdGhost impl * Move impls in LmdGhost trait def * Remove old reduced_tree fork choice * Combine two functions in to `compute_deltas` * Start testing * Add more compute_deltas tests * Add fork choice testing * Add more fork choice testing * Add more fork choice tests * Add more testing to proto-array * Remove old tests * Modify tests * Add more tests * Add more testing * Add comments and fixes * Re-organise crate * Tidy, finish pruning tests * Add ssz encoding, other pub fns * Rename lmd_ghost > proto_array_fork_choice * Integrate proto_array into lighthouse * Add first pass at fixing filter * Clean out old comments * Add more comments * Attempt to fix prune error * Adjust TODO * Fix test compile errors * Add extra justification change check * Update cargo.lock * Fix fork choice test compile errors * Most remove ffg_update_required * Fix bug with epoch of attestation votes * Start adding new test format * Make fork choice tests declarative * Create test def concept * Move test defs into crate * Add binary, re-org crate * Shuffle files * Start adding ffg tests * Add more fork choice tests * Add fork choice JSON dumping * Add more detail to best node error * Ensure fin+just checkpoints from from same block * Rename JustificationManager * Move checkpoint manager into own file * Tidy * Add targetted logging for sneaky sync bug * Fix justified balances bug * Add cache metrics * Add metrics for log levels * Fix bug in checkpoint manager * Fix compile error in fork choice tests * Ignore duplicate blocks in fork choice * Add block to fock choice before db * Rename on_new_block fn * Fix spec inconsistency in `CheckpointManager` * Remove BlockRootTree * Remove old reduced_tree code fragment * Add API endpoint for fork choice * Add more ffg tests * Remove block_root_tree reminents * Ensure effective balances are used * Remove old debugging code, fix API fault * Add check to ensure parent block is in fork choice * Update readme dates * Fix readme * Tidy checkpoint manager * Remove fork choice yaml files from repo * Remove fork choice yaml from repo * General tidy * Address majority of Michael's comments * Tidy bin/lib business * Remove dangling file * Undo changes for rpc/handler from master * Revert "Undo changes for rpc/handler from master" This reverts commit 876edff0e4a501aafbb47113454852826dcc24e8. Co-authored-by: Age Manning <Age@AgeManning.com>
2020-01-29 04:05:00 +00:00
check_bytes_round_trip(&fork_choice);
}
Operation::ProcessAttestation {
validator_index,
block_root,
target_epoch,
} => {
fork_choice
.process_attestation(validator_index, block_root, target_epoch)
.unwrap_or_else(|_| {
panic!(
"process_attestation op at index {} returned error",
op_index
)
});
Add proto_array fork choice (#804) * Start implementing proto_array * Add progress * Add unfinished progress * Add further progress * Add progress * Add tree filtering * Add half-finished modifications * Add refactored version * Tidy, add incomplete LmdGhost impl * Move impls in LmdGhost trait def * Remove old reduced_tree fork choice * Combine two functions in to `compute_deltas` * Start testing * Add more compute_deltas tests * Add fork choice testing * Add more fork choice testing * Add more fork choice tests * Add more testing to proto-array * Remove old tests * Modify tests * Add more tests * Add more testing * Add comments and fixes * Re-organise crate * Tidy, finish pruning tests * Add ssz encoding, other pub fns * Rename lmd_ghost > proto_array_fork_choice * Integrate proto_array into lighthouse * Add first pass at fixing filter * Clean out old comments * Add more comments * Attempt to fix prune error * Adjust TODO * Fix test compile errors * Add extra justification change check * Update cargo.lock * Fix fork choice test compile errors * Most remove ffg_update_required * Fix bug with epoch of attestation votes * Start adding new test format * Make fork choice tests declarative * Create test def concept * Move test defs into crate * Add binary, re-org crate * Shuffle files * Start adding ffg tests * Add more fork choice tests * Add fork choice JSON dumping * Add more detail to best node error * Ensure fin+just checkpoints from from same block * Rename JustificationManager * Move checkpoint manager into own file * Tidy * Add targetted logging for sneaky sync bug * Fix justified balances bug * Add cache metrics * Add metrics for log levels * Fix bug in checkpoint manager * Fix compile error in fork choice tests * Ignore duplicate blocks in fork choice * Add block to fock choice before db * Rename on_new_block fn * Fix spec inconsistency in `CheckpointManager` * Remove BlockRootTree * Remove old reduced_tree code fragment * Add API endpoint for fork choice * Add more ffg tests * Remove block_root_tree reminents * Ensure effective balances are used * Remove old debugging code, fix API fault * Add check to ensure parent block is in fork choice * Update readme dates * Fix readme * Tidy checkpoint manager * Remove fork choice yaml files from repo * Remove fork choice yaml from repo * General tidy * Address majority of Michael's comments * Tidy bin/lib business * Remove dangling file * Undo changes for rpc/handler from master * Revert "Undo changes for rpc/handler from master" This reverts commit 876edff0e4a501aafbb47113454852826dcc24e8. Co-authored-by: Age Manning <Age@AgeManning.com>
2020-01-29 04:05:00 +00:00
check_bytes_round_trip(&fork_choice);
}
Operation::Prune {
finalized_root,
prune_threshold,
expected_len,
} => {
fork_choice.set_prune_threshold(prune_threshold);
fork_choice
.maybe_prune(finalized_root)
.expect("update_finalized_root op at index {} returned error");
// Ensure that no pruning happened.
assert_eq!(
fork_choice.len(),
expected_len,
"Prune op at index {} failed with {} instead of {}",
op_index,
fork_choice.len(),
expected_len
);
}
}
}
}
}
/// Gives a hash that is not the zero hash (unless i is `usize::max_value)`.
fn get_hash(i: u64) -> Hash256 {
Hash256::from_low_u64_be(i)
}
fn check_bytes_round_trip(original: &ProtoArrayForkChoice) {
let bytes = original.as_bytes();
let decoded =
ProtoArrayForkChoice::from_bytes(&bytes).expect("fork choice should decode from bytes");
assert!(
*original == decoded,
"fork choice should encode and decode without change"
);
}