lighthouse/beacon_node/beacon_chain/src/beacon_chain.rs

1315 lines
51 KiB
Rust
Raw Normal View History

use crate::checkpoint::CheckPoint;
use crate::errors::{BeaconChainError as Error, BlockProductionError};
use crate::fork_choice::{Error as ForkChoiceError, ForkChoice};
use crate::iter::{ReverseBlockRootIterator, ReverseStateRootIterator};
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
use crate::metrics;
use crate::persisted_beacon_chain::{PersistedBeaconChain, BEACON_CHAIN_DB_KEY};
use lmd_ghost::LmdGhost;
2019-06-20 00:59:19 +00:00
use log::trace;
use operation_pool::DepositInsertStatus;
2019-06-18 07:55:18 +00:00
use operation_pool::{OperationPool, PersistedOperationPool};
use parking_lot::{RwLock, RwLockReadGuard};
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
use slog::{error, info, warn, Logger};
use slot_clock::SlotClock;
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
use state_processing::per_block_processing::{
errors::{
AttestationValidationError, AttesterSlashingValidationError, DepositValidationError,
ExitValidationError, ProposerSlashingValidationError, TransferValidationError,
},
verify_attestation_for_state, VerifySignatures,
};
use state_processing::{
2019-03-07 01:25:29 +00:00
per_block_processing, per_block_processing_without_verifying_block_signature,
per_slot_processing, BlockProcessingError,
};
use std::sync::Arc;
use store::iter::{BlockRootsIterator, StateRootsIterator};
2019-05-21 08:20:23 +00:00
use store::{Error as DBError, Store};
use tree_hash::TreeHash;
use types::*;
// Text included in blocks.
// Must be 32-bytes or panic.
//
// |-------must be this long------|
pub const GRAFFITI: &str = "sigp/lighthouse-0.0.0-prerelease";
#[derive(Debug, PartialEq)]
pub enum BlockProcessingOutcome {
/// Block was valid and imported into the block graph.
2019-06-20 08:46:03 +00:00
Processed { block_root: Hash256 },
/// The blocks parent_root is unknown.
ParentUnknown { parent: Hash256 },
/// The block slot is greater than the present slot.
FutureSlot {
present_slot: Slot,
block_slot: Slot,
},
/// The block state_root does not match the generated state.
StateRootMismatch,
/// The block was a genesis block, these blocks cannot be re-imported.
GenesisBlock,
/// The slot is finalized, no need to import.
FinalizedSlot,
/// Block is already known, no need to re-import.
BlockIsAlreadyKnown,
/// The block could not be applied to the state, it is invalid.
PerBlockProcessingError(BlockProcessingError),
}
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
#[derive(Debug, PartialEq)]
pub enum AttestationProcessingOutcome {
Processed,
UnknownHeadBlock {
beacon_block_root: Hash256,
},
/// The attestation is attesting to a state that is later than itself. (Viz., attesting to the
/// future).
AttestsToFutureState {
state: Slot,
attestation: Slot,
},
/// The slot is finalized, no need to import.
FinalizedSlot {
attestation: Epoch,
finalized: Epoch,
},
Invalid(AttestationValidationError),
}
Bootstrap (#501) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement * Add first attempts at HTTP bootstrap * Add beacon_block methods to rest api * Fix serde for block.body.grafitti * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Add network routes to API * Fix rustc warnings * Add best_slot method * Add --bootstrap arg to beacon node * Get bootstrapper working for ENR address * Store intermediate states during block processing * Allow bootstrapper to scrape libp2p address * Update bootstrapper libp2p address finding * Add comments * Tidy API to be more consistent with recent decisions * Address some review comments * Make BeaconChainTypes Send + Sync + 'static * Add `/network/listen_port` API endpoint * Abandon starting the node if libp2p doesn't start * Update bootstrapper for API changes * Remove unnecessary trait bounds
2019-08-23 05:53:53 +00:00
pub trait BeaconChainTypes: Send + Sync + 'static {
type Store: store::Store;
type SlotClock: slot_clock::SlotClock;
type LmdGhost: LmdGhost<Self::Store, Self::EthSpec>;
type EthSpec: types::EthSpec;
}
/// Represents the "Beacon Chain" component of Ethereum 2.0. Allows import of blocks and block
/// operations and chooses a canonical head.
pub struct BeaconChain<T: BeaconChainTypes> {
pub spec: ChainSpec,
/// Persistent storage for blocks, states, etc. Typically an on-disk store, such as LevelDB.
pub store: Arc<T::Store>,
/// Reports the current slot, typically based upon the system clock.
pub slot_clock: T::SlotClock,
/// Stores all operations (e.g., `Attestation`, `Deposit`, etc) that are candidates for
/// inclusion in a block.
pub op_pool: OperationPool<T::EthSpec>,
2019-07-26 19:26:06 +00:00
/// Stores a "snapshot" of the chain at the time the head-of-the-chain block was received.
canonical_head: RwLock<CheckPoint<T::EthSpec>>,
/// The same state from `self.canonical_head`, but updated at the start of each slot with a
2019-07-26 19:26:06 +00:00
/// skip slot if no block is received. This is effectively a cache that avoids repeating calls
/// to `per_slot_processing`.
state: RwLock<BeaconState<T::EthSpec>>,
/// The root of the genesis block.
pub genesis_block_root: Hash256,
/// A state-machine that is updated with information from the network and chooses a canonical
/// head block.
2019-06-15 19:05:34 +00:00
pub fork_choice: ForkChoice<T>,
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
/// Logging to CLI, etc.
log: Logger,
}
impl<T: BeaconChainTypes> BeaconChain<T> {
/// Instantiate a new Beacon Chain, from genesis.
pub fn from_genesis(
store: Arc<T::Store>,
slot_clock: T::SlotClock,
mut genesis_state: BeaconState<T::EthSpec>,
mut genesis_block: BeaconBlock<T::EthSpec>,
spec: ChainSpec,
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
log: Logger,
) -> Result<Self, Error> {
2019-06-16 06:24:33 +00:00
genesis_state.build_all_caches(&spec)?;
let genesis_state_root = genesis_state.canonical_root();
store.put(&genesis_state_root, &genesis_state)?;
genesis_block.state_root = genesis_state_root;
let genesis_block_root = genesis_block.block_header().canonical_root();
store.put(&genesis_block_root, &genesis_block)?;
2019-06-07 06:46:07 +00:00
// Also store the genesis block under the `ZERO_HASH` key.
let genesis_block_root = genesis_block.canonical_root();
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
store.put(&Hash256::zero(), &genesis_block)?;
2019-06-07 06:46:07 +00:00
let canonical_head = RwLock::new(CheckPoint::new(
genesis_block.clone(),
genesis_block_root,
genesis_state.clone(),
genesis_state_root,
));
info!(log, "BeaconChain init";
"genesis_validator_count" => genesis_state.validators.len(),
"genesis_state_root" => format!("{}", genesis_state_root),
"genesis_block_root" => format!("{}", genesis_block_root),
);
Ok(Self {
spec,
slot_clock,
op_pool: OperationPool::new(),
state: RwLock::new(genesis_state),
canonical_head,
genesis_block_root,
fork_choice: ForkChoice::new(store.clone(), &genesis_block, genesis_block_root),
2019-06-15 22:19:08 +00:00
store,
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
log,
})
}
/// Attempt to load an existing instance from the given `store`.
pub fn from_store(
store: Arc<T::Store>,
spec: ChainSpec,
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
log: Logger,
) -> Result<Option<BeaconChain<T>>, Error> {
let key = Hash256::from_slice(&BEACON_CHAIN_DB_KEY.as_bytes());
let p: PersistedBeaconChain<T> = match store.get(&key) {
Err(e) => return Err(e.into()),
Ok(None) => return Ok(None),
Ok(Some(p)) => p,
};
let slot_clock = T::SlotClock::new(
spec.genesis_slot,
p.state.genesis_time,
spec.seconds_per_slot,
);
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
let last_finalized_root = p.canonical_head.beacon_state.finalized_checkpoint.root;
let last_finalized_block = &p.canonical_head.beacon_block;
2019-06-18 07:55:18 +00:00
let op_pool = p.op_pool.into_operation_pool(&p.state, &spec);
Ok(Some(BeaconChain {
spec,
slot_clock,
fork_choice: ForkChoice::new(store.clone(), last_finalized_block, last_finalized_root),
2019-06-18 07:55:18 +00:00
op_pool,
canonical_head: RwLock::new(p.canonical_head),
state: RwLock::new(p.state),
genesis_block_root: p.genesis_block_root,
store,
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
log,
}))
}
/// Attempt to save this instance to `self.store`.
pub fn persist(&self) -> Result<(), Error> {
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
let timer = metrics::start_timer(&metrics::PERSIST_CHAIN);
let p: PersistedBeaconChain<T> = PersistedBeaconChain {
canonical_head: self.canonical_head.read().clone(),
2019-06-18 07:55:18 +00:00
op_pool: PersistedOperationPool::from_operation_pool(&self.op_pool),
genesis_block_root: self.genesis_block_root,
state: self.state.read().clone(),
};
let key = Hash256::from_slice(&BEACON_CHAIN_DB_KEY.as_bytes());
self.store.put(&key, &p)?;
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::stop_timer(timer);
Ok(())
}
/// Returns the beacon block body for each beacon block root in `roots`.
///
/// Fails if any root in `roots` does not have a corresponding block.
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
pub fn get_block_bodies(
&self,
roots: &[Hash256],
) -> Result<Vec<BeaconBlockBody<T::EthSpec>>, Error> {
let bodies: Result<Vec<_>, _> = roots
.iter()
.map(|root| match self.get_block(root)? {
Some(block) => Ok(block.body),
None => Err(Error::DBInconsistent(format!("Missing block: {}", root))),
})
.collect();
Ok(bodies?)
}
/// Returns the beacon block header for each beacon block root in `roots`.
///
/// Fails if any root in `roots` does not have a corresponding block.
pub fn get_block_headers(&self, roots: &[Hash256]) -> Result<Vec<BeaconBlockHeader>, Error> {
let headers: Result<Vec<BeaconBlockHeader>, _> = roots
.iter()
.map(|root| match self.get_block(root)? {
Some(block) => Ok(block.block_header()),
None => Err(Error::DBInconsistent("Missing block".into())),
})
.collect();
Ok(headers?)
}
/// Iterates across all `(block_root, slot)` pairs from the head of the chain (inclusive) to
/// the earliest reachable ancestor (may or may not be genesis).
2019-06-05 03:33:32 +00:00
///
/// ## Notes
2019-06-05 03:33:32 +00:00
///
/// `slot` always decreases by `1`.
/// - Skipped slots contain the root of the closest prior
/// non-skipped slot (identical to the way they are stored in `state.block_roots`) .
/// - Iterator returns `(Hash256, Slot)`.
/// - As this iterator starts at the `head` of the chain (viz., the best block), the first slot
/// returned may be earlier than the wall-clock slot.
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
pub fn rev_iter_block_roots(&self) -> ReverseBlockRootIterator<T::EthSpec, T::Store> {
let state = &self.head().beacon_state;
let block_root = self.head().beacon_block_root;
let block_slot = state.slot;
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
let iter = BlockRootsIterator::owned(self.store.clone(), state.clone());
ReverseBlockRootIterator::new((block_root, block_slot), iter)
}
/// Iterates across all `(state_root, slot)` pairs from the head of the chain (inclusive) to
/// the earliest reachable ancestor (may or may not be genesis).
///
/// ## Notes
///
/// `slot` always decreases by `1`.
/// - Iterator returns `(Hash256, Slot)`.
/// - As this iterator starts at the `head` of the chain (viz., the best block), the first slot
/// returned may be earlier than the wall-clock slot.
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
pub fn rev_iter_state_roots(&self) -> ReverseStateRootIterator<T::EthSpec, T::Store> {
let state = &self.head().beacon_state;
let state_root = self.head().beacon_state_root;
let state_slot = state.slot;
2019-08-05 06:25:21 +00:00
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
let iter = StateRootsIterator::owned(self.store.clone(), state.clone());
ReverseStateRootIterator::new((state_root, state_slot), iter)
2019-06-05 03:33:32 +00:00
}
2019-03-23 02:23:44 +00:00
/// Returns the block at the given root, if any.
///
/// ## Errors
///
/// May return a database error.
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
pub fn get_block(
&self,
block_root: &Hash256,
) -> Result<Option<BeaconBlock<T::EthSpec>>, Error> {
2019-05-21 07:27:06 +00:00
Ok(self.store.get(block_root)?)
2019-03-23 02:23:44 +00:00
}
/// Returns a read-lock guarded `BeaconState` which is the `canonical_head` that has been
/// updated to match the current slot clock.
pub fn speculative_state(&self) -> Result<RwLockReadGuard<BeaconState<T::EthSpec>>, Error> {
Ok(self.state.read())
}
/// Returns a read-lock guarded `CheckPoint` struct for reading the head (as chosen by the
/// fork-choice rule).
///
/// It is important to note that the `beacon_state` returned may not match the present slot. It
/// is the state as it was when the head block was received, which could be some slots prior to
/// now.
pub fn head<'a>(&'a self) -> RwLockReadGuard<'a, CheckPoint<T::EthSpec>> {
self.canonical_head.read()
}
/// Returns the slot of the highest block in the canonical chain.
pub fn best_slot(&self) -> Slot {
self.canonical_head.read().beacon_block.slot
}
/// Ensures the current canonical `BeaconState` has been transitioned to match the `slot_clock`.
pub fn catchup_state(&self) -> Result<(), Error> {
let spec = &self.spec;
let present_slot = match self.slot_clock.present_slot() {
Ok(Some(slot)) => slot,
_ => return Err(Error::UnableToReadSlot),
};
2019-06-20 00:59:19 +00:00
if self.state.read().slot < present_slot {
let mut state = self.state.write();
2019-06-20 00:59:19 +00:00
// If required, transition the new state to the present slot.
for _ in state.slot.as_u64()..present_slot.as_u64() {
// Ensure the next epoch state caches are built in case of an epoch transition.
state.build_committee_cache(RelativeEpoch::Next, spec)?;
2019-06-20 00:59:19 +00:00
per_slot_processing(&mut *state, spec)?;
}
2019-06-20 00:59:19 +00:00
state.build_all_caches(spec)?;
}
Ok(())
}
/// Build all of the caches on the current state.
///
/// Ideally this shouldn't be required, however we leave it here for testing.
pub fn ensure_state_caches_are_built(&self) -> Result<(), Error> {
self.state.write().build_all_caches(&self.spec)?;
Ok(())
}
/// Returns the validator index (if any) for the given public key.
///
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
/// Information is retrieved from the present `beacon_state.validators`.
pub fn validator_index(&self, pubkey: &PublicKey) -> Option<usize> {
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
for (i, validator) in self.head().beacon_state.validators.iter().enumerate() {
if validator.pubkey == *pubkey {
return Some(i);
}
}
None
}
/// Reads the slot clock, returns `None` if the slot is unavailable.
///
/// The slot might be unavailable due to an error with the system clock, or if the present time
/// is before genesis (i.e., a negative slot).
///
/// This is distinct to `present_slot`, which simply reads the latest state. If a
/// call to `read_slot_clock` results in a higher slot than a call to `present_slot`,
/// `self.state` should undergo per slot processing.
pub fn read_slot_clock(&self) -> Option<Slot> {
match self.slot_clock.present_slot() {
Ok(Some(some_slot)) => Some(some_slot),
Ok(None) => None,
_ => None,
}
}
2019-03-26 23:36:20 +00:00
/// Reads the slot clock (see `self.read_slot_clock()` and returns the number of slots since
/// genesis.
pub fn slots_since_genesis(&self) -> Option<SlotHeight> {
let now = self.read_slot_clock()?;
let genesis_slot = self.spec.genesis_slot;
2019-03-26 23:36:20 +00:00
if now < genesis_slot {
2019-03-26 23:36:20 +00:00
None
} else {
Some(SlotHeight::from(now.as_u64() - genesis_slot.as_u64()))
2019-03-26 23:36:20 +00:00
}
}
/// Returns slot of the present state.
///
/// This is distinct to `read_slot_clock`, which reads from the actual system clock. If
/// `self.state` has not been transitioned it is possible for the system clock to be on a
/// different slot to what is returned from this call.
pub fn present_slot(&self) -> Slot {
self.state.read().slot
}
/// Returns the block proposer for a given slot.
///
2019-06-20 00:59:19 +00:00
/// Information is read from the present `beacon_state` shuffling, only information from the
/// present epoch is available.
pub fn block_proposer(&self, slot: Slot) -> Result<usize, Error> {
// Ensures that the present state has been advanced to the present slot, skipping slots if
// blocks are not present.
self.catchup_state()?;
// TODO: permit lookups of the proposer at any slot.
let index = self.state.read().get_beacon_proposer_index(
slot,
RelativeEpoch::Current,
&self.spec,
)?;
Ok(index)
}
/// Returns the attestation slot and shard for a given validator index.
///
/// Information is read from the current state, so only information from the present and prior
/// epoch is available.
2019-07-26 19:26:06 +00:00
pub fn validator_attestation_slot_and_shard(
&self,
validator_index: usize,
) -> Result<Option<(Slot, u64)>, BeaconStateError> {
trace!(
2019-07-26 19:26:06 +00:00
"BeaconChain::validator_attestation_slot_and_shard: validator_index: {}",
validator_index
);
if let Some(attestation_duty) = self
.state
.read()
.get_attestation_duties(validator_index, RelativeEpoch::Current)?
{
Ok(Some((attestation_duty.slot, attestation_duty.shard)))
} else {
Ok(None)
}
}
/// Produce an `AttestationData` that is valid for the present `slot` and given `shard`.
///
/// Attests to the canonical chain.
pub fn produce_attestation_data(&self, shard: u64) -> Result<AttestationData, Error> {
2019-06-20 08:46:03 +00:00
let state = self.state.read();
let head_block_root = self.head().beacon_block_root;
let head_block_slot = self.head().beacon_block.slot;
self.produce_attestation_data_for_block(shard, head_block_root, head_block_slot, &*state)
}
/// Produce an `AttestationData` that attests to the chain denoted by `block_root` and `state`.
///
/// Permits attesting to any arbitrary chain. Generally, the `produce_attestation_data`
/// function should be used as it attests to the canonical chain.
2019-06-20 08:46:03 +00:00
pub fn produce_attestation_data_for_block(
&self,
shard: u64,
head_block_root: Hash256,
head_block_slot: Slot,
state: &BeaconState<T::EthSpec>,
) -> Result<AttestationData, Error> {
// Collect some metrics.
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::inc_counter(&metrics::ATTESTATION_PRODUCTION_REQUESTS);
let timer = metrics::start_timer(&metrics::ATTESTATION_PRODUCTION_TIMES);
2019-06-20 08:46:03 +00:00
let slots_per_epoch = T::EthSpec::slots_per_epoch();
let current_epoch_start_slot = state.current_epoch().start_slot(slots_per_epoch);
// The `target_root` is the root of the first block of the current epoch.
//
// The `state` does not know the root of the block for it's current slot (it only knows
// about blocks from prior slots). This creates an edge-case when the state is on the first
// slot of the epoch -- we're unable to obtain the `target_root` because it is not a prior
// root.
//
// This edge case is handled in two ways:
//
// - If the head block is on the same slot as the state, we use it's root.
// - Otherwise, assume the current slot has been skipped and use the block root from the
// prior slot.
//
// For all other cases, we simply read the `target_root` from `state.latest_block_roots`.
let target_root = if state.slot == current_epoch_start_slot {
if head_block_slot == current_epoch_start_slot {
head_block_root
} else {
*state.get_block_root(current_epoch_start_slot - 1)?
}
} else {
*state.get_block_root(current_epoch_start_slot)?
};
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
let target = Checkpoint {
epoch: state.current_epoch(),
root: target_root,
};
2019-06-20 08:46:03 +00:00
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
let parent_crosslink = state.get_current_crosslink(shard)?;
let crosslink = Crosslink {
shard,
parent_root: Hash256::from_slice(&parent_crosslink.tree_hash_root()),
start_epoch: parent_crosslink.end_epoch,
end_epoch: std::cmp::min(
target.epoch,
parent_crosslink.end_epoch + self.spec.max_epochs_per_crosslink,
),
data_root: Hash256::zero(),
};
2019-06-20 08:46:03 +00:00
// Collect some metrics.
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::inc_counter(&metrics::ATTESTATION_PRODUCTION_SUCCESSES);
metrics::stop_timer(timer);
2019-06-20 08:46:03 +00:00
Ok(AttestationData {
beacon_block_root: head_block_root,
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
source: state.current_justified_checkpoint.clone(),
target,
crosslink,
2019-06-20 08:46:03 +00:00
})
}
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
/// Accept a new, potentially invalid attestation from the network.
///
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
/// If valid, the attestation is added to `self.op_pool` and `self.fork_choice`.
///
/// Returns an `Ok(AttestationProcessingOutcome)` if the chain was able to make a determination
/// about the `attestation` (whether it was invalid or not). Returns an `Err` if there was an
/// error during this process and no determination was able to be made.
///
/// ## Notes
///
/// - Whilst the `attestation` is added to fork choice, the head is not updated. That must be
/// done separately.
pub fn process_attestation(
&self,
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
attestation: Attestation<T::EthSpec>,
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
) -> Result<AttestationProcessingOutcome, Error> {
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::inc_counter(&metrics::ATTESTATION_PROCESSING_REQUESTS);
let timer = metrics::start_timer(&metrics::ATTESTATION_PROCESSING_TIMES);
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
// From the store, load the attestation's "head block".
//
// An honest validator would have set this block to be the head of the chain (i.e., the
// result of running fork choice).
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
let result = if let Some(attestation_head_block) = self
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
.store
.get::<BeaconBlock<T::EthSpec>>(&attestation.data.beacon_block_root)?
{
// Attempt to process the attestation using the `self.head()` state.
//
// This is purely an effort to avoid loading a `BeaconState` unnecessarily from the DB.
// Take a read lock on the head beacon state.
let state = &self.head().beacon_state;
// If it turns out that the attestation was made using the head state, then there
// is no need to load a state from the database to process the attestation.
//
// Note: use the epoch of the target because it indicates which epoch the
// attestation was created in. You cannot use the epoch of the head block, because
// the block doesn't necessarily need to be in the same epoch as the attestation
// (e.g., if there are skip slots between the epoch the block was created in and
// the epoch for the attestation).
//
// This check also ensures that the slot for `data.beacon_block_root` is not higher
// than `state.root` by ensuring that the block is in the history of `state`.
if state.current_epoch() == attestation.data.target.epoch
&& (attestation.data.beacon_block_root == self.head().beacon_block_root
|| state
.get_block_root(attestation_head_block.slot)
.map(|root| *root == attestation.data.beacon_block_root)
.unwrap_or_else(|_| false))
{
// The head state is able to be used to validate this attestation. No need to load
// anything from the database.
return self.process_attestation_for_state_and_block(
attestation.clone(),
state,
&attestation_head_block,
);
}
// Ensure the read-lock from `self.head()` is dropped.
//
// This is likely unnecessary, however it remains as a reminder to ensure this lock
// isn't hogged.
std::mem::drop(state);
// Use the `data.beacon_block_root` to load the state from the latest non-skipped
// slot preceding the attestation's creation.
//
// This state is guaranteed to be in the same chain as the attestation, but it's
// not guaranteed to be from the same slot or epoch as the attestation.
let mut state: BeaconState<T::EthSpec> = self
.store
.get(&attestation_head_block.state_root)?
.ok_or_else(|| Error::MissingBeaconState(attestation_head_block.state_root))?;
// Ensure the state loaded from the database matches the state of the attestation
// head block.
//
// The state needs to be advanced from the current slot through to the epoch in
// which the attestation was created in. It would be an error to try and use
// `state.get_attestation_data_slot(..)` because the state matching the
// `data.beacon_block_root` isn't necessarily in a nearby epoch to the attestation
// (e.g., if there were lots of skip slots since the head of the chain and the
// epoch creation epoch).
for _ in state.slot.as_u64()
..attestation
.data
.target
.epoch
.start_slot(T::EthSpec::slots_per_epoch())
.as_u64()
{
per_slot_processing(&mut state, &self.spec)?;
}
state.build_committee_cache(RelativeEpoch::Current, &self.spec)?;
let attestation_slot = state.get_attestation_data_slot(&attestation.data)?;
// Reject any attestation where the `state` loaded from `data.beacon_block_root`
// has a higher slot than the attestation.
//
// Permitting this would allow for attesters to vote on _future_ slots.
if attestation_slot > state.slot {
Ok(AttestationProcessingOutcome::AttestsToFutureState {
state: state.slot,
attestation: attestation_slot,
})
} else {
self.process_attestation_for_state_and_block(
attestation,
&state,
&attestation_head_block,
)
}
} else {
// Drop any attestation where we have not processed `attestation.data.beacon_block_root`.
//
// This is likely overly restrictive, we could store the attestation for later
// processing.
warn!(
self.log,
"Dropped attestation for unknown block";
"block" => format!("{}", attestation.data.beacon_block_root)
);
Ok(AttestationProcessingOutcome::UnknownHeadBlock {
beacon_block_root: attestation.data.beacon_block_root,
})
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
};
metrics::stop_timer(timer);
if let Ok(AttestationProcessingOutcome::Processed) = &result {
metrics::inc_counter(&metrics::ATTESTATION_PROCESSING_SUCCESSES);
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
}
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
result
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
}
/// Verifies the `attestation` against the `state` to which it is attesting.
///
/// Updates fork choice with any new latest messages, but _does not_ find or update the head.
///
/// ## Notes
///
/// The given `state` must fulfil one of the following conditions:
///
/// - `state` corresponds to the `block.state_root` identified by
/// `attestation.data.beacon_block_root`. (Viz., `attestation` was created using `state`).
/// - `state.slot` is in the same epoch as `data.target.epoch` and
/// `attestation.data.beacon_block_root` is in the history of `state`.
///
/// Additionally, `attestation.data.beacon_block_root` **must** be available to read in
/// `self.store` _and_ be the root of the given `block`.
///
/// If the given conditions are not fulfilled, the function may error or provide a false
/// negative (indicating that a given `attestation` is invalid when it is was validly formed).
fn process_attestation_for_state_and_block(
&self,
attestation: Attestation<T::EthSpec>,
state: &BeaconState<T::EthSpec>,
block: &BeaconBlock<T::EthSpec>,
) -> Result<AttestationProcessingOutcome, Error> {
// Find the highest between:
//
// - The highest valid finalized epoch we've ever seen (i.e., the head).
// - The finalized epoch that this attestation was created against.
let finalized_epoch = std::cmp::max(
self.head().beacon_state.finalized_checkpoint.epoch,
state.finalized_checkpoint.epoch,
);
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
// A helper function to allow attestation processing to be metered.
let verify_attestation_for_state = |state, attestation, spec, verify_signatures| {
let timer = metrics::start_timer(&metrics::ATTESTATION_PROCESSING_CORE);
let result = verify_attestation_for_state(state, attestation, spec, verify_signatures);
metrics::stop_timer(timer);
result
};
if block.slot <= finalized_epoch.start_slot(T::EthSpec::slots_per_epoch()) {
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
// Ignore any attestation where the slot of `data.beacon_block_root` is equal to or
// prior to the finalized epoch.
//
// For any valid attestation if the `beacon_block_root` is prior to finalization, then
// all other parameters (source, target, etc) must all be prior to finalization and
// therefore no longer interesting.
Ok(AttestationProcessingOutcome::FinalizedSlot {
attestation: block.slot.epoch(T::EthSpec::slots_per_epoch()),
finalized: finalized_epoch,
})
} else if let Err(e) =
verify_attestation_for_state(state, &attestation, &self.spec, VerifySignatures::True)
{
warn!(
self.log,
"Invalid attestation";
"state_epoch" => state.current_epoch(),
"error" => format!("{:?}", e),
);
Ok(AttestationProcessingOutcome::Invalid(e))
} else {
// Provide the attestation to fork choice, updating the validator latest messages but
// _without_ finding and updating the head.
if let Err(e) = self
.fork_choice
.process_attestation(&state, &attestation, block)
{
error!(
self.log,
"Add attestation to fork choice failed";
"fork_choice_integrity" => format!("{:?}", self.fork_choice.verify_integrity()),
"beacon_block_root" => format!("{}", attestation.data.beacon_block_root),
"error" => format!("{:?}", e)
);
return Err(e.into());
}
2019-06-15 22:19:08 +00:00
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
// Provide the valid attestation to op pool, which may choose to retain the
// attestation for inclusion in a future block.
self.op_pool
.insert_attestation(attestation, state, &self.spec)?;
// Update the metrics.
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::inc_counter(&metrics::ATTESTATION_PROCESSING_SUCCESSES);
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
Ok(AttestationProcessingOutcome::Processed)
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
}
}
/// Accept some deposit and queue it for inclusion in an appropriate block.
pub fn process_deposit(
&self,
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
index: u64,
deposit: Deposit,
) -> Result<DepositInsertStatus, DepositValidationError> {
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
self.op_pool.insert_deposit(index, deposit)
}
/// Accept some exit and queue it for inclusion in an appropriate block.
pub fn process_voluntary_exit(&self, exit: VoluntaryExit) -> Result<(), ExitValidationError> {
self.op_pool
.insert_voluntary_exit(exit, &*self.state.read(), &self.spec)
}
2019-03-07 06:23:11 +00:00
/// Accept some transfer and queue it for inclusion in an appropriate block.
pub fn process_transfer(&self, transfer: Transfer) -> Result<(), TransferValidationError> {
self.op_pool
.insert_transfer(transfer, &*self.state.read(), &self.spec)
2019-03-07 06:23:11 +00:00
}
/// Accept some proposer slashing and queue it for inclusion in an appropriate block.
pub fn process_proposer_slashing(
&self,
proposer_slashing: ProposerSlashing,
) -> Result<(), ProposerSlashingValidationError> {
self.op_pool
.insert_proposer_slashing(proposer_slashing, &*self.state.read(), &self.spec)
}
/// Accept some attester slashing and queue it for inclusion in an appropriate block.
pub fn process_attester_slashing(
&self,
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
attester_slashing: AttesterSlashing<T::EthSpec>,
) -> Result<(), AttesterSlashingValidationError> {
self.op_pool
.insert_attester_slashing(attester_slashing, &*self.state.read(), &self.spec)
}
/// Accept some block and attempt to add it to block DAG.
///
/// Will accept blocks from prior slots, however it will reject any block from a future slot.
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
pub fn process_block(
&self,
block: BeaconBlock<T::EthSpec>,
) -> Result<BlockProcessingOutcome, Error> {
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::inc_counter(&metrics::BLOCK_PROCESSING_REQUESTS);
let full_timer = metrics::start_timer(&metrics::BLOCK_PROCESSING_TIMES);
let finalized_slot = self
.state
.read()
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
.finalized_checkpoint
.epoch
.start_slot(T::EthSpec::slots_per_epoch());
2019-06-20 00:59:19 +00:00
if block.slot <= finalized_slot {
return Ok(BlockProcessingOutcome::FinalizedSlot);
}
2019-06-07 06:46:07 +00:00
if block.slot == 0 {
return Ok(BlockProcessingOutcome::GenesisBlock);
2019-06-07 06:46:07 +00:00
}
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
let block_root_timer = metrics::start_timer(&metrics::BLOCK_PROCESSING_BLOCK_ROOT);
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
let block_root = block.canonical_root();
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::stop_timer(block_root_timer);
2019-06-07 06:46:07 +00:00
if block_root == self.genesis_block_root {
return Ok(BlockProcessingOutcome::GenesisBlock);
2019-06-07 06:46:07 +00:00
}
2019-06-20 00:59:19 +00:00
let present_slot = self
.read_slot_clock()
.ok_or_else(|| Error::UnableToReadSlot)?;
if block.slot > present_slot {
return Ok(BlockProcessingOutcome::FutureSlot {
present_slot,
block_slot: block.slot,
});
}
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
if self.store.exists::<BeaconBlock<T::EthSpec>>(&block_root)? {
return Ok(BlockProcessingOutcome::BlockIsAlreadyKnown);
}
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
// Records the time taken to load the block and state from the database during block
// processing.
let db_read_timer = metrics::start_timer(&metrics::BLOCK_PROCESSING_DB_READ);
// Load the blocks parent block from the database, returning invalid if that block is not
// found.
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
let parent_block: BeaconBlock<T::EthSpec> = match self.store.get(&block.parent_root)? {
Some(block) => block,
None => {
return Ok(BlockProcessingOutcome::ParentUnknown {
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
parent: block.parent_root,
});
}
};
// Load the parent blocks state from the database, returning an error if it is not found.
// It is an error because if know the parent block we should also know the parent state.
let parent_state_root = parent_block.state_root;
let parent_state = self
2019-05-21 07:27:06 +00:00
.store
.get(&parent_state_root)?
.ok_or_else(|| Error::DBInconsistent(format!("Missing state {}", parent_state_root)))?;
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::stop_timer(db_read_timer);
let catchup_timer = metrics::start_timer(&metrics::BLOCK_PROCESSING_CATCHUP_STATE);
Bootstrap (#501) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement * Add first attempts at HTTP bootstrap * Add beacon_block methods to rest api * Fix serde for block.body.grafitti * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Add network routes to API * Fix rustc warnings * Add best_slot method * Add --bootstrap arg to beacon node * Get bootstrapper working for ENR address * Store intermediate states during block processing * Allow bootstrapper to scrape libp2p address * Update bootstrapper libp2p address finding * Add comments * Tidy API to be more consistent with recent decisions * Address some review comments * Make BeaconChainTypes Send + Sync + 'static * Add `/network/listen_port` API endpoint * Abandon starting the node if libp2p doesn't start * Update bootstrapper for API changes * Remove unnecessary trait bounds
2019-08-23 05:53:53 +00:00
// Keep a list of any states that were "skipped" (block-less) in between the parent state
// slot and the block slot. These will need to be stored in the database.
let mut intermediate_states = vec![];
2019-03-23 07:48:09 +00:00
// Transition the parent state to the block slot.
let mut state: BeaconState<T::EthSpec> = parent_state;
Bootstrap (#501) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement * Add first attempts at HTTP bootstrap * Add beacon_block methods to rest api * Fix serde for block.body.grafitti * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Add network routes to API * Fix rustc warnings * Add best_slot method * Add --bootstrap arg to beacon node * Get bootstrapper working for ENR address * Store intermediate states during block processing * Allow bootstrapper to scrape libp2p address * Update bootstrapper libp2p address finding * Add comments * Tidy API to be more consistent with recent decisions * Address some review comments * Make BeaconChainTypes Send + Sync + 'static * Add `/network/listen_port` API endpoint * Abandon starting the node if libp2p doesn't start * Update bootstrapper for API changes * Remove unnecessary trait bounds
2019-08-23 05:53:53 +00:00
for i in state.slot.as_u64()..block.slot.as_u64() {
if i > 0 {
intermediate_states.push(state.clone());
}
per_slot_processing(&mut state, &self.spec)?;
}
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::stop_timer(catchup_timer);
let committee_timer = metrics::start_timer(&metrics::BLOCK_PROCESSING_COMMITTEE);
Attestation processing (#497) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement
2019-08-14 00:55:24 +00:00
state.build_committee_cache(RelativeEpoch::Previous, &self.spec)?;
state.build_committee_cache(RelativeEpoch::Current, &self.spec)?;
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::stop_timer(committee_timer);
let core_timer = metrics::start_timer(&metrics::BLOCK_PROCESSING_CORE);
// Apply the received block to its parent state (which has been transitioned into this
// slot).
match per_block_processing(&mut state, &block, &self.spec) {
Err(BlockProcessingError::BeaconStateError(e)) => {
return Err(Error::BeaconStateError(e))
}
Err(e) => return Ok(BlockProcessingOutcome::PerBlockProcessingError(e)),
_ => {}
}
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::stop_timer(core_timer);
let state_root_timer = metrics::start_timer(&metrics::BLOCK_PROCESSING_STATE_ROOT);
let state_root = state.canonical_root();
if block.state_root != state_root {
return Ok(BlockProcessingOutcome::StateRootMismatch);
}
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::stop_timer(state_root_timer);
let db_write_timer = metrics::start_timer(&metrics::BLOCK_PROCESSING_DB_WRITE);
Bootstrap (#501) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Fix minor PR comments * Remove duplicated attestation finalization check * Remove awkward `let` statement * Add first attempts at HTTP bootstrap * Add beacon_block methods to rest api * Fix serde for block.body.grafitti * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Add network routes to API * Fix rustc warnings * Add best_slot method * Add --bootstrap arg to beacon node * Get bootstrapper working for ENR address * Store intermediate states during block processing * Allow bootstrapper to scrape libp2p address * Update bootstrapper libp2p address finding * Add comments * Tidy API to be more consistent with recent decisions * Address some review comments * Make BeaconChainTypes Send + Sync + 'static * Add `/network/listen_port` API endpoint * Abandon starting the node if libp2p doesn't start * Update bootstrapper for API changes * Remove unnecessary trait bounds
2019-08-23 05:53:53 +00:00
// Store all the states between the parent block state and this blocks slot before storing
// the final state.
for (i, intermediate_state) in intermediate_states.iter().enumerate() {
// To avoid doing an unnecessary tree hash, use the following (slot + 1) state's
// state_roots field to find the root.
let following_state = match intermediate_states.get(i + 1) {
Some(following_state) => following_state,
None => &state,
};
let intermediate_state_root =
following_state.get_state_root(intermediate_state.slot)?;
self.store
.put(&intermediate_state_root, intermediate_state)?;
}
// Store the block and state.
2019-05-21 07:27:06 +00:00
self.store.put(&block_root, &block)?;
self.store.put(&state_root, &state)?;
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::stop_timer(db_write_timer);
let fork_choice_register_timer =
metrics::start_timer(&metrics::BLOCK_PROCESSING_FORK_CHOICE_REGISTER);
// Register the new block with the fork choice service.
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
if let Err(e) = self.fork_choice.process_block(&state, &block, block_root) {
error!(
self.log,
"Add block to fork choice failed";
"fork_choice_integrity" => format!("{:?}", self.fork_choice.verify_integrity()),
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
"block_root" => format!("{}", block_root),
"error" => format!("{:?}", e),
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
)
}
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::stop_timer(fork_choice_register_timer);
let find_head_timer =
metrics::start_timer(&metrics::BLOCK_PROCESSING_FORK_CHOICE_FIND_HEAD);
// Execute the fork choice algorithm, enthroning a new head if discovered.
//
// Note: in the future we may choose to run fork-choice less often, potentially based upon
// some heuristic around number of attestations seen for the block.
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
if let Err(e) = self.fork_choice() {
error!(
self.log,
"fork choice failed to find head";
"error" => format!("{:?}", e)
)
};
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::stop_timer(find_head_timer);
metrics::inc_counter(&metrics::BLOCK_PROCESSING_SUCCESSES);
metrics::observe(
&metrics::OPERATIONS_PER_BLOCK_ATTESTATION,
block.body.attestations.len() as f64,
);
metrics::stop_timer(full_timer);
2019-06-20 08:46:03 +00:00
Ok(BlockProcessingOutcome::Processed { block_root })
}
/// Produce a new block at the present slot.
///
/// The produced block will not be inherently valid, it must be signed by a block producer.
/// Block signing is out of the scope of this function and should be done by a separate program.
pub fn produce_block(
&self,
randao_reveal: Signature,
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
) -> Result<(BeaconBlock<T::EthSpec>, BeaconState<T::EthSpec>), BlockProductionError> {
let state = self.state.read().clone();
let slot = self
.read_slot_clock()
.ok_or_else(|| BlockProductionError::UnableToReadSlot)?;
self.produce_block_on_state(state, slot, randao_reveal)
}
/// Produce a block for some `slot` upon the given `state`.
///
/// Typically the `self.produce_block()` function should be used, instead of calling this
/// function directly. This function is useful for purposefully creating forks or blocks at
/// non-current slots.
///
/// The given state will be advanced to the given `produce_at_slot`, then a block will be
/// produced at that slot height.
pub fn produce_block_on_state(
&self,
mut state: BeaconState<T::EthSpec>,
produce_at_slot: Slot,
randao_reveal: Signature,
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
) -> Result<(BeaconBlock<T::EthSpec>, BeaconState<T::EthSpec>), BlockProductionError> {
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::inc_counter(&metrics::BLOCK_PRODUCTION_REQUESTS);
let timer = metrics::start_timer(&metrics::BLOCK_PRODUCTION_TIMES);
// If required, transition the new state to the present slot.
2019-06-20 08:46:03 +00:00
while state.slot < produce_at_slot {
per_slot_processing(&mut state, &self.spec)?;
}
2019-03-21 20:11:04 +00:00
state.build_committee_cache(RelativeEpoch::Current, &self.spec)?;
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
let parent_root = if state.slot > 0 {
2019-06-07 06:46:07 +00:00
*state
.get_block_root(state.slot - 1)
.map_err(|_| BlockProductionError::UnableToGetBlockRootFromState)?
} else {
state.latest_block_header.canonical_root()
};
let mut graffiti: [u8; 32] = [0; 32];
graffiti.copy_from_slice(GRAFFITI.as_bytes());
let (proposer_slashings, attester_slashings) =
self.op_pool.get_slashings(&state, &self.spec);
let mut block = BeaconBlock {
slot: state.slot,
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
parent_root,
state_root: Hash256::zero(), // Updated after the state is calculated.
2019-06-03 23:37:40 +00:00
signature: Signature::empty_signature(), // To be completed by a validator.
body: BeaconBlockBody {
randao_reveal,
2019-06-20 00:59:19 +00:00
// TODO: replace with real data.
eth1_data: Eth1Data {
deposit_count: state.eth1_data.deposit_count,
deposit_root: Hash256::zero(),
block_hash: Hash256::zero(),
},
graffiti,
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
proposer_slashings: proposer_slashings.into(),
attester_slashings: attester_slashings.into(),
attestations: self.op_pool.get_attestations(&state, &self.spec).into(),
deposits: self.op_pool.get_deposits(&state).into(),
voluntary_exits: self.op_pool.get_voluntary_exits(&state, &self.spec).into(),
transfers: self.op_pool.get_transfers(&state, &self.spec).into(),
},
};
per_block_processing_without_verifying_block_signature(&mut state, &block, &self.spec)?;
let state_root = state.canonical_root();
block.state_root = state_root;
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::inc_counter(&metrics::BLOCK_PRODUCTION_SUCCESSES);
metrics::stop_timer(timer);
Ok((block, state))
}
/// Execute the fork choice algorithm and enthrone the result as the canonical head.
pub fn fork_choice(&self) -> Result<(), Error> {
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::inc_counter(&metrics::FORK_CHOICE_REQUESTS);
2019-05-31 07:51:32 +00:00
// Start fork choice metrics timer.
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
let timer = metrics::start_timer(&metrics::FORK_CHOICE_TIMES);
2019-05-31 07:51:32 +00:00
// Determine the root of the block that is the head of the chain.
2019-06-15 19:05:34 +00:00
let beacon_block_root = self.fork_choice.find_head(&self)?;
2019-05-31 07:51:32 +00:00
// If a new head was chosen.
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
let result = if beacon_block_root != self.head().beacon_block_root {
metrics::inc_counter(&metrics::FORK_CHOICE_CHANGED_HEAD);
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
let beacon_block: BeaconBlock<T::EthSpec> = self
2019-05-21 07:27:06 +00:00
.store
2019-05-31 07:51:32 +00:00
.get(&beacon_block_root)?
.ok_or_else(|| Error::MissingBeaconBlock(beacon_block_root))?;
let beacon_state_root = beacon_block.state_root;
let beacon_state: BeaconState<T::EthSpec> = self
2019-05-21 07:27:06 +00:00
.store
2019-05-31 07:51:32 +00:00
.get(&beacon_state_root)?
.ok_or_else(|| Error::MissingBeaconState(beacon_state_root))?;
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
let previous_slot = self.head().beacon_block.slot;
let new_slot = beacon_block.slot;
2019-05-31 07:51:32 +00:00
// If we switched to a new chain (instead of building atop the present chain).
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
if self.head().beacon_block_root != beacon_block.parent_root {
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::inc_counter(&metrics::FORK_CHOICE_REORG_COUNT);
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
warn!(
self.log,
"Beacon chain re-org";
"previous_slot" => previous_slot,
"new_slot" => new_slot
);
} else {
info!(
self.log,
"new head block";
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
"justified_root" => format!("{}", beacon_state.current_justified_checkpoint.root),
"finalized_root" => format!("{}", beacon_state.finalized_checkpoint.root),
Testnet stability (#451) * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Change reduced tree for adding weightless node * Add more comments for reduced tree fork choice * Small refactor on reduced tree for readability * Move test_harness forking logic into itself * Add new `AncestorIter` trait to store * Add unfinished tests to fork choice * Make `beacon_state.genesis_block_root` public * Add failing lmd_ghost fork choice tests * Extend fork_choice tests, create failing test * Implement Debug for generic ReducedTree * Add lazy_static to fork choice tests * Add verify_integrity fn to reduced tree * Fix bugs in reduced tree * Ensure all reduced tree tests verify integrity * Slightly alter reduce tree test params * Add (failing) reduced tree test * Fix bug in fork choice Iter ancestors was not working well with skip slots * Put maximum depth for common ancestor search Ensures that we don't search back past the finalized root. * Add basic finalization tests for reduced tree * Add network dir CLI flag * Simplify "NewSlot" log message * Rename network-dir CLI flag * Change fork choice to use beacon_block_root Previously it was using target_root, which was wrong * Update db dir size for metrics * Change slog to use `FullFormat` logging * Update some comments and log formatting * Add prom gauge for best block root * Only add known target blocks to fork choice * Add finalized and justified root prom metrics * Add CLI flag for setting log level * Add logger to beacon chain * Add debug-level CLI flag to validator * Allow block processing if fork choice fails * Create warn log when there's low libp2p peer count * Minor change to logging * Make ancestor iter return option * Disable fork choice test when !debug_assertions * Fix type, removed code fragment * Tidy some borrow-checker evading * Lower reduced tree random test iterations
2019-07-29 03:45:45 +00:00
"root" => format!("{}", beacon_block_root),
"slot" => new_slot,
);
};
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
let old_finalized_epoch = self.head().beacon_state.finalized_checkpoint.epoch;
let new_finalized_epoch = beacon_state.finalized_checkpoint.epoch;
let finalized_root = beacon_state.finalized_checkpoint.root;
// Never revert back past a finalized epoch.
if new_finalized_epoch < old_finalized_epoch {
Err(Error::RevertedFinalizedEpoch {
previous_epoch: old_finalized_epoch,
new_epoch: new_finalized_epoch,
})
} else {
self.update_canonical_head(CheckPoint {
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
beacon_block,
beacon_block_root,
beacon_state,
beacon_state_root,
})?;
if new_finalized_epoch != old_finalized_epoch {
self.after_finalization(old_finalized_epoch, finalized_root)?;
}
Ok(())
}
} else {
Ok(())
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
};
// End fork choice metrics timer.
metrics::stop_timer(timer);
if let Err(_) = result {
metrics::inc_counter(&metrics::FORK_CHOICE_ERRORS);
}
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
result
}
/// Update the canonical head to `new_head`.
fn update_canonical_head(&self, new_head: CheckPoint<T::EthSpec>) -> Result<(), Error> {
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
let timer = metrics::start_timer(&metrics::UPDATE_HEAD_TIMES);
// Update the checkpoint that stores the head of the chain at the time it received the
// block.
*self.canonical_head.write() = new_head;
// Update the always-at-the-present-slot state we keep around for performance gains.
*self.state.write() = {
let mut state = self.canonical_head.read().beacon_state.clone();
let present_slot = match self.slot_clock.present_slot() {
Ok(Some(slot)) => slot,
_ => return Err(Error::UnableToReadSlot),
};
// If required, transition the new state to the present slot.
for _ in state.slot.as_u64()..present_slot.as_u64() {
per_slot_processing(&mut state, &self.spec)?;
}
state.build_all_caches(&self.spec)?;
state
};
// Save `self` to `self.store`.
self.persist()?;
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
metrics::stop_timer(timer);
Ok(())
}
/// Called after `self` has had a new block finalized.
///
/// Performs pruning and finality-based optimizations.
fn after_finalization(
&self,
old_finalized_epoch: Epoch,
finalized_block_root: Hash256,
) -> Result<(), Error> {
let finalized_block = self
.store
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
.get::<BeaconBlock<T::EthSpec>>(&finalized_block_root)?
.ok_or_else(|| Error::MissingBeaconBlock(finalized_block_root))?;
let new_finalized_epoch = finalized_block.slot.epoch(T::EthSpec::slots_per_epoch());
if new_finalized_epoch < old_finalized_epoch {
Err(Error::RevertedFinalizedEpoch {
previous_epoch: old_finalized_epoch,
new_epoch: new_finalized_epoch,
})
} else {
self.fork_choice
.process_finalization(&finalized_block, finalized_block_root)?;
Fix block processing blowup, upgrade metrics (#500) * Renamed fork_choice::process_attestation_from_block * Processing attestation in fork choice * Retrieving state from store and checking signature * Looser check on beacon state validity. * Cleaned up get_attestation_state * Expanded fork choice api to provide latest validator message. * Checking if the an attestation contains a latest message * Correct process_attestation error handling. * Copy paste error in comment fixed. * Tidy ancestor iterators * Getting attestation slot via helper method * Refactored attestation creation in test utils * Revert "Refactored attestation creation in test utils" This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306. * Integration tests for free attestation processing * Implicit conflicts resolved. * formatting * Do first pass on Grants code * Add another attestation processing test * Tidy attestation processing * Remove old code fragment * Add non-compiling half finished changes * Simplify, fix bugs, add tests for chain iters * Remove attestation processing from op pool * Fix bug with fork choice, tidy * Fix overly restrictive check in fork choice. * Ensure committee cache is build during attn proc * Ignore unknown blocks at fork choice * Various minor fixes * Make fork choice write lock in to read lock * Remove unused method * Tidy comments * Fix attestation prod. target roots change * Fix compile error in store iters * Reject any attestation prior to finalization * Begin metrics refactor * Move beacon_chain to new metrics structure. * Make metrics not panic if already defined * Use global prometheus gather at rest api * Unify common metric fns into a crate * Add heavy metering to block processing * Remove hypen from prometheus metric name * Add more beacon chain metrics * Add beacon chain persistence metric * Prune op pool on finalization * Add extra prom beacon chain metrics * Prefix BeaconChain metrics with "beacon_" * Add more store metrics * Add basic metrics to libp2p * Add metrics to HTTP server * Remove old `http_server` crate * Update metrics names to be more like standard * Fix broken beacon chain metrics, add slot clock metrics * Add lighthouse_metrics gather fn * Remove http args * Fix wrong state given to op pool prune * Make prom metric names more consistent * Add more metrics, tidy existing metrics * Fix store block read metrics * Tidy attestation metrics * Fix minor PR comments * Allow travis failures on beta (see desc) There's a non-backward compatible change in `cargo fmt`. Stable and beta do not agree. * Tidy `lighthouse_metrics` docs * Fix typo
2019-08-19 11:02:34 +00:00
let finalized_state = self
.store
.get::<BeaconState<T::EthSpec>>(&finalized_block.state_root)?
.ok_or_else(|| Error::MissingBeaconState(finalized_block.state_root))?;
self.op_pool.prune_all(&finalized_state, &self.spec);
Ok(())
}
}
/// Returns `true` if the given block root has not been processed.
pub fn is_new_block_root(&self, beacon_block_root: &Hash256) -> Result<bool, Error> {
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
Ok(!self
.store
.exists::<BeaconBlock<T::EthSpec>>(beacon_block_root)?)
}
/// Dumps the entire canonical chain, from the head to genesis to a vector for analysis.
///
/// This could be a very expensive operation and should only be done in testing/analysis
/// activities.
pub fn chain_dump(&self) -> Result<Vec<CheckPoint<T::EthSpec>>, Error> {
let mut dump = vec![];
let mut last_slot = CheckPoint {
beacon_block: self.head().beacon_block.clone(),
beacon_block_root: self.head().beacon_block_root,
beacon_state: self.head().beacon_state.clone(),
beacon_state_root: self.head().beacon_state_root,
};
dump.push(last_slot.clone());
loop {
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
let beacon_block_root = last_slot.beacon_block.parent_root;
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
if beacon_block_root == Hash256::zero() {
break; // Genesis has been reached.
}
Update to frozen spec ❄️ (v0.8.1) (#444) * types: first updates for v0.8 * state_processing: epoch processing v0.8.0 * state_processing: block processing v0.8.0 * tree_hash_derive: support generics in SignedRoot * types v0.8: update to use ssz_types * state_processing v0.8: use ssz_types * ssz_types: add bitwise methods and from_elem * types: fix v0.8 FIXMEs * ssz_types: add bitfield shift_up * ssz_types: iterators and DerefMut for VariableList * types,state_processing: use VariableList * ssz_types: fix BitVector Decode impl Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it to be considered variable length! * types: fix test modules for v0.8 update * types: remove slow type-level arithmetic * state_processing: fix tests for v0.8 * op_pool: update for v0.8 * ssz_types: Bitfield difference length-independent Allow computing the difference of two bitfields of different lengths. * Implement compact committee support * epoch_processing: committee & active index roots * state_processing: genesis state builder v0.8 * state_processing: implement v0.8.1 * Further improve tree_hash * Strip examples, tests from cached_tree_hash * Update TreeHash, un-impl CachedTreeHash * Update bitfield TreeHash, un-impl CachedTreeHash * Update FixedLenVec TreeHash, unimpl CachedTreeHash * Update update tree_hash_derive for new TreeHash * Fix TreeHash, un-impl CachedTreeHash for ssz_types * Remove fixed_len_vec, ssz benches SSZ benches relied upon fixed_len_vec -- it is easier to just delete them and rebuild them later (when necessary) * Remove boolean_bitfield crate * Fix fake_crypto BLS compile errors * Update ef_tests for new v.8 type params * Update ef_tests submodule to v0.8.1 tag * Make fixes to support parsing ssz ef_tests * `compact_committee...` to `compact_committees...` * Derive more traits for `CompactCommittee` * Flip bitfield byte-endianness * Fix tree_hash for bitfields * Modify CLI output for ef_tests * Bump ssz crate version * Update ssz_types doc comment * Del cached tree hash tests from ssz_static tests * Tidy SSZ dependencies * Rename ssz_types crate to eth2_ssz_types * validator_client: update for v0.8 * ssz_types: update union/difference for bit order swap * beacon_node: update for v0.8, EthSpec * types: disable cached tree hash, update min spec * state_processing: fix slot bug in committee update * tests: temporarily disable fork choice harness test See #447 * committee cache: prevent out-of-bounds access In the case where we tried to access the committee of a shard that didn't have a committee in the current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This commit adds a check to make the failure safe and explicit. * fix bug in get_indexed_attestation and simplify There was a bug in our implementation of get_indexed_attestation whereby incorrect "committee indices" were used to index into the custody bitfield. The bug was only observable in the case where some bits of the custody bitfield were set to 1. The implementation has been simplified to remove the bug, and a test added. * state_proc: workaround for compact committees bug https://github.com/ethereum/eth2.0-specs/issues/1315 * v0.8: updates to make the EF tests pass * Remove redundant max operation checks. * Always supply both messages when checking attestation signatures -- allowing verification of an attestation with no signatures. * Swap the order of the fork and domain constant in `get_domain`, to match the spec. * rustfmt * ef_tests: add new epoch processing tests * Integrate v0.8 into master (compiles) * Remove unused crates, fix clippy lints * Replace v0.6.3 tags w/ v0.8.1 * Remove old comment * Ensure lmd ghost tests only run in release * Update readme
2019-07-30 02:44:51 +00:00
let beacon_block: BeaconBlock<T::EthSpec> =
2019-05-21 07:27:06 +00:00
self.store.get(&beacon_block_root)?.ok_or_else(|| {
Error::DBInconsistent(format!("Missing block {}", beacon_block_root))
})?;
let beacon_state_root = beacon_block.state_root;
2019-05-21 07:27:06 +00:00
let beacon_state = self.store.get(&beacon_state_root)?.ok_or_else(|| {
Error::DBInconsistent(format!("Missing state {}", beacon_state_root))
})?;
let slot = CheckPoint {
beacon_block,
beacon_block_root,
beacon_state,
beacon_state_root,
};
dump.push(slot.clone());
last_slot = slot;
}
dump.reverse();
Ok(dump)
}
}
impl From<DBError> for Error {
fn from(e: DBError) -> Error {
2019-05-21 07:27:06 +00:00
Error::DBError(e)
}
}
impl From<ForkChoiceError> for Error {
fn from(e: ForkChoiceError) -> Error {
Error::ForkChoiceError(e)
}
}
impl From<BeaconStateError> for Error {
fn from(e: BeaconStateError) -> Error {
Error::BeaconStateError(e)
}
}