cerc-io/lighthouse
16
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Refactor op pool for speed and correctness (#3312)

Browse Source 
## Proposed Changes

This PR has two aims: to speed up attestation packing in the op pool, and to fix bugs in the verification of attester slashings, proposer slashings and voluntary exits. The changes are bundled into a single database schema upgrade (v12).

Attestation packing is sped up by removing several inefficiencies: 

- No more recalculation of `attesting_indices` during packing.
- No (unnecessary) examination of the `ParticipationFlags`: a bitfield suffices. See `RewardCache`.
- No re-checking of attestation validity during packing: the `AttestationMap` provides attestations which are "correct by construction" (I have checked this using Hydra).
- No SSZ re-serialization for the clunky `AttestationId` type (it can be removed in a future release).

So far the speed-up seems to be roughly 2-10x, from 500ms down to 50-100ms.

Verification of attester slashings, proposer slashings and voluntary exits is fixed by:

- Tracking the `ForkVersion`s that were used to verify each message inside the `SigVerifiedOp`. This allows us to quickly re-verify that they match the head state's opinion of what the `ForkVersion` should be at the epoch(s) relevant to the message.
- Storing the `SigVerifiedOp` on disk rather than the raw operation. This allows us to continue track the fork versions after a reboot.

This is mostly contained in this commit 52bb1840ae5c4356a8fc3a51e5df23ed65ed2c7f.

## Additional Info

The schema upgrade uses the justified state to re-verify attestations and compute `attesting_indices` for them. It will drop any attestations that fail to verify, by the logic that attestations are most valuable in the few slots after they're observed, and are probably stale and useless by the time a node restarts. Exits and proposer slashings and similarly re-verified to obtain `SigVerifiedOp`s.

This PR contains a runtime killswitch `--paranoid-block-proposal` which opts out of all the optimisations in favour of closely verifying every included message. Although I'm quite sure that the optimisations are correct this flag could be useful in the event of an unforeseen emergency.

Finally, you might notice that the `RewardCache` appears quite useless in its current form because it is only updated on the hot-path immediately before proposal. My hope is that in future we can shift calls to `RewardCache::update` into the background, e.g. while performing the state advance. It is also forward-looking to `tree-states` compatibility, where iterating and indexing `state.{previous,current}_epoch_participation` is expensive and needs to be minimised.
This commit is contained in:
Michael Sproul 2022-08-29 09:10:26 +00:00
parent 1c9ec42dcb
commit 66eca1a882
37 changed files with 1710 additions and 521 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
Cargo.lock generated
View File

@ -4410,12 +4410,14 @@ name = "operation_pool"
version = "0.2.0"
dependencies = [
"beacon_chain",
"bitvec 1.0.1",
"derivative",
"eth2_ssz",
"eth2_ssz_derive",
"itertools",
"lazy_static",
"lighthouse_metrics",
"maplit",
"parking_lot 0.12.1",
"rayon",
"serde",
@ -6271,9 +6273,11 @@ dependencies = [
"arbitrary",
"beacon_chain",
"bls",
"derivative",
"env_logger 0.9.0",
"eth2_hashing",
"eth2_ssz",
"eth2_ssz_derive",
"eth2_ssz_types",
"int_to_bytes",
"integer-sqrt",

9
beacon_node/beacon_chain/src/attestation_verification.rs
View File

@ -318,10 +318,17 @@ impl<'a, T: BeaconChainTypes> Clone for IndexedUnaggregatedAttestation<'a, T> {
/// A helper trait implemented on wrapper types that can be progressed to a state where they can be
/// verified for application to fork choice.
pub trait VerifiedAttestation<T: BeaconChainTypes> {
pub trait VerifiedAttestation<T: BeaconChainTypes>: Sized {
fn attestation(&self) -> &Attestation<T::EthSpec>;
fn indexed_attestation(&self) -> &IndexedAttestation<T::EthSpec>;
// Inefficient default implementation. This is overridden for gossip verified attestations.
fn into_attestation_and_indices(self) -> (Attestation<T::EthSpec>, Vec<u64>) {
let attestation = self.attestation().clone();
let attesting_indices = self.indexed_attestation().attesting_indices.clone().into();
(attestation, attesting_indices)
}
}
impl<'a, T: BeaconChainTypes> VerifiedAttestation<T> for VerifiedAggregatedAttestation<'a, T> {

153
beacon_node/beacon_chain/src/beacon_chain.rs
View File

@ -63,7 +63,7 @@ use fork_choice::{
use futures::channel::mpsc::Sender;
use itertools::process_results;
use itertools::Itertools;
use operation_pool::{OperationPool, PersistedOperationPool};
use operation_pool::{AttestationRef, OperationPool, PersistedOperationPool};
use parking_lot::{Mutex, RwLock};
use safe_arith::SafeArith;
use slasher::Slasher;
@ -71,12 +71,15 @@ use slog::{crit, debug, error, info, trace, warn, Logger};
use slot_clock::SlotClock;
use ssz::Encode;
use state_processing::{
common::get_indexed_attestation,
common::{get_attesting_indices_from_state, get_indexed_attestation},
per_block_processing,
per_block_processing::errors::AttestationValidationError,
per_block_processing::{
errors::AttestationValidationError, verify_attestation_for_block_inclusion,
VerifySignatures,
},
per_slot_processing,
state_advance::{complete_state_advance, partial_state_advance},
BlockSignatureStrategy, SigVerifiedOp, VerifyBlockRoot,
BlockSignatureStrategy, SigVerifiedOp, VerifyBlockRoot, VerifyOperation,
};
use std::cmp::Ordering;
use std::collections::HashMap;
@ -1904,25 +1907,22 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
/// Accepts a `VerifiedAttestation` and attempts to apply it to `self.op_pool`.
///
/// The op pool is used by local block producers to pack blocks with operations.
pub fn add_to_block_inclusion_pool(
pub fn add_to_block_inclusion_pool<A>(
&self,
verified_attestation: &impl VerifiedAttestation<T>,
) -> Result<(), AttestationError> {
verified_attestation: A,
) -> Result<(), AttestationError>
where
A: VerifiedAttestation<T>,
{
let _timer = metrics::start_timer(&metrics::ATTESTATION_PROCESSING_APPLY_TO_OP_POOL);
// If there's no eth1 chain then it's impossible to produce blocks and therefore
// useless to put things in the op pool.
if self.eth1_chain.is_some() {
let fork = self.canonical_head.cached_head().head_fork();
let (attestation, attesting_indices) =
verified_attestation.into_attestation_and_indices();
self.op_pool
.insert_attestation(
// TODO: address this clone.
verified_attestation.attestation().clone(),
&fork,
self.genesis_validators_root,
&self.spec,
)
.insert_attestation(attestation, attesting_indices)
.map_err(Error::from)?;
}
@ -1955,15 +1955,15 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
pub fn filter_op_pool_attestation(
&self,
filter_cache: &mut HashMap<(Hash256, Epoch), bool>,
att: &Attestation<T::EthSpec>,
att: &AttestationRef<T::EthSpec>,
state: &BeaconState<T::EthSpec>,
) -> bool {
*filter_cache
.entry((att.data.beacon_block_root, att.data.target.epoch))
.entry((att.data.beacon_block_root, att.checkpoint.target_epoch))
.or_insert_with(|| {
self.shuffling_is_compatible(
&att.data.beacon_block_root,
att.data.target.epoch,
att.checkpoint.target_epoch,
state,
)
})
@ -2045,7 +2045,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
pub fn verify_voluntary_exit_for_gossip(
&self,
exit: SignedVoluntaryExit,
) -> Result<ObservationOutcome<SignedVoluntaryExit>, Error> {
) -> Result<ObservationOutcome<SignedVoluntaryExit, T::EthSpec>, Error> {
// NOTE: this could be more efficient if it avoided cloning the head state
let wall_clock_state = self.wall_clock_state()?;
Ok(self
@ -2066,7 +2066,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
}
/// Accept a pre-verified exit and queue it for inclusion in an appropriate block.
pub fn import_voluntary_exit(&self, exit: SigVerifiedOp<SignedVoluntaryExit>) {
pub fn import_voluntary_exit(&self, exit: SigVerifiedOp<SignedVoluntaryExit, T::EthSpec>) {
if self.eth1_chain.is_some() {
self.op_pool.insert_voluntary_exit(exit)
}
@ -2076,7 +2076,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
pub fn verify_proposer_slashing_for_gossip(
&self,
proposer_slashing: ProposerSlashing,
) -> Result<ObservationOutcome<ProposerSlashing>, Error> {
) -> Result<ObservationOutcome<ProposerSlashing, T::EthSpec>, Error> {
let wall_clock_state = self.wall_clock_state()?;
Ok(self.observed_proposer_slashings.lock().verify_and_observe(
proposer_slashing,
@ -2086,7 +2086,10 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
}
/// Accept some proposer slashing and queue it for inclusion in an appropriate block.
pub fn import_proposer_slashing(&self, proposer_slashing: SigVerifiedOp<ProposerSlashing>) {
pub fn import_proposer_slashing(
&self,
proposer_slashing: SigVerifiedOp<ProposerSlashing, T::EthSpec>,
) {
if self.eth1_chain.is_some() {
self.op_pool.insert_proposer_slashing(proposer_slashing)
}
@ -2096,7 +2099,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
pub fn verify_attester_slashing_for_gossip(
&self,
attester_slashing: AttesterSlashing<T::EthSpec>,
) -> Result<ObservationOutcome<AttesterSlashing<T::EthSpec>>, Error> {
) -> Result<ObservationOutcome<AttesterSlashing<T::EthSpec>, T::EthSpec>, Error> {
let wall_clock_state = self.wall_clock_state()?;
Ok(self.observed_attester_slashings.lock().verify_and_observe(
attester_slashing,
@ -2111,7 +2114,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
/// 2. Add it to the op pool.
pub fn import_attester_slashing(
&self,
attester_slashing: SigVerifiedOp<AttesterSlashing<T::EthSpec>>,
attester_slashing: SigVerifiedOp<AttesterSlashing<T::EthSpec>, T::EthSpec>,
) {
// Add to fork choice.
self.canonical_head
@ -2120,10 +2123,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
// Add to the op pool (if we have the ability to propose blocks).
if self.eth1_chain.is_some() {
self.op_pool.insert_attester_slashing(
attester_slashing,
self.canonical_head.cached_head().head_fork(),
)
self.op_pool.insert_attester_slashing(attester_slashing)
}
}
@ -3351,7 +3351,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
}
};
let (proposer_slashings, attester_slashings, voluntary_exits) =
let (mut proposer_slashings, mut attester_slashings, mut voluntary_exits) =
self.op_pool.get_slashings_and_exits(&state, &self.spec);
let eth1_data = eth1_chain.eth1_data_for_block_production(&state, &self.spec)?;
@ -3362,12 +3362,12 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
let unagg_import_timer =
metrics::start_timer(&metrics::BLOCK_PRODUCTION_UNAGGREGATED_TIMES);
for attestation in self.naive_aggregation_pool.read().iter() {
if let Err(e) = self.op_pool.insert_attestation(
attestation.clone(),
&state.fork(),
state.genesis_validators_root(),
&self.spec,
) {
let import = |attestation: &Attestation<T::EthSpec>| {
let attesting_indices = get_attesting_indices_from_state(&state, attestation)?;
self.op_pool
.insert_attestation(attestation.clone(), attesting_indices)
};
if let Err(e) = import(attestation) {
// Don't stop block production if there's an error, just create a log.
error!(
self.log,
@ -3388,15 +3388,15 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
metrics::start_timer(&metrics::BLOCK_PRODUCTION_ATTESTATION_TIMES);
let mut prev_filter_cache = HashMap::new();
let prev_attestation_filter = |att: &&Attestation<T::EthSpec>| {
self.filter_op_pool_attestation(&mut prev_filter_cache, *att, &state)
let prev_attestation_filter = |att: &AttestationRef<T::EthSpec>| {
self.filter_op_pool_attestation(&mut prev_filter_cache, att, &state)
};
let mut curr_filter_cache = HashMap::new();
let curr_attestation_filter = |att: &&Attestation<T::EthSpec>| {
self.filter_op_pool_attestation(&mut curr_filter_cache, *att, &state)
let curr_attestation_filter = |att: &AttestationRef<T::EthSpec>| {
self.filter_op_pool_attestation(&mut curr_filter_cache, att, &state)
};
let attestations = self
let mut attestations = self
.op_pool
.get_attestations(
&state,
@ -3407,6 +3407,77 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
.map_err(BlockProductionError::OpPoolError)?;
drop(attestation_packing_timer);
// If paranoid mode is enabled re-check the signatures of every included message.
// This will be a lot slower but guards against bugs in block production and can be
// quickly rolled out without a release.
if self.config.paranoid_block_proposal {
attestations.retain(|att| {
verify_attestation_for_block_inclusion(
&state,
att,
VerifySignatures::True,
&self.spec,
)
.map_err(|e| {
warn!(
self.log,
"Attempted to include an invalid attestation";
"err" => ?e,
"block_slot" => state.slot(),
"attestation" => ?att
);
})
.is_ok()
});
proposer_slashings.retain(|slashing| {
slashing
.clone()
.validate(&state, &self.spec)
.map_err(|e| {
warn!(
self.log,
"Attempted to include an invalid proposer slashing";
"err" => ?e,
"block_slot" => state.slot(),
"slashing" => ?slashing
);
})
.is_ok()
});
attester_slashings.retain(|slashing| {
slashing
.clone()
.validate(&state, &self.spec)
.map_err(|e| {
warn!(
self.log,
"Attempted to include an invalid attester slashing";
"err" => ?e,
"block_slot" => state.slot(),
"slashing" => ?slashing
);
})
.is_ok()
});
voluntary_exits.retain(|exit| {
exit.clone()
.validate(&state, &self.spec)
.map_err(|e| {
warn!(
self.log,
"Attempted to include an invalid proposer slashing";
"err" => ?e,
"block_slot" => state.slot(),
"exit" => ?exit
);
})
.is_ok()
});
}
let slot = state.slot();
let proposer_index = state.get_beacon_proposer_index(state.slot(), &self.spec)? as u64;

33
beacon_node/beacon_chain/src/block_reward.rs
View File

@ -1,7 +1,10 @@
use crate::{BeaconChain, BeaconChainError, BeaconChainTypes};
use eth2::lighthouse::{AttestationRewards, BlockReward, BlockRewardMeta};
use operation_pool::{AttMaxCover, MaxCover};
use state_processing::per_block_processing::altair::sync_committee::compute_sync_aggregate_rewards;
use operation_pool::{AttMaxCover, MaxCover, RewardCache, SplitAttestation};
use state_processing::{
common::get_attesting_indices_from_state,
per_block_processing::altair::sync_committee::compute_sync_aggregate_rewards,
};
use types::{BeaconBlockRef, BeaconState, EthSpec, ExecPayload, Hash256};
impl<T: BeaconChainTypes> BeaconChain<T> {
@ -10,20 +13,38 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
block: BeaconBlockRef<'_, T::EthSpec, Payload>,
block_root: Hash256,
state: &BeaconState<T::EthSpec>,
reward_cache: &mut RewardCache,
include_attestations: bool,
) -> Result<BlockReward, BeaconChainError> {
if block.slot() != state.slot() {
return Err(BeaconChainError::BlockRewardSlotError);
}
reward_cache.update(state)?;
let total_active_balance = state.get_total_active_balance()?;
let mut per_attestation_rewards = block
let split_attestations = block
.body()
.attestations()
.iter()
.map(|att| {
AttMaxCover::new(att, state, total_active_balance, &self.spec)
.ok_or(BeaconChainError::BlockRewardAttestationError)
let attesting_indices = get_attesting_indices_from_state(state, att)?;
Ok(SplitAttestation::new(att.clone(), attesting_indices))
})
.collect::<Result<Vec<_>, BeaconChainError>>()?;
let mut per_attestation_rewards = split_attestations
.iter()
.map(|att| {
AttMaxCover::new(
att.as_ref(),
state,
reward_cache,
total_active_balance,
&self.spec,
)
.ok_or(BeaconChainError::BlockRewardAttestationError)
})
.collect::<Result<Vec<_>, _>>()?;
@ -34,7 +55,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
let latest_att = &updated[i];
for att in to_update {
att.update_covering_set(latest_att.object(), latest_att.covering_set());
att.update_covering_set(latest_att.intermediate(), latest_att.covering_set());
}
}

10
beacon_node/beacon_chain/src/block_verification.rs
View File

@ -1307,8 +1307,14 @@ impl<T: BeaconChainTypes> ExecutionPendingBlock<T> {
*/
if let Some(ref event_handler) = chain.event_handler {
if event_handler.has_block_reward_subscribers() {
let block_reward =
chain.compute_block_reward(block.message(), block_root, &state, true)?;
let mut reward_cache = Default::default();
let block_reward = chain.compute_block_reward(
block.message(),
block_root,
&state,
&mut reward_cache,
true,
)?;
event_handler.register(EventKind::BlockReward(block_reward));
}
}

3
beacon_node/beacon_chain/src/chain_config.rs
View File

@ -35,6 +35,8 @@ pub struct ChainConfig {
/// Whether any chain health checks should be considered when deciding whether to use the builder API.
pub builder_fallback_disable_checks: bool,
pub count_unrealized: bool,
/// Whether to apply paranoid checks to blocks proposed by this beacon node.
pub paranoid_block_proposal: bool,
}
impl Default for ChainConfig {
@ -52,6 +54,7 @@ impl Default for ChainConfig {
builder_fallback_epochs_since_finalization: 3,
builder_fallback_disable_checks: false,
count_unrealized: true,
paranoid_block_proposal: false,
}
}
}

34
beacon_node/beacon_chain/src/observed_operations.rs
View File

@ -1,10 +1,12 @@
use derivative::Derivative;
use smallvec::SmallVec;
use ssz::{Decode, Encode};
use state_processing::{SigVerifiedOp, VerifyOperation};
use std::collections::HashSet;
use std::marker::PhantomData;
use types::{
AttesterSlashing, BeaconState, ChainSpec, EthSpec, ProposerSlashing, SignedVoluntaryExit,
AttesterSlashing, BeaconState, ChainSpec, EthSpec, ForkName, ProposerSlashing,
SignedVoluntaryExit, Slot,
};
/// Number of validator indices to store on the stack in `observed_validators`.
@ -24,13 +26,16 @@ pub struct ObservedOperations<T: ObservableOperation<E>, E: EthSpec> {
/// previously seen attester slashings, i.e. those validators in the intersection of
/// `attestation_1.attester_indices` and `attestation_2.attester_indices`.
observed_validator_indices: HashSet<u64>,
/// The name of the current fork. The default will be overwritten on first use.
#[derivative(Default(value = "ForkName::Base"))]
current_fork: ForkName,
_phantom: PhantomData<(T, E)>,
}
/// Was the observed operation new and valid for further processing, or a useless duplicate?
#[derive(Debug, PartialEq, Eq, Clone)]
pub enum ObservationOutcome<T> {
New(SigVerifiedOp<T>),
pub enum ObservationOutcome<T: Encode + Decode, E: EthSpec> {
New(SigVerifiedOp<T, E>),
AlreadyKnown,
}
@ -81,7 +86,9 @@ impl<T: ObservableOperation<E>, E: EthSpec> ObservedOperations<T, E> {
op: T,
head_state: &BeaconState<E>,
spec: &ChainSpec,
) -> Result<ObservationOutcome<T>, T::Error> {
) -> Result<ObservationOutcome<T, E>, T::Error> {
self.reset_at_fork_boundary(head_state.slot(), spec);
let observed_validator_indices = &mut self.observed_validator_indices;
let new_validator_indices = op.observed_validators();
@ -107,4 +114,23 @@ impl<T: ObservableOperation<E>, E: EthSpec> ObservedOperations<T, E> {
Ok(ObservationOutcome::New(verified_op))
}
/// Reset the cache when crossing a fork boundary.
///
/// This prevents an attacker from crafting a self-slashing which is only valid before the fork
/// (e.g. using the Altair fork domain at a Bellatrix epoch), in order to prevent propagation of
/// all other slashings due to the duplicate check.
///
/// It doesn't matter if this cache gets reset too often, as we reset it on restart anyway and a
/// false negative just results in propagation of messages which should have been ignored.
///
/// In future we could check slashing relevance against the op pool itself, but that would
/// require indexing the attester slashings in the op pool by validator index.
fn reset_at_fork_boundary(&mut self, head_slot: Slot, spec: &ChainSpec) {
let head_fork = spec.fork_name_at_slot::<E>(head_slot);
if head_fork != self.current_fork {
self.observed_validator_indices.clear();
self.current_fork = head_fork;
}
}
}

11
beacon_node/beacon_chain/src/schema_change.rs
View File

@ -1,6 +1,7 @@
//! Utilities for managing database schema changes.
mod migration_schema_v10;
mod migration_schema_v11;
mod migration_schema_v12;
mod migration_schema_v6;
mod migration_schema_v7;
mod migration_schema_v8;
@ -196,6 +197,16 @@ pub fn migrate_schema<T: BeaconChainTypes>(
Ok(())
}
// Upgrade from v11 to v12 to store richer metadata in the attestation op pool.
(SchemaVersion(11), SchemaVersion(12)) => {
let ops = migration_schema_v12::upgrade_to_v12::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
// Downgrade from v12 to v11 to drop richer metadata from the attestation op pool.
(SchemaVersion(12), SchemaVersion(11)) => {
let ops = migration_schema_v12::downgrade_from_v12::<T>(db.clone(), log)?;
db.store_schema_version_atomically(to, ops)
}
// Anything else is an error.
(_, _) => Err(HotColdDBError::UnsupportedSchemaVersion {
target_version: to,

226
beacon_node/beacon_chain/src/schema_change/migration_schema_v12.rs Normal file
View File

@ -0,0 +1,226 @@
use crate::beacon_chain::{BeaconChainTypes, FORK_CHOICE_DB_KEY, OP_POOL_DB_KEY};
use crate::persisted_fork_choice::PersistedForkChoiceV11;
use operation_pool::{PersistedOperationPool, PersistedOperationPoolV12, PersistedOperationPoolV5};
use slog::{debug, info, Logger};
use state_processing::{
common::get_indexed_attestation, per_block_processing::is_valid_indexed_attestation,
VerifyOperation, VerifySignatures,
};
use std::sync::Arc;
use store::{Error, HotColdDB, KeyValueStoreOp, StoreItem};
pub fn upgrade_to_v12<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
let spec = db.get_chain_spec();
// Load a V5 op pool and transform it to V12.
let PersistedOperationPoolV5 {
attestations_v5,
sync_contributions,
attester_slashings_v5,
proposer_slashings_v5,
voluntary_exits_v5,
} = if let Some(op_pool) = db.get_item(&OP_POOL_DB_KEY)? {
op_pool
} else {
debug!(log, "Nothing to do, no operation pool stored");
return Ok(vec![]);
};
// Load the persisted fork choice so we can grab the state of the justified block and use
// it to verify the stored attestations, slashings and exits.
let fork_choice = db
.get_item::<PersistedForkChoiceV11>(&FORK_CHOICE_DB_KEY)?
.ok_or_else(|| Error::SchemaMigrationError("fork choice missing from database".into()))?;
let justified_block_root = fork_choice
.fork_choice_store
.unrealized_justified_checkpoint
.root;
let justified_block = db
.get_blinded_block(&justified_block_root)?
.ok_or_else(|| {
Error::SchemaMigrationError(format!(
"unrealized justified block missing for migration: {justified_block_root:?}",
))
})?;
let justified_state_root = justified_block.state_root();
let mut state = db
.get_state(&justified_state_root, Some(justified_block.slot()))?
.ok_or_else(|| {
Error::SchemaMigrationError(format!(
"justified state missing for migration: {justified_state_root:?}"
))
})?;
state.build_all_committee_caches(spec).map_err(|e| {
Error::SchemaMigrationError(format!("unable to build committee caches: {e:?}"))
})?;
// Re-verify attestations while adding attesting indices.
let attestations = attestations_v5
.into_iter()
.flat_map(|(_, attestations)| attestations)
.filter_map(|attestation| {
let res = state
.get_beacon_committee(attestation.data.slot, attestation.data.index)
.map_err(Into::into)
.and_then(|committee| get_indexed_attestation(committee.committee, &attestation))
.and_then(|indexed_attestation| {
is_valid_indexed_attestation(
&state,
&indexed_attestation,
VerifySignatures::True,
spec,
)?;
Ok(indexed_attestation)
});
match res {
Ok(indexed) => Some((attestation, indexed.attesting_indices.into())),
Err(e) => {
debug!(
log,
"Dropping attestation on migration";
"err" => ?e,
"head_block" => ?attestation.data.beacon_block_root,
);
None
}
}
})
.collect::<Vec<_>>();
let attester_slashings = attester_slashings_v5
.iter()
.filter_map(|(slashing, _)| {
slashing
.clone()
.validate(&state, spec)
.map_err(|e| {
debug!(
log,
"Dropping attester slashing on migration";
"err" => ?e,
"slashing" => ?slashing,
);
})
.ok()
})
.collect::<Vec<_>>();
let proposer_slashings = proposer_slashings_v5
.iter()
.filter_map(|slashing| {
slashing
.clone()
.validate(&state, spec)
.map_err(|e| {
debug!(
log,
"Dropping proposer slashing on migration";
"err" => ?e,
"slashing" => ?slashing,
);
})
.ok()
})
.collect::<Vec<_>>();
let voluntary_exits = voluntary_exits_v5
.iter()
.filter_map(|exit| {
exit.clone()
.validate(&state, spec)
.map_err(|e| {
debug!(
log,
"Dropping voluntary exit on migration";
"err" => ?e,
"exit" => ?exit,
);
})
.ok()
})
.collect::<Vec<_>>();
debug!(
log,
"Migrated op pool";
"attestations" => attestations.len(),
"attester_slashings" => attester_slashings.len(),
"proposer_slashings" => proposer_slashings.len(),
"voluntary_exits" => voluntary_exits.len()
);
let v12 = PersistedOperationPool::V12(PersistedOperationPoolV12 {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
});
Ok(vec![v12.as_kv_store_op(OP_POOL_DB_KEY)])
}
pub fn downgrade_from_v12<T: BeaconChainTypes>(
db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
log: Logger,
) -> Result<Vec<KeyValueStoreOp>, Error> {
// Load a V12 op pool and transform it to V5.
let PersistedOperationPoolV12 {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
} = if let Some(PersistedOperationPool::<T::EthSpec>::V12(op_pool)) =
db.get_item(&OP_POOL_DB_KEY)?
{
op_pool
} else {
debug!(log, "Nothing to do, no operation pool stored");
return Ok(vec![]);
};
info!(
log,
"Dropping attestations from pool";
"count" => attestations.len(),
);
let attester_slashings_v5 = attester_slashings
.into_iter()
.filter_map(|slashing| {
let fork_version = slashing.first_fork_verified_against()?;
Some((slashing.into_inner(), fork_version))
})
.collect::<Vec<_>>();
let proposer_slashings_v5 = proposer_slashings
.into_iter()
.map(|slashing| slashing.into_inner())
.collect::<Vec<_>>();
let voluntary_exits_v5 = voluntary_exits
.into_iter()
.map(|exit| exit.into_inner())
.collect::<Vec<_>>();
info!(
log,
"Migrated slashings and exits";
"attester_slashings" => attester_slashings_v5.len(),
"proposer_slashings" => proposer_slashings_v5.len(),
"voluntary_exits" => voluntary_exits_v5.len(),
);
let v5 = PersistedOperationPoolV5 {
attestations_v5: vec![],
sync_contributions,
attester_slashings_v5,
proposer_slashings_v5,
voluntary_exits_v5,
};
Ok(vec![v5.as_kv_store_op(OP_POOL_DB_KEY)])
}

33
beacon_node/beacon_chain/src/test_utils.rs
View File

@ -1175,6 +1175,19 @@ where
}
pub fn make_attester_slashing(&self, validator_indices: Vec<u64>) -> AttesterSlashing<E> {
self.make_attester_slashing_with_epochs(validator_indices, None, None, None, None)
}
pub fn make_attester_slashing_with_epochs(
&self,
validator_indices: Vec<u64>,
source1: Option<Epoch>,
target1: Option<Epoch>,
source2: Option<Epoch>,
target2: Option<Epoch>,
) -> AttesterSlashing<E> {
let fork = self.chain.canonical_head.cached_head().head_fork();
let mut attestation_1 = IndexedAttestation {
attesting_indices: VariableList::new(validator_indices).unwrap(),
data: AttestationData {
@ -1183,11 +1196,11 @@ where
beacon_block_root: Hash256::zero(),
target: Checkpoint {
root: Hash256::zero(),
epoch: Epoch::new(0),
epoch: target1.unwrap_or(fork.epoch),
},
source: Checkpoint {
root: Hash256::zero(),
epoch: Epoch::new(0),
epoch: source1.unwrap_or(Epoch::new(0)),
},
},
signature: AggregateSignature::infinity(),
@ -1195,8 +1208,9 @@ where
let mut attestation_2 = attestation_1.clone();
attestation_2.data.index += 1;
attestation_2.data.source.epoch = source2.unwrap_or(Epoch::new(0));
attestation_2.data.target.epoch = target2.unwrap_or(fork.epoch);
let fork = self.chain.canonical_head.cached_head().head_fork();
for attestation in &mut [&mut attestation_1, &mut attestation_2] {
for &i in &attestation.attesting_indices {
let sk = &self.validator_keypairs[i as usize].sk;
@ -1280,8 +1294,19 @@ where
}
pub fn make_proposer_slashing(&self, validator_index: u64) -> ProposerSlashing {
self.make_proposer_slashing_at_slot(validator_index, None)
}
pub fn make_proposer_slashing_at_slot(
&self,
validator_index: u64,
slot_override: Option<Slot>,
) -> ProposerSlashing {
let mut block_header_1 = self.chain.head_beacon_block().message().block_header();
block_header_1.proposer_index = validator_index;
if let Some(slot) = slot_override {
block_header_1.slot = slot;
}
let mut block_header_2 = block_header_1.clone();
block_header_2.state_root = Hash256::zero();
@ -1488,7 +1513,7 @@ where
self.chain
.apply_attestation_to_fork_choice(&verified)
.unwrap();
self.chain.add_to_block_inclusion_pool(&verified).unwrap();
self.chain.add_to_block_inclusion_pool(verified).unwrap();
}
}

4
beacon_node/beacon_chain/tests/store_tests.rs
View File

@ -795,9 +795,7 @@ async fn multiple_attestations_per_block() {
snapshot
.beacon_block
.as_ref()
.clone()
.deconstruct()
.0
.message()
.body()
.attestations()
.len() as u64,

11
beacon_node/http_api/src/block_rewards.rs
View File

@ -52,6 +52,7 @@ pub fn get_block_rewards<T: BeaconChainTypes>(
.build_all_caches(&chain.spec)
.map_err(beacon_state_error)?;
let mut reward_cache = Default::default();
let mut block_rewards = Vec::with_capacity(blocks.len());
let block_replayer = BlockReplayer::new(state, &chain.spec)
@ -63,6 +64,7 @@ pub fn get_block_rewards<T: BeaconChainTypes>(
block.message(),
block.canonical_root(),
state,
&mut reward_cache,
query.include_attestations,
)?;
block_rewards.push(block_reward);
@ -100,6 +102,7 @@ pub fn compute_block_rewards<T: BeaconChainTypes>(
) -> Result<Vec<BlockReward>, warp::Rejection> {
let mut block_rewards = Vec::with_capacity(blocks.len());
let mut state_cache = LruCache::new(STATE_CACHE_SIZE);
let mut reward_cache = Default::default();
for block in blocks {
let parent_root = block.parent_root();
@ -170,7 +173,13 @@ pub fn compute_block_rewards<T: BeaconChainTypes>(
// Compute block reward.
let block_reward = chain
.compute_block_reward(block.to_ref(), block.canonical_root(), state, true)
.compute_block_reward(
block.to_ref(),
block.canonical_root(),
state,
&mut reward_cache,
true,
)
.map_err(beacon_chain_error)?;
block_rewards.push(block_reward);
}

34
beacon_node/http_api/src/lib.rs
View File

@ -45,11 +45,12 @@ use std::sync::Arc;
use tokio::sync::mpsc::UnboundedSender;
use tokio_stream::{wrappers::BroadcastStream, StreamExt};
use types::{
Attestation, AttesterSlashing, BeaconStateError, BlindedPayload, CommitteeCache,
ConfigAndPreset, Epoch, EthSpec, ForkName, FullPayload, ProposerPreparationData,
ProposerSlashing, RelativeEpoch, Signature, SignedAggregateAndProof, SignedBeaconBlock,
SignedBlindedBeaconBlock, SignedContributionAndProof, SignedValidatorRegistrationData,
SignedVoluntaryExit, Slot, SyncCommitteeMessage, SyncContributionData,
Attestation, AttestationData, AttesterSlashing, BeaconStateError, BlindedPayload,
CommitteeCache, ConfigAndPreset, Epoch, EthSpec, ForkName, FullPayload,
ProposerPreparationData, ProposerSlashing, RelativeEpoch, Signature, SignedAggregateAndProof,
SignedBeaconBlock, SignedBlindedBeaconBlock, SignedContributionAndProof,
SignedValidatorRegistrationData, SignedVoluntaryExit, Slot, SyncCommitteeMessage,
SyncContributionData,
};
use version::{
add_consensus_version_header, execution_optimistic_fork_versioned_response,
@ -1305,13 +1306,11 @@ pub fn serve<T: BeaconChainTypes>(
.and_then(
|chain: Arc<BeaconChain<T>>, query: api_types::AttestationPoolQuery| {
blocking_json_task(move || {
let query_filter = |attestation: &Attestation<T::EthSpec>| {
query
.slot
.map_or(true, |slot| slot == attestation.data.slot)
let query_filter = |data: &AttestationData| {
query.slot.map_or(true, |slot| slot == data.slot)
&& query
.committee_index
.map_or(true, |index| index == attestation.data.index)
.map_or(true, |index| index == data.index)
};
let mut attestations = chain.op_pool.get_filtered_attestations(query_filter);
@ -1321,7 +1320,7 @@ pub fn serve<T: BeaconChainTypes>(
.read()
.iter()
.cloned()
.filter(query_filter),
.filter(|att| query_filter(&att.data)),
);
Ok(api_types::GenericResponse::from(attestations))
})
@ -2317,12 +2316,13 @@ pub fn serve<T: BeaconChainTypes>(
);
failures.push(api_types::Failure::new(index, format!("Fork choice: {:?}", e)));
}
if let Err(e) = chain.add_to_block_inclusion_pool(&verified_aggregate) {
warn!(log,
"Could not add verified aggregate attestation to the inclusion pool";
"error" => format!("{:?}", e),
"request_index" => index,
);
if let Err(e) = chain.add_to_block_inclusion_pool(verified_aggregate) {
warn!(
log,
"Could not add verified aggregate attestation to the inclusion pool";
"error" => ?e,
"request_index" => index,
);
failures.push(api_types::Failure::new(index, format!("Op pool: {:?}", e)));
}
}

15
beacon_node/network/src/beacon_processor/worker/gossip_methods.rs
View File

@ -54,6 +54,12 @@ impl<T: BeaconChainTypes> VerifiedAttestation<T> for VerifiedUnaggregate<T> {
fn indexed_attestation(&self) -> &IndexedAttestation<T::EthSpec> {
&self.indexed_attestation
}
fn into_attestation_and_indices(self) -> (Attestation<T::EthSpec>, Vec<u64>) {
let attestation = *self.attestation;
let attesting_indices = self.indexed_attestation.attesting_indices.into();
(attestation, attesting_indices)
}
}
/// An attestation that failed validation by the `BeaconChain`.
@ -81,6 +87,13 @@ impl<T: BeaconChainTypes> VerifiedAttestation<T> for VerifiedAggregate<T> {
fn indexed_attestation(&self) -> &IndexedAttestation<T::EthSpec> {
&self.indexed_attestation
}
/// Efficient clone-free implementation that moves out of the `Box`.
fn into_attestation_and_indices(self) -> (Attestation<T::EthSpec>, Vec<u64>) {
let attestation = self.signed_aggregate.message.aggregate;
let attesting_indices = self.indexed_attestation.attesting_indices.into();
(attestation, attesting_indices)
}
}
/// An attestation that failed validation by the `BeaconChain`.
@ -595,7 +608,7 @@ impl<T: BeaconChainTypes> Worker<T> {
}
}
if let Err(e) = self.chain.add_to_block_inclusion_pool(&verified_aggregate) {
if let Err(e) = self.chain.add_to_block_inclusion_pool(verified_aggregate) {
debug!(
self.log,
"Attestation invalid for op pool";

2
beacon_node/operation_pool/Cargo.toml
View File

@ -18,7 +18,9 @@ rayon = "1.5.0"
serde = "1.0.116"
serde_derive = "1.0.116"
store = { path = "../store" }
bitvec = "1"
[dev-dependencies]
beacon_chain = { path = "../beacon_chain" }
tokio = { version = "1.14.0", features = ["rt-multi-thread"] }
maplit = "1.0.2"

74
beacon_node/operation_pool/src/attestation.rs
View File

@ -1,4 +1,6 @@
use crate::attestation_storage::AttestationRef;
use crate::max_cover::MaxCover;
use crate::reward_cache::RewardCache;
use state_processing::common::{
altair, base, get_attestation_participation_flag_indices, get_attesting_indices,
};
@ -12,34 +14,35 @@ use types::{
#[derive(Debug, Clone)]
pub struct AttMaxCover<'a, T: EthSpec> {
/// Underlying attestation.
pub att: &'a Attestation<T>,
pub att: AttestationRef<'a, T>,
/// Mapping of validator indices and their rewards.
pub fresh_validators_rewards: HashMap<u64, u64>,
}
impl<'a, T: EthSpec> AttMaxCover<'a, T> {
pub fn new(
att: &'a Attestation<T>,
att: AttestationRef<'a, T>,
state: &BeaconState<T>,
reward_cache: &'a RewardCache,
total_active_balance: u64,
spec: &ChainSpec,
) -> Option<Self> {
if let BeaconState::Base(ref base_state) = state {
Self::new_for_base(att, state, base_state, total_active_balance, spec)
} else {
Self::new_for_altair(att, state, total_active_balance, spec)
Self::new_for_altair(att, state, reward_cache, total_active_balance, spec)
}
}
/// Initialise an attestation cover object for base/phase0 hard fork.
pub fn new_for_base(
att: &'a Attestation<T>,
att: AttestationRef<'a, T>,
state: &BeaconState<T>,
base_state: &BeaconStateBase<T>,
total_active_balance: u64,
spec: &ChainSpec,
) -> Option<Self> {
let fresh_validators = earliest_attestation_validators(att, state, base_state);
let fresh_validators = earliest_attestation_validators(&att, state, base_state);
let committee = state
.get_beacon_committee(att.data.slot, att.data.index)
.ok()?;
@ -67,45 +70,41 @@ impl<'a, T: EthSpec> AttMaxCover<'a, T> {
/// Initialise an attestation cover object for Altair or later.
pub fn new_for_altair(
att: &'a Attestation<T>,
att: AttestationRef<'a, T>,
state: &BeaconState<T>,
reward_cache: &'a RewardCache,
total_active_balance: u64,
spec: &ChainSpec,
) -> Option<Self> {
let committee = state
.get_beacon_committee(att.data.slot, att.data.index)
.ok()?;
let attesting_indices =
get_attesting_indices::<T>(committee.committee, &att.aggregation_bits).ok()?;
let att_data = att.attestation_data();
let participation_list = if att.data.target.epoch == state.current_epoch() {
state.current_epoch_participation().ok()?
} else if att.data.target.epoch == state.previous_epoch() {
state.previous_epoch_participation().ok()?
} else {
return None;
};
let inclusion_delay = state.slot().as_u64().checked_sub(att.data.slot.as_u64())?;
let inclusion_delay = state.slot().as_u64().checked_sub(att_data.slot.as_u64())?;
let att_participation_flags =
get_attestation_participation_flag_indices(state, &att.data, inclusion_delay, spec)
get_attestation_participation_flag_indices(state, &att_data, inclusion_delay, spec)
.ok()?;
let base_reward_per_increment =
altair::BaseRewardPerIncrement::new(total_active_balance, spec).ok()?;
let fresh_validators_rewards = attesting_indices
let fresh_validators_rewards = att
.indexed
.attesting_indices
.iter()
.filter_map(|&index| {
if reward_cache
.has_attested_in_epoch(index, att_data.target.epoch)
.ok()?
{
return None;
}
let mut proposer_reward_numerator = 0;
let participation = participation_list.get(index)?;
let base_reward =
altair::get_base_reward(state, index, base_reward_per_increment, spec).ok()?;
altair::get_base_reward(state, index as usize, base_reward_per_increment, spec)
.ok()?;
for (flag_index, weight) in PARTICIPATION_FLAG_WEIGHTS.iter().enumerate() {
if att_participation_flags.contains(&flag_index)
&& !participation.has_flag(flag_index).ok()?
{
if att_participation_flags.contains(&flag_index) {
proposer_reward_numerator += base_reward.checked_mul(*weight)?;
}
}
@ -113,7 +112,7 @@ impl<'a, T: EthSpec> AttMaxCover<'a, T> {
let proposer_reward = proposer_reward_numerator
.checked_div(WEIGHT_DENOMINATOR.checked_mul(spec.proposer_reward_quotient)?)?;
Some((index as u64, proposer_reward)).filter(|_| proposer_reward != 0)
Some((index, proposer_reward)).filter(|_| proposer_reward != 0)
})
.collect();
@ -126,10 +125,15 @@ impl<'a, T: EthSpec> AttMaxCover<'a, T> {
impl<'a, T: EthSpec> MaxCover for AttMaxCover<'a, T> {
type Object = Attestation<T>;
type Intermediate = AttestationRef<'a, T>;
type Set = HashMap<u64, u64>;
fn object(&self) -> &Attestation<T> {
self.att
fn intermediate(&self) -> &AttestationRef<'a, T> {
&self.att
}
fn convert_to_object(att_ref: &AttestationRef<'a, T>) -> Attestation<T> {
att_ref.clone_as_attestation()
}
fn covering_set(&self) -> &HashMap<u64, u64> {
@ -148,7 +152,7 @@ impl<'a, T: EthSpec> MaxCover for AttMaxCover<'a, T> {
/// of slashable voting, which is rare.
fn update_covering_set(
&mut self,
best_att: &Attestation<T>,
best_att: &AttestationRef<'a, T>,
covered_validators: &HashMap<u64, u64>,
) {
if self.att.data.slot == best_att.data.slot && self.att.data.index == best_att.data.index {
@ -172,16 +176,16 @@ impl<'a, T: EthSpec> MaxCover for AttMaxCover<'a, T> {
///
/// This isn't optimal, but with the Altair fork this code is obsolete and not worth upgrading.
pub fn earliest_attestation_validators<T: EthSpec>(
attestation: &Attestation<T>,
attestation: &AttestationRef<T>,
state: &BeaconState<T>,
base_state: &BeaconStateBase<T>,
) -> BitList<T::MaxValidatorsPerCommittee> {
// Bitfield of validators whose attestations are new/fresh.
let mut new_validators = attestation.aggregation_bits.clone();
let mut new_validators = attestation.indexed.aggregation_bits.clone();
let state_attestations = if attestation.data.target.epoch == state.current_epoch() {
let state_attestations = if attestation.checkpoint.target_epoch == state.current_epoch() {
&base_state.current_epoch_attestations
} else if attestation.data.target.epoch == state.previous_epoch() {
} else if attestation.checkpoint.target_epoch == state.previous_epoch() {
&base_state.previous_epoch_attestations
} else {
return BitList::with_capacity(0).unwrap();

37
beacon_node/operation_pool/src/attestation_id.rs
View File

@ -1,45 +1,12 @@
use serde_derive::{Deserialize, Serialize};
use ssz::ssz_encode;
use ssz_derive::{Decode, Encode};
use types::{AttestationData, ChainSpec, Domain, Epoch, Fork, Hash256};
/// Serialized `AttestationData` augmented with a domain to encode the fork info.
///
/// [DEPRECATED] To be removed once all nodes have updated to schema v12.
#[derive(
PartialEq, Eq, Clone, Hash, Debug, PartialOrd, Ord, Encode, Decode, Serialize, Deserialize,
)]
pub struct AttestationId {
v: Vec<u8>,
}
/// Number of domain bytes that the end of an attestation ID is padded with.
const DOMAIN_BYTES_LEN: usize = std::mem::size_of::<Hash256>();
impl AttestationId {
pub fn from_data(
attestation: &AttestationData,
fork: &Fork,
genesis_validators_root: Hash256,
spec: &ChainSpec,
) -> Self {
let mut bytes = ssz_encode(attestation);
let epoch = attestation.target.epoch;
bytes.extend_from_slice(
AttestationId::compute_domain_bytes(epoch, fork, genesis_validators_root, spec)
.as_bytes(),
);
AttestationId { v: bytes }
}
pub fn compute_domain_bytes(
epoch: Epoch,
fork: &Fork,
genesis_validators_root: Hash256,
spec: &ChainSpec,
) -> Hash256 {
spec.get_domain(epoch, Domain::BeaconAttester, fork, genesis_validators_root)
}
pub fn domain_bytes_match(&self, domain_bytes: &Hash256) -> bool {
&self.v[self.v.len() - DOMAIN_BYTES_LEN..] == domain_bytes.as_bytes()
}
}

245
beacon_node/operation_pool/src/attestation_storage.rs Normal file
View File

@ -0,0 +1,245 @@
use crate::AttestationStats;
use itertools::Itertools;
use std::collections::HashMap;
use types::{
AggregateSignature, Attestation, AttestationData, BeaconState, BitList, Checkpoint, Epoch,
EthSpec, Hash256, Slot,
};
#[derive(Debug, PartialEq, Eq, Hash, Clone, Copy)]
pub struct CheckpointKey {
pub source: Checkpoint,
pub target_epoch: Epoch,
}
#[derive(Debug, PartialEq, Eq, Hash)]
pub struct CompactAttestationData {
pub slot: Slot,
pub index: u64,
pub beacon_block_root: Hash256,
pub target_root: Hash256,
}
#[derive(Debug, PartialEq)]
pub struct CompactIndexedAttestation<T: EthSpec> {
pub attesting_indices: Vec<u64>,
pub aggregation_bits: BitList<T::MaxValidatorsPerCommittee>,
pub signature: AggregateSignature,
}
#[derive(Debug)]
pub struct SplitAttestation<T: EthSpec> {
pub checkpoint: CheckpointKey,
pub data: CompactAttestationData,
pub indexed: CompactIndexedAttestation<T>,
}
#[derive(Debug, Clone)]
pub struct AttestationRef<'a, T: EthSpec> {
pub checkpoint: &'a CheckpointKey,
pub data: &'a CompactAttestationData,
pub indexed: &'a CompactIndexedAttestation<T>,
}
#[derive(Debug, Default, PartialEq)]
pub struct AttestationMap<T: EthSpec> {
checkpoint_map: HashMap<CheckpointKey, AttestationDataMap<T>>,
}
#[derive(Debug, Default, PartialEq)]
pub struct AttestationDataMap<T: EthSpec> {
attestations: HashMap<CompactAttestationData, Vec<CompactIndexedAttestation<T>>>,
}
impl<T: EthSpec> SplitAttestation<T> {
pub fn new(attestation: Attestation<T>, attesting_indices: Vec<u64>) -> Self {
let checkpoint = CheckpointKey {
source: attestation.data.source,
target_epoch: attestation.data.target.epoch,
};
let data = CompactAttestationData {
slot: attestation.data.slot,
index: attestation.data.index,
beacon_block_root: attestation.data.beacon_block_root,
target_root: attestation.data.target.root,
};
let indexed = CompactIndexedAttestation {
attesting_indices,
aggregation_bits: attestation.aggregation_bits,
signature: attestation.signature,
};
Self {
checkpoint,
data,
indexed,
}
}
pub fn as_ref(&self) -> AttestationRef<T> {
AttestationRef {
checkpoint: &self.checkpoint,
data: &self.data,
indexed: &self.indexed,
}
}
}
impl<'a, T: EthSpec> AttestationRef<'a, T> {
pub fn attestation_data(&self) -> AttestationData {
AttestationData {
slot: self.data.slot,
index: self.data.index,
beacon_block_root: self.data.beacon_block_root,
source: self.checkpoint.source,
target: Checkpoint {
epoch: self.checkpoint.target_epoch,
root: self.data.target_root,
},
}
}
pub fn clone_as_attestation(&self) -> Attestation<T> {
Attestation {
aggregation_bits: self.indexed.aggregation_bits.clone(),
data: self.attestation_data(),
signature: self.indexed.signature.clone(),
}
}
}
impl CheckpointKey {
/// Return two checkpoint keys: `(previous, current)` for the previous and current epochs of
/// the `state`.
pub fn keys_for_state<T: EthSpec>(state: &BeaconState<T>) -> (Self, Self) {
(
CheckpointKey {
source: state.previous_justified_checkpoint(),
target_epoch: state.previous_epoch(),
},
CheckpointKey {
source: state.current_justified_checkpoint(),
target_epoch: state.current_epoch(),
},
)
}
}
impl<T: EthSpec> CompactIndexedAttestation<T> {
pub fn signers_disjoint_from(&self, other: &Self) -> bool {
self.aggregation_bits
.intersection(&other.aggregation_bits)
.is_zero()
}
pub fn aggregate(&mut self, other: &Self) {
self.attesting_indices = self
.attesting_indices
.drain(..)
.merge(other.attesting_indices.iter().copied())
.dedup()
.collect();
self.aggregation_bits = self.aggregation_bits.union(&other.aggregation_bits);
self.signature.add_assign_aggregate(&other.signature);
}
}
impl<T: EthSpec> AttestationMap<T> {
pub fn insert(&mut self, attestation: Attestation<T>, attesting_indices: Vec<u64>) {
let SplitAttestation {
checkpoint,
data,
indexed,
} = SplitAttestation::new(attestation, attesting_indices);
let attestation_map = self
.checkpoint_map
.entry(checkpoint)
.or_insert_with(AttestationDataMap::default);
let attestations = attestation_map
.attestations
.entry(data)
.or_insert_with(Vec::new);
// Greedily aggregate the attestation with all existing attestations.
// NOTE: this is sub-optimal and in future we will remove this in favour of max-clique
// aggregation.
let mut aggregated = false;
for existing_attestation in attestations.iter_mut() {
if existing_attestation.signers_disjoint_from(&indexed) {
existing_attestation.aggregate(&indexed);
aggregated = true;
} else if *existing_attestation == indexed {
aggregated = true;
}
}
if !aggregated {
attestations.push(indexed);
}
}
/// Iterate all attestations matching the given `checkpoint_key`.
pub fn get_attestations<'a>(
&'a self,
checkpoint_key: &'a CheckpointKey,
) -> impl Iterator<Item = AttestationRef<'a, T>> + 'a {
self.checkpoint_map
.get(checkpoint_key)
.into_iter()
.flat_map(|attestation_map| attestation_map.iter(checkpoint_key))
}
/// Iterate all attestations in the map.
pub fn iter(&self) -> impl Iterator<Item = AttestationRef<T>> {
self.checkpoint_map
.iter()
.flat_map(|(checkpoint_key, attestation_map)| attestation_map.iter(checkpoint_key))
}
/// Prune attestations that are from before the previous epoch.
pub fn prune(&mut self, current_epoch: Epoch) {
self.checkpoint_map
.retain(|checkpoint_key, _| current_epoch <= checkpoint_key.target_epoch + 1);
}
/// Statistics about all attestations stored in the map.
pub fn stats(&self) -> AttestationStats {
self.checkpoint_map
.values()
.map(AttestationDataMap::stats)
.fold(AttestationStats::default(), |mut acc, new| {
acc.num_attestations += new.num_attestations;
acc.num_attestation_data += new.num_attestation_data;
acc.max_aggregates_per_data =
std::cmp::max(acc.max_aggregates_per_data, new.max_aggregates_per_data);
acc
})
}
}
impl<T: EthSpec> AttestationDataMap<T> {
pub fn iter<'a>(
&'a self,
checkpoint_key: &'a CheckpointKey,
) -> impl Iterator<Item = AttestationRef<'a, T>> + 'a {
self.attestations.iter().flat_map(|(data, vec_indexed)| {
vec_indexed.iter().map(|indexed| AttestationRef {
checkpoint: checkpoint_key,
data,
indexed,
})
})
}
pub fn stats(&self) -> AttestationStats {
let mut stats = AttestationStats::default();
for aggregates in self.attestations.values() {
stats.num_attestations += aggregates.len();
stats.num_attestation_data += 1;
stats.max_aggregates_per_data =
std::cmp::max(stats.max_aggregates_per_data, aggregates.len());
}
stats
}
}

8
beacon_node/operation_pool/src/attester_slashing.rs
View File

@ -39,14 +39,18 @@ impl<'a, T: EthSpec> AttesterSlashingMaxCover<'a, T> {
impl<'a, T: EthSpec> MaxCover for AttesterSlashingMaxCover<'a, T> {
/// The result type, of which we would eventually like a collection of maximal quality.
type Object = AttesterSlashing<T>;
type Intermediate = AttesterSlashing<T>;
/// The type used to represent sets.
type Set = HashMap<u64, u64>;
/// Extract an object for inclusion in a solution.
fn object(&self) -> &AttesterSlashing<T> {
fn intermediate(&self) -> &AttesterSlashing<T> {
self.slashing
}
fn convert_to_object(slashing: &AttesterSlashing<T>) -> AttesterSlashing<T> {
slashing.clone()
}
/// Get the set of elements covered.
fn covering_set(&self) -> &HashMap<u64, u64> {
&self.effective_balances

811
beacon_node/operation_pool/src/lib.rs
View File

File diff suppressed because it is too large Load Diff

22
beacon_node/operation_pool/src/max_cover.rs
View File

@ -11,16 +11,21 @@ use itertools::Itertools;
pub trait MaxCover: Clone {
/// The result type, of which we would eventually like a collection of maximal quality.
type Object: Clone;
/// The intermediate object type, which can be converted to `Object`.
type Intermediate: Clone;
/// The type used to represent sets.
type Set: Clone;
/// Extract an object for inclusion in a solution.
fn object(&self) -> &Self::Object;
/// Extract the intermediate object.
fn intermediate(&self) -> &Self::Intermediate;
/// Convert the borrowed intermediate object to an owned object for the solution.
fn convert_to_object(intermediate: &Self::Intermediate) -> Self::Object;
/// Get the set of elements covered.
fn covering_set(&self) -> &Self::Set;
/// Update the set of items covered, for the inclusion of some object in the solution.
fn update_covering_set(&mut self, max_obj: &Self::Object, max_set: &Self::Set);
fn update_covering_set(&mut self, max_obj: &Self::Intermediate, max_set: &Self::Set);
/// The quality of this item's covering set, usually its cardinality.
fn score(&self) -> usize;
}
@ -86,7 +91,7 @@ where
.filter(|x| x.available && x.item.score() != 0)
.for_each(|x| {
x.item
.update_covering_set(best.object(), best.covering_set())
.update_covering_set(best.intermediate(), best.covering_set())
});
result.push(best);
@ -106,7 +111,7 @@ where
.into_iter()
.merge_by(cover2, |item1, item2| item1.score() >= item2.score())
.take(limit)
.map(|item| item.object().clone())
.map(|item| T::convert_to_object(item.intermediate()))
.collect()
}
@ -121,12 +126,17 @@ mod test {
T: Clone + Eq + Hash,
{
type Object = Self;
type Intermediate = Self;
type Set = Self;
fn object(&self) -> &Self {
fn intermediate(&self) -> &Self {
self
}
fn convert_to_object(set: &Self) -> Self {
set.clone()
}
fn covering_set(&self) -> &Self {
self
}

4
beacon_node/operation_pool/src/metrics.rs
View File

@ -3,6 +3,10 @@ use lazy_static::lazy_static;
pub use lighthouse_metrics::*;
lazy_static! {
pub static ref BUILD_REWARD_CACHE_TIME: Result<Histogram> = try_create_histogram(
"op_pool_build_reward_cache_time",
"Time to build the reward cache before packing attestations"
);
pub static ref ATTESTATION_PREV_EPOCH_PACKING_TIME: Result<Histogram> = try_create_histogram(
"op_pool_attestation_prev_epoch_packing_time",
"Time to pack previous epoch attestations"

121
beacon_node/operation_pool/src/persistence.rs
View File

@ -1,12 +1,13 @@
use crate::attestation_id::AttestationId;
use crate::attestation_storage::AttestationMap;
use crate::sync_aggregate_id::SyncAggregateId;
use crate::OpPoolError;
use crate::OperationPool;
use derivative::Derivative;
use parking_lot::RwLock;
use serde_derive::{Deserialize, Serialize};
use ssz::{Decode, Encode};
use ssz_derive::{Decode, Encode};
use state_processing::SigVerifiedOp;
use store::{DBColumn, Error as StoreError, StoreItem};
use types::*;
@ -17,32 +18,42 @@ type PersistedSyncContributions<T> = Vec<(SyncAggregateId, Vec<SyncCommitteeCont
/// Operations are stored in arbitrary order, so it's not a good idea to compare instances
/// of this type (or its encoded form) for equality. Convert back to an `OperationPool` first.
#[superstruct(
variants(Altair),
variants(V5, V12),
variant_attributes(
derive(Derivative, PartialEq, Debug, Serialize, Deserialize, Encode, Decode),
serde(bound = "T: EthSpec", deny_unknown_fields),
derive(Derivative, PartialEq, Debug, Encode, Decode),
derivative(Clone),
),
partial_getter_error(ty = "OpPoolError", expr = "OpPoolError::IncorrectOpPoolVariant")
)]
#[derive(PartialEq, Debug, Serialize, Deserialize, Encode)]
#[serde(untagged)]
#[serde(bound = "T: EthSpec")]
#[derive(PartialEq, Debug, Encode)]
#[ssz(enum_behaviour = "transparent")]
pub struct PersistedOperationPool<T: EthSpec> {
/// Mapping from attestation ID to attestation mappings.
// We could save space by not storing the attestation ID, but it might
// be difficult to make that roundtrip due to eager aggregation.
attestations: Vec<(AttestationId, Vec<Attestation<T>>)>,
/// [DEPRECATED] Mapping from attestation ID to attestation mappings.
#[superstruct(only(V5))]
pub attestations_v5: Vec<(AttestationId, Vec<Attestation<T>>)>,
/// Attestations and their attesting indices.
#[superstruct(only(V12))]
pub attestations: Vec<(Attestation<T>, Vec<u64>)>,
/// Mapping from sync contribution ID to sync contributions and aggregate.
#[superstruct(only(Altair))]
sync_contributions: PersistedSyncContributions<T>,
pub sync_contributions: PersistedSyncContributions<T>,
/// [DEPRECATED] Attester slashings.
#[superstruct(only(V5))]
pub attester_slashings_v5: Vec<(AttesterSlashing<T>, ForkVersion)>,
/// Attester slashings.
attester_slashings: Vec<(AttesterSlashing<T>, ForkVersion)>,
/// Proposer slashings.
proposer_slashings: Vec<ProposerSlashing>,
/// Voluntary exits.
voluntary_exits: Vec<SignedVoluntaryExit>,
#[superstruct(only(V12))]
pub attester_slashings: Vec<SigVerifiedOp<AttesterSlashing<T>, T>>,
/// [DEPRECATED] Proposer slashings.
#[superstruct(only(V5))]
pub proposer_slashings_v5: Vec<ProposerSlashing>,
/// Proposer slashings with fork information.
#[superstruct(only(V12))]
pub proposer_slashings: Vec<SigVerifiedOp<ProposerSlashing, T>>,
/// [DEPRECATED] Voluntary exits.
#[superstruct(only(V5))]
pub voluntary_exits_v5: Vec<SignedVoluntaryExit>,
/// Voluntary exits with fork information.
#[superstruct(only(V12))]
pub voluntary_exits: Vec<SigVerifiedOp<SignedVoluntaryExit, T>>,
}
impl<T: EthSpec> PersistedOperationPool<T> {
@ -52,7 +63,12 @@ impl<T: EthSpec> PersistedOperationPool<T> {
.attestations
.read()
.iter()
.map(|(att_id, att)| (att_id.clone(), att.clone()))
.map(|att| {
(
att.clone_as_attestation(),
att.indexed.attesting_indices.clone(),
)
})
.collect();
let sync_contributions = operation_pool
@ -83,7 +99,7 @@ impl<T: EthSpec> PersistedOperationPool<T> {
.map(|(_, exit)| exit.clone())
.collect();
PersistedOperationPool::Altair(PersistedOperationPoolAltair {
PersistedOperationPool::V12(PersistedOperationPoolV12 {
attestations,
sync_contributions,
attester_slashings,
@ -92,45 +108,62 @@ impl<T: EthSpec> PersistedOperationPool<T> {
})
}
/// Reconstruct an `OperationPool`. Sets `sync_contributions` to its `Default` if `self` matches
/// `PersistedOperationPool::Base`.
/// Reconstruct an `OperationPool`.
pub fn into_operation_pool(self) -> Result<OperationPool<T>, OpPoolError> {
let attestations = RwLock::new(self.attestations().iter().cloned().collect());
let attester_slashings = RwLock::new(self.attester_slashings().iter().cloned().collect());
let attester_slashings = RwLock::new(self.attester_slashings()?.iter().cloned().collect());
let proposer_slashings = RwLock::new(
self.proposer_slashings()
self.proposer_slashings()?
.iter()
.cloned()
.map(|slashing| (slashing.signed_header_1.message.proposer_index, slashing))
.map(|slashing| (slashing.as_inner().proposer_index(), slashing))
.collect(),
);
let voluntary_exits = RwLock::new(
self.voluntary_exits()
self.voluntary_exits()?
.iter()
.cloned()
.map(|exit| (exit.message.validator_index, exit))
.map(|exit| (exit.as_inner().message.validator_index, exit))
.collect(),
);
let op_pool = match self {
PersistedOperationPool::Altair(_) => {
let sync_contributions =
RwLock::new(self.sync_contributions()?.iter().cloned().collect());
OperationPool {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
_phantom: Default::default(),
let sync_contributions = RwLock::new(self.sync_contributions().iter().cloned().collect());
let attestations = match self {
PersistedOperationPool::V5(_) => return Err(OpPoolError::IncorrectOpPoolVariant),
PersistedOperationPool::V12(pool) => {
let mut map = AttestationMap::default();
for (att, attesting_indices) in pool.attestations {
map.insert(att, attesting_indices);
}
RwLock::new(map)
}
};
let op_pool = OperationPool {
attestations,
sync_contributions,
attester_slashings,
proposer_slashings,
voluntary_exits,
reward_cache: Default::default(),
_phantom: Default::default(),
};
Ok(op_pool)
}
}
/// Deserialization for `PersistedOperationPool` defaults to `PersistedOperationPool::Altair`.
impl<T: EthSpec> StoreItem for PersistedOperationPoolV5<T> {
fn db_column() -> DBColumn {
DBColumn::OpPool
}
fn as_store_bytes(&self) -> Vec<u8> {
self.as_ssz_bytes()
}
fn from_store_bytes(bytes: &[u8]) -> Result<Self, StoreError> {
PersistedOperationPoolV5::from_ssz_bytes(bytes).map_err(Into::into)
}
}
/// Deserialization for `PersistedOperationPool` defaults to `PersistedOperationPool::V12`.
impl<T: EthSpec> StoreItem for PersistedOperationPool<T> {
fn db_column() -> DBColumn {
DBColumn::OpPool
@ -141,9 +174,9 @@ impl<T: EthSpec> StoreItem for PersistedOperationPool<T> {
}
fn from_store_bytes(bytes: &[u8]) -> Result<Self, StoreError> {
// Default deserialization to the Altair variant.
PersistedOperationPoolAltair::from_ssz_bytes(bytes)
.map(Self::Altair)
// Default deserialization to the latest variant.
PersistedOperationPoolV12::from_ssz_bytes(bytes)
.map(Self::V12)
.map_err(Into::into)
}
}

122
beacon_node/operation_pool/src/reward_cache.rs Normal file
View File

@ -0,0 +1,122 @@
use crate::OpPoolError;
use bitvec::vec::BitVec;
use types::{BeaconState, BeaconStateError, Epoch, EthSpec, Hash256, ParticipationFlags};
#[derive(Debug, PartialEq, Eq, Clone)]
struct Initialization {
current_epoch: Epoch,
latest_block_root: Hash256,
}
/// Cache to store pre-computed information for block proposal.
#[derive(Debug, Clone, Default)]
pub struct RewardCache {
initialization: Option<Initialization>,
/// `BitVec` of validator indices which don't have default participation flags for the prev. epoch.
///
/// We choose to only track whether validators have *any* participation flag set because
/// it's impossible to include a new attestation which is better than the existing participation
/// UNLESS the validator makes a slashable attestation, and we assume that this is rare enough
/// that it's acceptable to be slightly sub-optimal in this case.
previous_epoch_participation: BitVec,
/// `BitVec` of validator indices which don't have default participation flags for the current epoch.
current_epoch_participation: BitVec,
}
impl RewardCache {
pub fn has_attested_in_epoch(
&self,
validator_index: u64,
epoch: Epoch,
) -> Result<bool, OpPoolError> {
if let Some(init) = &self.initialization {
if init.current_epoch == epoch {
Ok(*self
.current_epoch_participation
.get(validator_index as usize)
.ok_or(OpPoolError::RewardCacheOutOfBounds)?)
} else if init.current_epoch == epoch + 1 {
Ok(*self
.previous_epoch_participation
.get(validator_index as usize)
.ok_or(OpPoolError::RewardCacheOutOfBounds)?)
} else {
Err(OpPoolError::RewardCacheWrongEpoch)
}
} else {
Err(OpPoolError::RewardCacheWrongEpoch)
}
}
/// Return the root of the latest block applied to `state`.
///
/// For simplicity at genesis we return the zero hash, which will cause one unnecessary
/// re-calculation in `update`.
fn latest_block_root<E: EthSpec>(state: &BeaconState<E>) -> Result<Hash256, OpPoolError> {
if state.slot() == 0 {
Ok(Hash256::zero())
} else {
Ok(*state
.get_block_root(state.slot() - 1)
.map_err(OpPoolError::RewardCacheGetBlockRoot)?)
}
}
/// Update the cache.
pub fn update<E: EthSpec>(&mut self, state: &BeaconState<E>) -> Result<(), OpPoolError> {
if matches!(state, BeaconState::Base(_)) {
return Ok(());
}
let current_epoch = state.current_epoch();
let latest_block_root = Self::latest_block_root(state)?;
let new_init = Initialization {
current_epoch,
latest_block_root,
};
// The participation flags change every block, and will almost always need updating when
// this function is called at a new slot.
if self
.initialization
.as_ref()
.map_or(true, |init| *init != new_init)
{
self.update_previous_epoch_participation(state)
.map_err(OpPoolError::RewardCacheUpdatePrevEpoch)?;
self.update_current_epoch_participation(state)
.map_err(OpPoolError::RewardCacheUpdateCurrEpoch)?;
self.initialization = Some(new_init);
}
Ok(())
}
fn update_previous_epoch_participation<E: EthSpec>(
&mut self,
state: &BeaconState<E>,
) -> Result<(), BeaconStateError> {
let default_participation = ParticipationFlags::default();
self.previous_epoch_participation = state
.previous_epoch_participation()?
.iter()
.map(|participation| *participation != default_participation)
.collect();
Ok(())
}
fn update_current_epoch_participation<E: EthSpec>(
&mut self,
state: &BeaconState<E>,
) -> Result<(), BeaconStateError> {
let default_participation = ParticipationFlags::default();
self.current_epoch_participation = state
.current_epoch_participation()?
.iter()
.map(|participation| *participation != default_participation)
.collect();
Ok(())
}
}

10
beacon_node/src/cli.rs
View File

@ -727,6 +727,16 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
.default_value("250")
.takes_value(true)
)
.arg(
Arg::with_name("paranoid-block-proposal")
.long("paranoid-block-proposal")
.help("Paranoid enough to be reading the source? Nice. This flag reverts some \
block proposal optimisations and forces the node to check every attestation \
it includes super thoroughly. This may be useful in an emergency, but not \
otherwise.")
.hidden(true)
.takes_value(false)
)
.arg(
Arg::with_name("builder-fallback-skips")
.long("builder-fallback-skips")

2
beacon_node/src/config.rs
View File

@ -644,6 +644,8 @@ pub fn get_config<E: EthSpec>(
client_config.chain.count_unrealized =
clap_utils::parse_required(cli_args, "count-unrealized")?;
client_config.chain.paranoid_block_proposal = cli_args.is_present("paranoid-block-proposal");
/*
* Builder fallback configs.
*/

2
beacon_node/store/src/metadata.rs
View File

@ -4,7 +4,7 @@ use ssz::{Decode, Encode};
use ssz_derive::{Decode, Encode};
use types::{Checkpoint, Hash256, Slot};
pub const CURRENT_SCHEMA_VERSION: SchemaVersion = SchemaVersion(11);
pub const CURRENT_SCHEMA_VERSION: SchemaVersion = SchemaVersion(12);
// All the keys that get stored under the `BeaconMeta` column.
//

1
book/src/database-migrations.md
View File

@ -23,6 +23,7 @@ validator client or the slasher**.
| v2.4.0 | Jul 2022 | v9 | yes (pre Bellatrix) |
| v2.5.0 | Aug 2022 | v11 | yes |
| v3.0.0 | Aug 2022 | v11 | yes |
| v3.1.0 | Sep 2022 | v12 | yes |
> **Note**: All point releases (e.g. v2.3.1) are schema-compatible with the prior minor release
> (e.g. v2.3.0).

2
consensus/state_processing/Cargo.toml
View File

@ -14,6 +14,7 @@ bls = { path = "../../crypto/bls" }
integer-sqrt = "0.1.5"
itertools = "0.10.0"
eth2_ssz = "0.4.1"
eth2_ssz_derive = "0.3.0"
eth2_ssz_types = "0.2.2"
merkle_proof = { path = "../merkle_proof" }
safe_arith = { path = "../safe_arith" }
@ -26,6 +27,7 @@ smallvec = "1.6.1"
arbitrary = { version = "1.0", features = ["derive"], optional = true }
lighthouse_metrics = { path = "../../common/lighthouse_metrics", optional = true }
lazy_static = { version = "1.4.0", optional = true }
derivative = "2.1.1"
[features]
default = ["legacy-arith", "metrics"]

15
consensus/state_processing/src/common/get_attesting_indices.rs
View File

@ -1,12 +1,10 @@
use types::*;
/// Returns validator indices which participated in the attestation, sorted by increasing index.
///
/// Spec v0.12.1
pub fn get_attesting_indices<T: EthSpec>(
committee: &[usize],
bitlist: &BitList<T::MaxValidatorsPerCommittee>,
) -> Result<Vec<usize>, BeaconStateError> {
) -> Result<Vec<u64>, BeaconStateError> {
if bitlist.len() != committee.len() {
return Err(BeaconStateError::InvalidBitfield);
}
@ -15,7 +13,7 @@ pub fn get_attesting_indices<T: EthSpec>(
for (i, validator_index) in committee.iter().enumerate() {
if let Ok(true) = bitlist.get(i) {
indices.push(*validator_index)
indices.push(*validator_index as u64)
}
}
@ -23,3 +21,12 @@ pub fn get_attesting_indices<T: EthSpec>(
Ok(indices)
}
/// Shortcut for getting the attesting indices while fetching the committee from the state's cache.
pub fn get_attesting_indices_from_state<T: EthSpec>(
state: &BeaconState<T>,
att: &Attestation<T>,
) -> Result<Vec<u64>, BeaconStateError> {
let committee = state.get_beacon_committee(att.data.slot, att.data.index)?;
get_attesting_indices::<T>(committee.committee, &att.aggregation_bits)
}

4
consensus/state_processing/src/common/get_indexed_attestation.rs
View File

@ -14,9 +14,7 @@ pub fn get_indexed_attestation<T: EthSpec>(
let attesting_indices = get_attesting_indices::<T>(committee, &attestation.aggregation_bits)?;
Ok(IndexedAttestation {
attesting_indices: VariableList::new(
attesting_indices.into_iter().map(|x| x as u64).collect(),
)?,
attesting_indices: VariableList::new(attesting_indices)?,
data: attestation.data.clone(),
signature: attestation.signature.clone(),
})

2
consensus/state_processing/src/common/mod.rs
View File

@ -10,7 +10,7 @@ pub mod base;
pub use deposit_data_tree::DepositDataTree;
pub use get_attestation_participation::get_attestation_participation_flag_indices;
pub use get_attesting_indices::get_attesting_indices;
pub use get_attesting_indices::{get_attesting_indices, get_attesting_indices_from_state};
pub use get_indexed_attestation::get_indexed_attestation;
pub use initiate_validator_exit::initiate_validator_exit;
pub use slash_validator::slash_validator;

4
consensus/state_processing/src/per_epoch_processing/base/validator_statuses.rs
View File

@ -278,8 +278,8 @@ impl ValidatorStatuses {
// Loop through the participating validator indices and update the status vec.
for validator_index in attesting_indices {
self.statuses
.get_mut(validator_index)
.ok_or(BeaconStateError::UnknownValidator(validator_index))?
.get_mut(validator_index as usize)
.ok_or(BeaconStateError::UnknownValidator(validator_index as usize))?
.update(&status);
}
}

4
consensus/state_processing/src/upgrade/altair.rs
View File

@ -32,8 +32,8 @@ pub fn translate_participation<E: EthSpec>(
for index in attesting_indices {
for flag_index in &participation_flag_indices {
epoch_participation
.get_mut(index)
.ok_or(Error::UnknownValidator(index))?
.get_mut(index as usize)
.ok_or(Error::UnknownValidator(index as usize))?
.add_flag(*flag_index)?;
}
}

137
consensus/state_processing/src/verify_operation.rs
View File

@ -5,36 +5,120 @@ use crate::per_block_processing::{
verify_attester_slashing, verify_exit, verify_proposer_slashing,
};
use crate::VerifySignatures;
use derivative::Derivative;
use smallvec::{smallvec, SmallVec};
use ssz::{Decode, Encode};
use ssz_derive::{Decode, Encode};
use std::marker::PhantomData;
use types::{
AttesterSlashing, BeaconState, ChainSpec, EthSpec, ProposerSlashing, SignedVoluntaryExit,
AttesterSlashing, BeaconState, ChainSpec, Epoch, EthSpec, Fork, ForkVersion, ProposerSlashing,
SignedVoluntaryExit,
};
const MAX_FORKS_VERIFIED_AGAINST: usize = 2;
/// Wrapper around an operation type that acts as proof that its signature has been checked.
///
/// The inner field is private, meaning instances of this type can only be constructed
/// The inner `op` field is private, meaning instances of this type can only be constructed
/// by calling `validate`.
#[derive(Debug, PartialEq, Eq, Clone)]
pub struct SigVerifiedOp<T>(T);
#[derive(Derivative, Debug, Clone, Encode, Decode)]
#[derivative(
PartialEq,
Eq,
Hash(bound = "T: Encode + Decode + std::hash::Hash, E: EthSpec")
)]
pub struct SigVerifiedOp<T: Encode + Decode, E: EthSpec> {
op: T,
verified_against: VerifiedAgainst,
#[ssz(skip_serializing, skip_deserializing)]
_phantom: PhantomData<E>,
}
/// Information about the fork versions that this message was verified against.
///
/// In general it is not safe to assume that a `SigVerifiedOp` constructed at some point in the past
/// will continue to be valid in the presence of a changing `state.fork()`. The reason for this
/// is that the fork versions that the message's epochs map to might change.
///
/// For example a proposer slashing at a phase0 slot verified against an Altair state will use
/// the phase0 fork version, but will become invalid once the Bellatrix fork occurs because that
/// slot will start to map to the Altair fork version. This is because `Fork::get_fork_version` only
/// remembers the most recent two forks.
///
/// In the other direction, a proposer slashing at a Bellatrix slot verified against an Altair state
/// will use the Altair fork version, but will become invalid once the Bellatrix fork occurs because
/// that slot will start to map to the Bellatrix fork version.
///
/// We need to store multiple `ForkVersion`s because attester slashings contain two indexed
/// attestations which may be signed using different versions.
#[derive(Debug, PartialEq, Eq, Clone, Hash, Encode, Decode)]
pub struct VerifiedAgainst {
fork_versions: SmallVec<[ForkVersion; MAX_FORKS_VERIFIED_AGAINST]>,
}
impl<T, E> SigVerifiedOp<T, E>
where
T: VerifyOperation<E>,
E: EthSpec,
{
/// This function must be private because it assumes that `op` has already been verified.
fn new(op: T, state: &BeaconState<E>) -> Self {
let verified_against = VerifiedAgainst {
fork_versions: op
.verification_epochs()
.into_iter()
.map(|epoch| state.fork().get_fork_version(epoch))
.collect(),
};
SigVerifiedOp {
op,
verified_against,
_phantom: PhantomData,
}
}
impl<T> SigVerifiedOp<T> {
pub fn into_inner(self) -> T {
self.0
self.op
}
pub fn as_inner(&self) -> &T {
&self.0
&self.op
}
pub fn signature_is_still_valid(&self, current_fork: &Fork) -> bool {
self.as_inner()
.verification_epochs()
.into_iter()
.zip(self.verified_against.fork_versions.iter())
.all(|(epoch, verified_fork_version)| {
current_fork.get_fork_version(epoch) == *verified_fork_version
})
}
/// Return one of the fork versions this message was verified against.
///
/// This is only required for the v12 schema downgrade and can be deleted once all nodes
/// are upgraded to v12.
pub fn first_fork_verified_against(&self) -> Option<ForkVersion> {
self.verified_against.fork_versions.first().copied()
}
}
/// Trait for operations that can be verified and transformed into a `SigVerifiedOp`.
pub trait VerifyOperation<E: EthSpec>: Sized {
pub trait VerifyOperation<E: EthSpec>: Encode + Decode + Sized {
type Error;
fn validate(
self,
state: &BeaconState<E>,
spec: &ChainSpec,
) -> Result<SigVerifiedOp<Self>, Self::Error>;
) -> Result<SigVerifiedOp<Self, E>, Self::Error>;
/// Return the epochs at which parts of this message were verified.
///
/// These need to map 1-to-1 to the `SigVerifiedOp::verified_against` for this type.
fn verification_epochs(&self) -> SmallVec<[Epoch; MAX_FORKS_VERIFIED_AGAINST]>;
}
impl<E: EthSpec> VerifyOperation<E> for SignedVoluntaryExit {
@ -44,9 +128,14 @@ impl<E: EthSpec> VerifyOperation<E> for SignedVoluntaryExit {
self,
state: &BeaconState<E>,
spec: &ChainSpec,
) -> Result<SigVerifiedOp<Self>, Self::Error> {
) -> Result<SigVerifiedOp<Self, E>, Self::Error> {
verify_exit(state, &self, VerifySignatures::True, spec)?;
Ok(SigVerifiedOp(self))
Ok(SigVerifiedOp::new(self, state))
}
#[allow(clippy::integer_arithmetic)]
fn verification_epochs(&self) -> SmallVec<[Epoch; MAX_FORKS_VERIFIED_AGAINST]> {
smallvec![self.message.epoch]
}
}
@ -57,9 +146,17 @@ impl<E: EthSpec> VerifyOperation<E> for AttesterSlashing<E> {
self,
state: &BeaconState<E>,
spec: &ChainSpec,
) -> Result<SigVerifiedOp<Self>, Self::Error> {
) -> Result<SigVerifiedOp<Self, E>, Self::Error> {
verify_attester_slashing(state, &self, VerifySignatures::True, spec)?;
Ok(SigVerifiedOp(self))
Ok(SigVerifiedOp::new(self, state))
}
#[allow(clippy::integer_arithmetic)]
fn verification_epochs(&self) -> SmallVec<[Epoch; MAX_FORKS_VERIFIED_AGAINST]> {
smallvec![
self.attestation_1.data.target.epoch,
self.attestation_2.data.target.epoch
]
}
}
@ -70,8 +167,18 @@ impl<E: EthSpec> VerifyOperation<E> for ProposerSlashing {
self,
state: &BeaconState<E>,
spec: &ChainSpec,
) -> Result<SigVerifiedOp<Self>, Self::Error> {
) -> Result<SigVerifiedOp<Self, E>, Self::Error> {
verify_proposer_slashing(&self, state, VerifySignatures::True, spec)?;
Ok(SigVerifiedOp(self))
Ok(SigVerifiedOp::new(self, state))
}
#[allow(clippy::integer_arithmetic)]
fn verification_epochs(&self) -> SmallVec<[Epoch; MAX_FORKS_VERIFIED_AGAINST]> {
// Only need a single epoch because the slots of the two headers must be equal.
smallvec![self
.signed_header_1
.message
.slot
.epoch(E::slots_per_epoch())]
}
}

7
consensus/types/src/proposer_slashing.rs
View File

@ -18,6 +18,13 @@ pub struct ProposerSlashing {
pub signed_header_2: SignedBeaconBlockHeader,
}
impl ProposerSlashing {
/// Get proposer index, assuming slashing validity has already been checked.
pub fn proposer_index(&self) -> u64 {
self.signed_header_1.message.proposer_index
}
}
#[cfg(test)]
mod tests {
use super::*;

15
lighthouse/tests/beacon_node.rs
View File

@ -132,6 +132,21 @@ fn fork_choice_before_proposal_timeout_zero() {
.with_config(|config| assert_eq!(config.chain.fork_choice_before_proposal_timeout_ms, 0));
}
#[test]
fn paranoid_block_proposal_default() {
CommandLineTest::new()
.run_with_zero_port()
.with_config(|config| assert!(!config.chain.paranoid_block_proposal));
}
#[test]
fn paranoid_block_proposal_on() {
CommandLineTest::new()
.flag("paranoid-block-proposal", None)
.run_with_zero_port()
.with_config(|config| assert!(config.chain.paranoid_block_proposal));
}
#[test]
fn count_unrealized_default() {
CommandLineTest::new()