Devnet6 (#4404)

* some blob reprocessing work * remove ForceBlockLookup * reorder enum match arms in sync manager * a lot more reprocessing work * impl logic for triggerng blob lookups along with block lookups * deal with rpc blobs in groups per block in the da checker. don't cache missing blob ids in the da checker. * make single block lookup generic * more work * add delayed processing logic and combine some requests * start fixing some compile errors * fix compilation in main block lookup mod * much work * get things compiling * parent blob lookups * fix compile * revert red/stevie changes * fix up sync manager delay message logic * add peer usefulness enum * should remove lookup refactor * consolidate retry error handling * improve peer scoring during certain failures in parent lookups * improve retry code * drop parent lookup if either req has a peer disconnect during download * refactor single block processed method * processing peer refactor * smol bugfix * fix some todos * fix lints * fix lints * fix compile in lookup tests * fix lints * fix lints * fix existing block lookup tests * renamings * fix after merge * cargo fmt * compilation fix in beacon chain tests * fix * refactor lookup tests to work with multiple forks and response types * make tests into macros * wrap availability check error * fix compile after merge * add random blobs * start fixing up lookup verify error handling * some bug fixes and the start of deneb only tests * make tests work for all forks * track information about peer source * error refactoring * improve peer scoring * fix test compilation * make sure blobs are sent for processing after stream termination, delete copied tests * add some tests and fix a bug * smol bugfixes and moar tests * add tests and fix some things * compile after merge * lots of refactoring * retry on invalid block/blob * merge unknown parent messages before current slot lookup * get tests compiling * penalize blob peer on invalid blobs * Check disk on in-memory cache miss * Update beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs * Update beacon_node/network/src/sync/network_context.rs Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com> * fix bug in matching blocks and blobs in range sync * pr feedback * fix conflicts * upgrade logs from warn to crit when we receive incorrect response in range * synced_and_connected_within_tolerance -> should_search_for_block * remove todo * add data gas used and update excess data gas to u64 * Fix Broken Overflow Tests * payload verification with commitments * fix merge conflicts * restore payload file * Restore payload file * remove todo * add max blob commitments per block * c-kzg lib update * Fix ef tests * Abstract over minimal/mainnet spec in kzg crate * Start integrating new KZG * checkpoint sync without alignment * checkpoint sync without alignment * add import * add import * query for checkpoint state by slot rather than state root (teku doesn't serve by state root) * query for checkpoint state by slot rather than state root (teku doesn't serve by state root) * loosen check * get state first and query by most recent block root * Revert "loosen check" This reverts commit 069d13dd63aa794a3505db9f17bd1a6b73f0be81. * get state first and query by most recent block root * merge max blobs change * simplify delay logic * rename unknown parent sync message variants * rename parameter, block_slot -> slot * add some docs to the lookup module * use interval instead of sleep * drop request if blocks and blobs requests both return `None` for `Id` * clean up `find_single_lookup` logic * add lookup source enum * clean up `find_single_lookup` logic * add docs to find_single_lookup_request * move LookupSource our of param where unnecessary * remove unnecessary todo * query for block by `state.latest_block_header.slot` * fix lint * fix merge transition ef tests * fix test * fix test * fix observed blob sidecars test * Add some metrics (#33) * fix protocol limits for blobs by root * Update Engine API for 1:1 Structure Method * make beacon chain tests to fix devnet 6 changes * get ckzg working and fix some tests * fix remaining tests * fix lints * Fix KZG linking issues * remove unused dep * lockfile * test fixes * remove dbgs * remove unwrap * cleanup tx generator * small fixes * fixing fixes * more self reivew * more self review * refactor genesis header initialization * refactor mock el instantiations * fix compile * fix network test, make sure they run for each fork * pr feedback * fix last test (hopefully) --------- Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com> Co-authored-by: Mark Mackey <mark@sigmaprime.io> Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com> Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-06-29 15:35:43 -04:00
parent 4c9fcf1e83
commit adbb62f7f3
69 changed files with 2114 additions and 1338 deletions
@@ -13,7 +13,6 @@ pub use self::verify_attester_slashing::{
 pub use self::verify_proposer_slashing::verify_proposer_slashing;
 pub use altair::sync_committee::process_sync_aggregate;
 pub use block_signature_verifier::{BlockSignatureVerifier, ParallelSignatureSets};
-pub use deneb::deneb::process_blob_kzg_commitments;
 pub use is_valid_indexed_attestation::is_valid_indexed_attestation;
 pub use process_operations::process_operations;
 pub use verify_attestation::{
@@ -163,11 +162,11 @@ pub fn per_block_processing<T: EthSpec, Payload: AbstractExecPayload<T>>(
    // `process_randao` as the former depends on the `randao_mix` computed with the reveal of the
    // previous block.
    if is_execution_enabled(state, block.body()) {
-        let payload = block.body().execution_payload()?;
+        let body = block.body();
        if state_processing_strategy == StateProcessingStrategy::Accurate {
-            process_withdrawals::<T, Payload>(state, payload, spec)?;
+            process_withdrawals::<T, Payload>(state, body.execution_payload()?, spec)?;
        }
-        process_execution_payload::<T, Payload>(state, payload, spec)?;
+        process_execution_payload::<T, Payload>(state, body, spec)?;
    }

    process_randao(state, block, verify_randao, ctxt, spec)?;
@@ -184,8 +183,6 @@ pub fn per_block_processing<T: EthSpec, Payload: AbstractExecPayload<T>>(
        )?;
    }

-    process_blob_kzg_commitments(block.body(), ctxt)?;
-
    Ok(())
 }

@@ -350,9 +347,10 @@ pub fn get_new_eth1_data<T: EthSpec>(
 pub fn partially_verify_execution_payload<T: EthSpec, Payload: AbstractExecPayload<T>>(
    state: &BeaconState<T>,
    block_slot: Slot,
-    payload: Payload::Ref<'_>,
+    body: BeaconBlockBodyRef<T, Payload>,
    spec: &ChainSpec,
 ) -> Result<(), BlockProcessingError> {
+    let payload = body.execution_payload()?;
    if is_merge_transition_complete(state) {
        block_verify!(
            payload.parent_hash() == state.latest_execution_payload_header()?.block_hash(),
@@ -379,6 +377,17 @@ pub fn partially_verify_execution_payload<T: EthSpec, Payload: AbstractExecPaylo
        }
    );

+    if let Ok(blob_commitments) = body.blob_kzg_commitments() {
+        // Verify commitments are under the limit.
+        block_verify!(
+            blob_commitments.len() <= T::max_blobs_per_block(),
+            BlockProcessingError::ExecutionInvalidBlobsLen {
+                max: T::max_blobs_per_block(),
+                actual: blob_commitments.len(),
+            }
+        );
+    }
+
    Ok(())
 }

@@ -391,11 +400,11 @@ pub fn partially_verify_execution_payload<T: EthSpec, Payload: AbstractExecPaylo
 /// https://github.com/ethereum/consensus-specs/blob/v1.1.5/specs/merge/beacon-chain.md#process_execution_payload
 pub fn process_execution_payload<T: EthSpec, Payload: AbstractExecPayload<T>>(
    state: &mut BeaconState<T>,
-    payload: Payload::Ref<'_>,
+    body: BeaconBlockBodyRef<T, Payload>,
    spec: &ChainSpec,
 ) -> Result<(), BlockProcessingError> {
-    partially_verify_execution_payload::<T, Payload>(state, state.slot(), payload, spec)?;
-
+    partially_verify_execution_payload::<T, Payload>(state, state.slot(), body, spec)?;
+    let payload = body.execution_payload()?;
    match state.latest_execution_payload_header_mut()? {
        ExecutionPayloadHeaderRefMut::Merge(header_mut) => {
            match payload.to_execution_payload_header() {
@@ -423,15 +432,19 @@ pub fn process_execution_payload<T: EthSpec, Payload: AbstractExecPayload<T>>(
 /// These functions will definitely be called before the merge. Their entire purpose is to check if
 /// the merge has happened or if we're on the transition block. Thus we don't want to propagate
 /// errors from the `BeaconState` being an earlier variant than `BeaconStateMerge` as we'd have to
-/// repeaetedly write code to treat these errors as false.
+/// repeatedly write code to treat these errors as false.
 /// https://github.com/ethereum/consensus-specs/blob/dev/specs/bellatrix/beacon-chain.md#is_merge_transition_complete
 pub fn is_merge_transition_complete<T: EthSpec>(state: &BeaconState<T>) -> bool {
-    // We must check defaultness against the payload header with 0x0 roots, as that's what's meant
-    // by `ExecutionPayloadHeader()` in the spec.
-    state
-        .latest_execution_payload_header()
-        .map(|header| !header.is_default_with_zero_roots())
-        .unwrap_or(false)
+    match state {
+        // We must check defaultness against the payload header with 0x0 roots, as that's what's meant
+        // by `ExecutionPayloadHeader()` in the spec.
+        BeaconState::Merge(_) => state
+            .latest_execution_payload_header()
+            .map(|header| !header.is_default_with_zero_roots())
+            .unwrap_or(false),
+        BeaconState::Deneb(_) | BeaconState::Capella(_) => true,
+        BeaconState::Base(_) | BeaconState::Altair(_) => false,
+    }
 }
 /// https://github.com/ethereum/consensus-specs/blob/dev/specs/bellatrix/beacon-chain.md#is_merge_transition_block
 pub fn is_merge_transition_block<T: EthSpec, Payload: AbstractExecPayload<T>>(
@@ -1,125 +1,8 @@
-use crate::{BlockProcessingError, ConsensusContext};
 use ethereum_hashing::hash_fixed;
-use itertools::{EitherOrBoth, Itertools};
-use safe_arith::SafeArith;
-use ssz::Decode;
-use types::consts::deneb::{BLOB_TX_TYPE, VERSIONED_HASH_VERSION_KZG};
-use types::{
-    AbstractExecPayload, BeaconBlockBodyRef, EthSpec, ExecPayload, KzgCommitment, Transaction,
-    Transactions, VersionedHash,
-};
+use types::consts::deneb::VERSIONED_HASH_VERSION_KZG;
+use types::{KzgCommitment, VersionedHash};

-pub fn process_blob_kzg_commitments<T: EthSpec, Payload: AbstractExecPayload<T>>(
-    block_body: BeaconBlockBodyRef<T, Payload>,
-    ctxt: &mut ConsensusContext<T>,
-) -> Result<(), BlockProcessingError> {
-    // Return early if this check has already been run.
-    if ctxt.kzg_commitments_consistent() {
-        return Ok(());
-    }
-    if let (Ok(payload), Ok(kzg_commitments)) = (
-        block_body.execution_payload(),
-        block_body.blob_kzg_commitments(),
-    ) {
-        if let Some(transactions) = payload.transactions() {
-            if !verify_kzg_commitments_against_transactions::<T>(transactions, kzg_commitments)? {
-                return Err(BlockProcessingError::BlobVersionHashMismatch);
-            }
-        }
-    }
-
-    Ok(())
-}
-
-pub fn verify_kzg_commitments_against_transactions<T: EthSpec>(
-    transactions: &Transactions<T>,
-    kzg_commitments: &[KzgCommitment],
-) -> Result<bool, BlockProcessingError> {
-    let nested_iter = transactions
-        .into_iter()
-        .filter(|tx| {
-            tx.first()
-                .map(|tx_type| *tx_type == BLOB_TX_TYPE)
-                .unwrap_or(false)
-        })
-        .map(|tx| tx_peek_versioned_hashes::<T>(tx));
-
-    itertools::process_results(nested_iter, |iter| {
-        let zipped_iter = iter
-            .flatten()
-            // Need to use `itertools::zip_longest` here because just zipping hides if one iter is shorter
-            // and `itertools::zip_eq` panics.
-            .zip_longest(kzg_commitments.iter())
-            .enumerate()
-            .map(|(index, next)| match next {
-                EitherOrBoth::Both(hash, commitment) => Ok((hash?, commitment)),
-                // The number of versioned hashes from the blob transactions exceeds the number of
-                // commitments in the block.
-                EitherOrBoth::Left(_) => Err(BlockProcessingError::BlobNumCommitmentsMismatch {
-                    commitments_processed_in_block: index,
-                    commitments_processed_in_transactions: index.safe_add(1)?,
-                }),
-                // The number of commitments in the block exceeds the number of versioned hashes
-                // in the blob transactions.
-                EitherOrBoth::Right(_) => Err(BlockProcessingError::BlobNumCommitmentsMismatch {
-                    commitments_processed_in_block: index.safe_add(1)?,
-                    commitments_processed_in_transactions: index,
-                }),
-            });
-
-        itertools::process_results(zipped_iter, |mut iter| {
-            iter.all(|(tx_versioned_hash, commitment)| {
-                tx_versioned_hash == kzg_commitment_to_versioned_hash(commitment)
-            })
-        })
-    })?
-}
-
-/// Only transactions of type `BLOB_TX_TYPE` should be passed into this function.
-fn tx_peek_versioned_hashes<T: EthSpec>(
-    opaque_tx: &Transaction<T::MaxBytesPerTransaction>,
-) -> Result<
-    impl IntoIterator<Item = Result<VersionedHash, BlockProcessingError>> + '_,
-    BlockProcessingError,
-> {
-    let tx_len = opaque_tx.len();
-    let message_offset = 1.safe_add(u32::from_ssz_bytes(opaque_tx.get(1..5).ok_or(
-        BlockProcessingError::BlobVersionHashIndexOutOfBounds {
-            length: tx_len,
-            index: 5,
-        },
-    )?)?)?;
-
-    let message_offset_usize = message_offset as usize;
-
-    // field offset: 32 + 8 + 32 + 32 + 8 + 4 + 32 + 4 + 4 + 32 = 188
-    let versioned_hashes_offset = message_offset.safe_add(u32::from_ssz_bytes(
-        opaque_tx
-            .get(message_offset_usize.safe_add(188)?..message_offset_usize.safe_add(192)?)
-            .ok_or(BlockProcessingError::BlobVersionHashIndexOutOfBounds {
-                length: tx_len,
-                index: message_offset_usize.safe_add(192)?,
-            })?,
-    )?)?;
-
-    let num_hashes = tx_len
-        .safe_sub(versioned_hashes_offset as usize)?
-        .safe_div(32)?;
-
-    Ok((0..num_hashes).map(move |i| {
-        let next_version_hash_index =
-            (versioned_hashes_offset as usize).safe_add(i.safe_mul(32)?)?;
-        let bytes = opaque_tx
-            .get(next_version_hash_index..next_version_hash_index.safe_add(32)?)
-            .ok_or(BlockProcessingError::BlobVersionHashIndexOutOfBounds {
-                length: tx_len,
-                index: (next_version_hash_index).safe_add(32)?,
-            })?;
-        Ok(VersionedHash::from_slice(bytes))
-    }))
-}
-
-fn kzg_commitment_to_versioned_hash(kzg_commitment: &KzgCommitment) -> VersionedHash {
+pub fn kzg_commitment_to_versioned_hash(kzg_commitment: &KzgCommitment) -> VersionedHash {
    let mut hashed_commitment = hash_fixed(&kzg_commitment.0);
    hashed_commitment[0] = VERSIONED_HASH_VERSION_KZG;
    VersionedHash::from(hashed_commitment)
@@ -76,6 +76,10 @@ pub enum BlockProcessingError {
        expected: u64,
        found: u64,
    },
+    ExecutionInvalidBlobsLen {
+        max: usize,
+        actual: usize,
+    },
    ExecutionInvalid,
    ConsensusContext(ContextError),
    WithdrawalsRootMismatch {