Devnet6 (#4404)

* some blob reprocessing work

* remove ForceBlockLookup

* reorder enum match arms in sync manager

* a lot more reprocessing work

* impl logic for triggerng blob lookups along with block lookups

* deal with rpc blobs in groups per block in the da checker. don't cache missing blob ids in the da checker.

* make single block lookup generic

* more work

* add delayed processing logic and combine some requests

* start fixing some compile errors

* fix compilation in main block lookup mod

* much work

* get things compiling

* parent blob lookups

* fix compile

* revert red/stevie changes

* fix up sync manager delay message logic

* add peer usefulness enum

* should remove lookup refactor

* consolidate retry error handling

* improve peer scoring during certain failures in parent lookups

* improve retry code

* drop parent lookup if either req has a peer disconnect during download

* refactor single block processed method

* processing peer refactor

* smol bugfix

* fix some todos

* fix lints

* fix lints

* fix compile in lookup tests

* fix lints

* fix lints

* fix existing block lookup tests

* renamings

* fix after merge

* cargo fmt

* compilation fix in beacon chain tests

* fix

* refactor lookup tests to work with multiple forks and response types

* make tests into macros

* wrap availability check error

* fix compile after merge

* add random blobs

* start fixing up lookup verify error handling

* some bug fixes and the start of deneb only tests

* make tests work for all forks

* track information about peer source

* error refactoring

* improve peer scoring

* fix test compilation

* make sure blobs are sent for processing after stream termination, delete copied tests

* add some tests and fix a bug

* smol bugfixes and moar tests

* add tests and fix some things

* compile after merge

* lots of refactoring

* retry on invalid block/blob

* merge unknown parent messages before current slot lookup

* get tests compiling

* penalize blob peer on invalid blobs

* Check disk on in-memory cache miss

* Update beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

* Update beacon_node/network/src/sync/network_context.rs

Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>

* fix bug in matching blocks and blobs in range sync

* pr feedback

* fix conflicts

* upgrade logs from warn to crit when we receive incorrect response in range

* synced_and_connected_within_tolerance -> should_search_for_block

* remove todo

* add data gas used and update excess data gas to u64

* Fix Broken Overflow Tests

* payload verification with commitments

* fix merge conflicts

* restore payload file

* Restore payload file

* remove todo

* add max blob commitments per block

* c-kzg lib update

* Fix ef tests

* Abstract over minimal/mainnet spec in kzg crate

* Start integrating new KZG

* checkpoint sync without alignment

* checkpoint sync without alignment

* add import

* add import

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* loosen check

* get state first and query by most recent block root

* Revert "loosen check"

This reverts commit 069d13dd63aa794a3505db9f17bd1a6b73f0be81.

* get state first and query by most recent block root

* merge max blobs change

* simplify delay logic

* rename unknown parent sync message variants

* rename parameter, block_slot -> slot

* add some docs to the lookup module

* use interval instead of sleep

* drop request if blocks and blobs requests both return `None` for `Id`

* clean up `find_single_lookup` logic

* add lookup source enum

* clean up `find_single_lookup` logic

* add docs to find_single_lookup_request

* move LookupSource our of param where unnecessary

* remove unnecessary todo

* query for block by `state.latest_block_header.slot`

* fix lint

* fix merge transition ef tests

* fix test

* fix test

* fix observed  blob sidecars test

* Add some metrics (#33)

* fix protocol limits for blobs by root

* Update Engine API for 1:1 Structure Method

* make beacon chain tests to fix devnet 6 changes

* get ckzg working and fix some tests

* fix remaining tests

* fix lints

* Fix KZG linking issues

* remove unused dep

* lockfile

* test fixes

* remove dbgs

* remove unwrap

* cleanup tx generator

* small fixes

* fixing fixes

* more self reivew

* more self review

* refactor genesis header initialization

* refactor mock el instantiations

* fix compile

* fix network test, make sure they run for each fork

* pr feedback

* fix last test (hopefully)

---------

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>

testing/ef_tests/Makefile

@@ -1,4 +1,4 @@
-TESTS_TAG := v1.3.0
+TESTS_TAG := v1.4.0-alpha.2
 TESTS = general minimal mainnet
 TARBALLS = $(patsubst %,%-$(TESTS_TAG).tar.gz,$(TESTS))
 

testing/ef_tests/check_all_files_accessed.py

@@ -54,7 +54,7 @@ excluded_paths = [
     # FIXME(sean)
     "tests/mainnet/capella/light_client/single_merkle_proof/BeaconBlockBody/*",
     "tests/mainnet/deneb/light_client/single_merkle_proof/BeaconBlockBody/*",
-    "tests/general/deneb/kzg"
+    "tests/.*/eip6110"
 ]
 
 

testing/ef_tests/src/cases/kzg_blob_to_kzg_commitment.rs

@@ -31,7 +31,7 @@ impl<E: EthSpec> Case for KZGBlobToKZGCommitment<E> {
     }
 
     fn result(&self, _case_index: usize, _fork_name: ForkName) -> Result<(), Error> {
-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;
 
         let commitment = parse_blob::<E>(&self.input.blob).and_then(|blob| {
             blob_to_kzg_commitment::<E>(&kzg, blob).map_err(|e| {

testing/ef_tests/src/cases/kzg_compute_blob_kzg_proof.rs

@@ -38,7 +38,7 @@ impl<E: EthSpec> Case for KZGComputeBlobKZGProof<E> {
             Ok((blob, commitment))
         };
 
-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;
         let proof = parse_input(&self.input).and_then(|(blob, commitment)| {
             compute_blob_kzg_proof::<E>(&kzg, &blob, commitment)
                 .map_err(|e| Error::InternalError(format!("Failed to compute kzg proof: {:?}", e)))

testing/ef_tests/src/cases/kzg_compute_kzg_proof.rs

@@ -45,7 +45,7 @@ impl<E: EthSpec> Case for KZGComputeKZGProof<E> {
             Ok((blob, z))
         };
 
-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;
         let proof = parse_input(&self.input).and_then(|(blob, z)| {
             compute_kzg_proof::<E>(&kzg, blob, z)
                 .map_err(|e| Error::InternalError(format!("Failed to compute kzg proof: {:?}", e)))

testing/ef_tests/src/cases/kzg_verify_blob_kzg_proof.rs

@@ -1,15 +1,15 @@
 use super::*;
 use crate::case_result::compare_result;
 use beacon_chain::kzg_utils::validate_blob;
-use eth2_network_config::TRUSTED_SETUP;
-use kzg::{Kzg, KzgCommitment, KzgProof, TrustedSetup};
+use eth2_network_config::get_trusted_setup;
+use kzg::{Kzg, KzgCommitment, KzgPreset, KzgProof, TrustedSetup};
 use serde_derive::Deserialize;
 use std::convert::TryInto;
 use std::marker::PhantomData;
 use types::Blob;
 
-pub fn get_kzg() -> Result<Kzg, Error> {
-    let trusted_setup: TrustedSetup = serde_json::from_reader(TRUSTED_SETUP)
+pub fn get_kzg<P: KzgPreset>() -> Result<Kzg<P>, Error> {
+    let trusted_setup: TrustedSetup = serde_json::from_reader(get_trusted_setup::<P>())
         .map_err(|e| Error::InternalError(format!("Failed to initialize kzg: {:?}", e)))?;
     Kzg::new_from_trusted_setup(trusted_setup)
         .map_err(|e| Error::InternalError(format!("Failed to initialize kzg: {:?}", e)))
@@ -81,7 +81,7 @@ impl<E: EthSpec> Case for KZGVerifyBlobKZGProof<E> {
             Ok((blob, commitment, proof))
         };
 
-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;
         let result = parse_input(&self.input).and_then(|(blob, commitment, proof)| {
             validate_blob::<E>(&kzg, blob, commitment, proof)
                 .map_err(|e| Error::InternalError(format!("Failed to validate blob: {:?}", e)))

testing/ef_tests/src/cases/kzg_verify_blob_kzg_proof_batch.rs

@@ -51,7 +51,7 @@ impl<E: EthSpec> Case for KZGVerifyBlobKZGProofBatch<E> {
             Ok((commitments, blobs, proofs))
         };
 
-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;
         let result = parse_input(&self.input).and_then(|(commitments, blobs, proofs)| {
             validate_blobs::<E>(&kzg, &commitments, &blobs, &proofs)
                 .map_err(|e| Error::InternalError(format!("Failed to validate blobs: {:?}", e)))

testing/ef_tests/src/cases/kzg_verify_kzg_proof.rs

@@ -41,7 +41,7 @@ impl<E: EthSpec> Case for KZGVerifyKZGProof<E> {
             Ok((commitment, z, y, proof))
         };
 
-        let kzg = get_kzg()?;
+        let kzg = get_kzg::<E::Kzg>()?;
         let result = parse_input(&self.input).and_then(|(commitment, z, y, proof)| {
             verify_kzg_proof::<E>(&kzg, commitment, proof, z, y)
                 .map_err(|e| Error::InternalError(format!("Failed to validate proof: {:?}", e)))

testing/ef_tests/src/cases/operations.rs

@@ -4,6 +4,7 @@ use crate::case_result::compare_beacon_state_results_without_caches;
 use crate::decode::{ssz_decode_file, ssz_decode_file_with, ssz_decode_state, yaml_decode_file};
 use crate::testing_spec;
 use serde_derive::Deserialize;
+use ssz::Decode;
 use state_processing::{
     per_block_processing::{
         errors::BlockProcessingError,
@@ -19,7 +20,8 @@ use state_processing::{
 use std::fmt::Debug;
 use std::path::Path;
 use types::{
-    Attestation, AttesterSlashing, BeaconBlock, BeaconState, BlindedPayload, ChainSpec, Deposit,
+    Attestation, AttesterSlashing, BeaconBlock, BeaconBlockBody, BeaconBlockBodyCapella,
+    BeaconBlockBodyDeneb, BeaconBlockBodyMerge, BeaconState, BlindedPayload, ChainSpec, Deposit,
     EthSpec, ExecutionPayload, ForkName, FullPayload, ProposerSlashing, SignedBlsToExecutionChange,
     SignedVoluntaryExit, SyncAggregate,
 };
@@ -259,13 +261,13 @@ impl<E: EthSpec> Operation<E> for SyncAggregate<E> {
     }
 }
 
-impl<E: EthSpec> Operation<E> for FullPayload<E> {
+impl<E: EthSpec> Operation<E> for BeaconBlockBody<E, FullPayload<E>> {
     fn handler_name() -> String {
         "execution_payload".into()
     }
 
     fn filename() -> String {
-        "execution_payload.ssz_snappy".into()
+        "body.ssz_snappy".into()
     }
 
     fn is_enabled_for_fork(fork_name: ForkName) -> bool {
@@ -274,9 +276,13 @@ impl<E: EthSpec> Operation<E> for FullPayload<E> {
 
     fn decode(path: &Path, fork_name: ForkName, _spec: &ChainSpec) -> Result<Self, Error> {
         ssz_decode_file_with(path, |bytes| {
-            ExecutionPayload::from_ssz_bytes(bytes, fork_name)
+            Ok(match fork_name {
+                ForkName::Merge => BeaconBlockBody::Merge(<_>::from_ssz_bytes(bytes)?),
+                ForkName::Capella => BeaconBlockBody::Capella(<_>::from_ssz_bytes(bytes)?),
+                ForkName::Deneb => BeaconBlockBody::Deneb(<_>::from_ssz_bytes(bytes)?),
+                _ => panic!(),
+            })
         })
-        .map(Into::into)
     }
 
     fn apply_to(
@@ -296,13 +302,13 @@ impl<E: EthSpec> Operation<E> for FullPayload<E> {
         }
     }
 }
-impl<E: EthSpec> Operation<E> for BlindedPayload<E> {
+impl<E: EthSpec> Operation<E> for BeaconBlockBody<E, BlindedPayload<E>> {
     fn handler_name() -> String {
         "execution_payload".into()
     }
 
     fn filename() -> String {
-        "execution_payload.ssz_snappy".into()
+        "body.ssz_snappy".into()
     }
 
     fn is_enabled_for_fork(fork_name: ForkName) -> bool {
@@ -311,9 +317,22 @@ impl<E: EthSpec> Operation<E> for BlindedPayload<E> {
 
     fn decode(path: &Path, fork_name: ForkName, _spec: &ChainSpec) -> Result<Self, Error> {
         ssz_decode_file_with(path, |bytes| {
-            ExecutionPayload::from_ssz_bytes(bytes, fork_name)
+            Ok(match fork_name {
+                ForkName::Merge => {
+                    let inner = <BeaconBlockBodyMerge<E, FullPayload<E>>>::from_ssz_bytes(bytes)?;
+                    BeaconBlockBody::Merge(inner.clone_as_blinded())
+                }
+                ForkName::Capella => {
+                    let inner = <BeaconBlockBodyCapella<E, FullPayload<E>>>::from_ssz_bytes(bytes)?;
+                    BeaconBlockBody::Capella(inner.clone_as_blinded())
+                }
+                ForkName::Deneb => {
+                    let inner = <BeaconBlockBodyDeneb<E, FullPayload<E>>>::from_ssz_bytes(bytes)?;
+                    BeaconBlockBody::Deneb(inner.clone_as_blinded())
+                }
+                _ => panic!(),
+            })
         })
-        .map(Into::into)
     }
 
     fn apply_to(

testing/ef_tests/tests/tests.rs

@@ -72,14 +72,14 @@ fn operations_sync_aggregate() {
 
 #[test]
 fn operations_execution_payload_full() {
-    OperationsHandler::<MinimalEthSpec, FullPayload<_>>::default().run();
-    OperationsHandler::<MainnetEthSpec, FullPayload<_>>::default().run();
+    OperationsHandler::<MinimalEthSpec, BeaconBlockBody<_, FullPayload<_>>>::default().run();
+    OperationsHandler::<MainnetEthSpec, BeaconBlockBody<_, FullPayload<_>>>::default().run();
 }
 
 #[test]
 fn operations_execution_payload_blinded() {
-    OperationsHandler::<MinimalEthSpec, BlindedPayload<_>>::default().run();
-    OperationsHandler::<MainnetEthSpec, BlindedPayload<_>>::default().run();
+    OperationsHandler::<MinimalEthSpec, BeaconBlockBody<_, BlindedPayload<_>>>::default().run();
+    OperationsHandler::<MainnetEthSpec, BeaconBlockBody<_, BlindedPayload<_>>>::default().run();
 }
 
 #[test]

testing/execution_engine_integration/src/test_rig.rs

@@ -371,7 +371,7 @@ impl<E: GenericExecutionEngine> TestRig<E> {
         let status = self
             .ee_a
             .execution_layer
-            .notify_new_payload(&valid_payload)
+            .notify_new_payload(&valid_payload, None)
             .await
             .unwrap();
         assert_eq!(status, PayloadStatus::Valid);
@@ -424,7 +424,7 @@ impl<E: GenericExecutionEngine> TestRig<E> {
         let status = self
             .ee_a
             .execution_layer
-            .notify_new_payload(&invalid_payload)
+            .notify_new_payload(&invalid_payload, None)
             .await
             .unwrap();
         assert!(matches!(
@@ -486,7 +486,7 @@ impl<E: GenericExecutionEngine> TestRig<E> {
         let status = self
             .ee_a
             .execution_layer
-            .notify_new_payload(&second_payload)
+            .notify_new_payload(&second_payload, None)
             .await
             .unwrap();
         assert_eq!(status, PayloadStatus::Valid);
@@ -533,7 +533,7 @@ impl<E: GenericExecutionEngine> TestRig<E> {
         let status = self
             .ee_b
             .execution_layer
-            .notify_new_payload(&second_payload)
+            .notify_new_payload(&second_payload, None)
             .await
             .unwrap();
         // TODO: we should remove the `Accepted` status here once Geth fixes it
@@ -574,7 +574,7 @@ impl<E: GenericExecutionEngine> TestRig<E> {
         let status = self
             .ee_b
             .execution_layer
-            .notify_new_payload(&valid_payload)
+            .notify_new_payload(&valid_payload, None)
             .await
             .unwrap();
         assert_eq!(status, PayloadStatus::Valid);
@@ -588,7 +588,7 @@ impl<E: GenericExecutionEngine> TestRig<E> {
         let status = self
             .ee_b
             .execution_layer
-            .notify_new_payload(&second_payload)
+            .notify_new_payload(&second_payload, None)
             .await
             .unwrap();
         assert_eq!(status, PayloadStatus::Valid);