Re-merge updated block processing to v0.2.0 (#962)

* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Disable/delete two outdated tests

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Remove decompressed member from pubkey bytes

* Add hashmap for indices lookup

* Add state cache, remove store cache

* Only build the head committee cache

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Tidy

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Tidy

* Fix todos

* Improve tests

* Fix compile error

* Fix compile error from merge

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Rename state_cache -> checkpoint_cache

* Rename Checkpoint -> Snapshot

* Tidy, add comments

* Tidy up find_head function

* Change some checkpoint -> snapshot

* Add tests

* Expose max_len

* Remove dead code

* Tidy

* Fix bug

* Add sync-speed metric

* Add first attempt at VerifiableBlock

* Start integrating into beacon chain

* Integrate VerifiableBlock

* Rename VerifableBlock -> PartialBlockVerification

* Add start of typed methods

* Add progress

* Add further progress

* Rename structs

* Add full block verification to block_processing.rs

* Further beacon chain integration

* Update checks for gossip

* Add todo

* Start adding segement verification

* Add passing chain segement test

* Initial integration with batch sync

* Minor changes

* Tidy, add more error checking

* Start adding chain_segment tests

* Finish invalid signature tests

* Include single and gossip verified blocks in tests

* Add gossip verification tests

* Start adding docs

* Finish adding comments to block_processing.rs

* Rename block_processing.rs -> block_verification

* Start removing old block processing code

* Fixes beacon_chain compilation

* Fix project-wide compile errors

* Remove old code

* Fix bug with beacon proposer index

* Fix shim for BlockProcessingError

* Only process one epoch at a time

* Fix loop in chain segment processing

* Add caching for state.eth1_data_votes

* Add BeaconChain::validator_pubkey

* Revert "Add caching for state.eth1_data_votes"

This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.

* Add sync-speed metric (#898)

* Add PH & MS slot clock changes

* Account for genesis time

* Use checked mul

* Account for genesis slot

* Change API

* Allow for clock disparity

* Refactor "duration to..." functions

* Ensure errors are returned during batch processing

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>

beacon_node/beacon_chain/src/beacon_chain.rs

@@ -80,6 +80,11 @@ pub enum AttestationType {
     Aggregated,
 }
 
+/// The accepted clock drift for nodes gossiping blocks and attestations (spec v0.11.0). See:
+///
+/// https://github.com/ethereum/eth2.0-specs/blob/v0.11.0/specs/phase0/p2p-interface.md#configuration
+pub const MAXIMUM_GOSSIP_CLOCK_DISPARITY: Duration = Duration::from_millis(500);
+
 #[derive(Debug, PartialEq)]
 pub enum AttestationProcessingOutcome {
     Processed,

beacon_node/beacon_chain/src/block_verification.rs

@@ -42,10 +42,14 @@
 //! ```
 use crate::validator_pubkey_cache::ValidatorPubkeyCache;
 use crate::{
-    beacon_chain::{BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT, VALIDATOR_PUBKEY_CACHE_LOCK_TIMEOUT},
+    beacon_chain::{
+        BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT, MAXIMUM_GOSSIP_CLOCK_DISPARITY,
+        VALIDATOR_PUBKEY_CACHE_LOCK_TIMEOUT,
+    },
     metrics, BeaconChain, BeaconChainError, BeaconChainTypes, BeaconSnapshot,
 };
 use parking_lot::RwLockReadGuard;
+use slot_clock::SlotClock;
 use state_processing::{
     block_signature_verifier::{
         BlockSignatureVerifier, Error as BlockSignatureVerifierError, G1Point,
@@ -259,19 +263,18 @@ impl<T: BeaconChainTypes> GossipVerifiedBlock<T> {
         chain: &BeaconChain<T>,
     ) -> Result<Self, BlockError> {
         // Do not gossip or process blocks from future slots.
-        //
-        // TODO: adjust this to allow for clock disparity tolerance.
-        let present_slot = chain.slot()?;
-        if block.slot() > present_slot {
+        let present_slot_with_tolerance = chain
+            .slot_clock
+            .now_with_future_tolerance(MAXIMUM_GOSSIP_CLOCK_DISPARITY)
+            .ok_or_else(|| BeaconChainError::UnableToReadSlot)?;
+        if block.slot() > present_slot_with_tolerance {
             return Err(BlockError::FutureSlot {
-                present_slot,
+                present_slot: present_slot_with_tolerance,
                 block_slot: block.slot(),
             });
         }
 
         // Do not gossip a block from a finalized slot.
-        //
-        // TODO: adjust this to allow for clock disparity tolerance.
         check_block_against_finalized_slot(&block.message, chain)?;
 
         // TODO: add check for the `(block.proposer_index, block.slot)` tuple once we have v0.11.0

beacon_node/beacon_chain/src/errors.rs

@@ -72,8 +72,8 @@ easy_from_to!(SlotProcessingError, BeaconChainError);
 easy_from_to!(AttestationValidationError, BeaconChainError);
 easy_from_to!(SszTypesError, BeaconChainError);
 easy_from_to!(OpPoolError, BeaconChainError);
-easy_from_to!(BlockSignatureVerifierError, BeaconChainError);
 easy_from_to!(NaiveAggregationError, BeaconChainError);
+easy_from_to!(BlockSignatureVerifierError, BeaconChainError);
 
 #[derive(Debug, PartialEq)]
 pub enum BlockProductionError {

beacon_node/network/src/sync/range_sync/batch_processing.rs

@@ -68,6 +68,8 @@ fn process_batch<T: BeaconChainTypes>(
                     log, "Parent block is unknown";
                     "parent_root" => format!("{}", parent),
                 );
+
+                return Err(format!("Block has an unknown parent: {}", parent));
             }
             Err(BlockError::BlockIsAlreadyKnown) => {
                 // this block is already known to us, move to the next
@@ -97,6 +99,11 @@ fn process_batch<T: BeaconChainTypes>(
                         "FUTURE_SLOT_TOLERANCE" => FUTURE_SLOT_TOLERANCE,
                     );
                 }
+
+                return Err(format!(
+                    "Block with slot {} is higher than the current slot {}",
+                    block_slot, present_slot
+                ));
             }
             Err(BlockError::WouldRevertFinalizedSlot { .. }) => {
                 debug!(
@@ -115,6 +122,8 @@ fn process_batch<T: BeaconChainTypes>(
                     "msg" => "unexpected condition in processing block.",
                     "outcome" => format!("{:?}", e)
                 );
+
+                return Err(format!("Internal error whilst processing block: {:?}", e));
             }
             other => {
                 warn!(
@@ -122,6 +131,8 @@ fn process_batch<T: BeaconChainTypes>(
                     "msg" => "peer sent invalid block",
                     "outcome" => format!("{:?}", other),
                 );
+
+                return Err(format!("Peer sent invalid block. Reason: {:?}", other));
             }
         }
     } else {