Initial work towards v0.2.0 (#924)

* Remove ping protocol

* Initial renaming of network services

* Correct rebasing relative to latest master

* Start updating types

* Adds HashMapDelay struct to utils

* Initial network restructure

* Network restructure. Adds new types for v0.2.0

* Removes build artefacts

* Shift validation to beacon chain

* Temporarily remove gossip validation

This is to be updated to match current optimisation efforts.

* Adds AggregateAndProof

* Begin rebuilding pubsub encoding/decoding

* Signature hacking

* Shift gossipsup decoding into eth2_libp2p

* Existing EF tests passing with fake_crypto

* Shifts block encoding/decoding into RPC

* Delete outdated API spec

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Fast aggregate verify test

* Update REST API docs

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Progress on attestation service

* Address review comments; remove unused dependency

* Initial work on removing libp2p lock

* Add LRU caches to store (rollup)

* Update attestation validation for DB changes (WIP)

* Initial version of should_forward_block

* Scaffold

* Progress on attestation validation

Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.

* Removes lock from libp2p service

* Completed network lock removal

* Finish(?) attestation processing

* Correct network termination future

* Add slot check to block check

* Correct fmt issues

* Remove Drop implementation for network service

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Address reviewers suggestions

* Disable/delete two outdated tests

* Modification of validator for subscriptions

* Add slot signing to validator client

* Further progress on validation subscription

* Adds necessary validator subscription functionality

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Clean up validator <-> beacon node http types

* Add aggregator status to ValidatorDuty

* Impl Clone for manual slot clock

* Fix minor errors

* Further progress validator client subscription

* Initial subscription and aggregation handling

* Remove decompressed member from pubkey bytes

* Progress to modifying val client for attestation aggregation

* First draft of validator client upgrade for aggregate attestations

* Add hashmap for indices lookup

* Add state cache, remove store cache

* Only build the head committee cache

* Removes lock on a network channel

* Partially implement beacon node subscription http api

* Correct compilation issues

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Partial implementation of timer

* Adds timer, removes exit_future, http api to op pool

* Partial multiple aggregate attestation handling

* Permits bulk messages accross gossipsub network channel

* Correct compile issues

* Improve gosispsub messaging and correct rest api helpers

* Added global gossipsub subscriptions

* Update validator subscriptions data structs

* Tidy

* Re-structure validator subscriptions

* Initial handling of subscriptions

* Re-structure network service

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add HashSetDelay and introduce into attestation service

* Handles validator subscriptions

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Peer subnet discovery gets queued for future subscriptions

* Reorganise attestation service functions

* Initial wiring of attestation service

* First draft of attestation service timing logic

* Correct minor typos

* Tidy

* Fix todos

* Improve tests

* Add PeerInfo to connected peers mapping

* Fix compile error

* Fix compile error from merge

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Rename state_cache -> checkpoint_cache

* Rename Checkpoint -> Snapshot

* Tidy, add comments

* Tidy up find_head function

* Change some checkpoint -> snapshot

* Add tests

* Expose max_len

* Remove dead code

* Tidy

* Fix bug

* Add sync-speed metric

* Add first attempt at VerifiableBlock

* Start integrating into beacon chain

* Integrate VerifiableBlock

* Rename VerifableBlock -> PartialBlockVerification

* Add start of typed methods

* Add progress

* Add further progress

* Rename structs

* Add full block verification to block_processing.rs

* Further beacon chain integration

* Update checks for gossip

* Add todo

* Start adding segement verification

* Add passing chain segement test

* Initial integration with batch sync

* Minor changes

* Tidy, add more error checking

* Start adding chain_segment tests

* Finish invalid signature tests

* Include single and gossip verified blocks in tests

* Add gossip verification tests

* Start adding docs

* Finish adding comments to block_processing.rs

* Rename block_processing.rs -> block_verification

* Start removing old block processing code

* Fixes beacon_chain compilation

* Fix project-wide compile errors

* Remove old code

* Correct code to pass all tests

* Fix bug with beacon proposer index

* Fix shim for BlockProcessingError

* Only process one epoch at a time

* Fix loop in chain segment processing

* Correct tests from master merge

* Add caching for state.eth1_data_votes

* Add BeaconChain::validator_pubkey

* Revert "Add caching for state.eth1_data_votes"

This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.

Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>

beacon_node/store/src/hot_cold_store.rs

@@ -10,7 +10,7 @@ use crate::{
     leveldb_store::LevelDB, DBColumn, Error, PartialBeaconState, SimpleStoreItem, Store, StoreItem,
 };
 use lru::LruCache;
-use parking_lot::{Mutex, RwLock};
+use parking_lot::RwLock;
 use slog::{debug, trace, warn, Logger};
 use ssz::{Decode, Encode};
 use ssz_derive::{Decode, Encode};
@@ -22,7 +22,6 @@ use std::convert::TryInto;
 use std::marker::PhantomData;
 use std::path::Path;
 use std::sync::Arc;
-use types::beacon_state::CloneConfig;
 use types::*;
 
 /// 32-byte key for accessing the `split` of the freezer DB.
@@ -46,9 +45,7 @@ pub struct HotColdDB<E: EthSpec> {
     /// The hot database also contains all blocks.
     pub(crate) hot_db: LevelDB<E>,
     /// LRU cache of deserialized blocks. Updated whenever a block is loaded.
-    block_cache: Mutex<LruCache<Hash256, SignedBeaconBlock<E>>>,
-    /// LRU cache of deserialized states. Updated whenever a state is loaded.
-    state_cache: Mutex<LruCache<Hash256, BeaconState<E>>>,
+    block_cache: RwLock<LruCache<Hash256, SignedBeaconBlock<E>>>,
     /// Chain spec.
     spec: ChainSpec,
     /// Logger.
@@ -112,7 +109,7 @@ impl<E: EthSpec> Store<E> for HotColdDB<E> {
         self.put(block_root, &block)?;
 
         // Update cache.
-        self.block_cache.lock().put(*block_root, block);
+        self.block_cache.write().put(*block_root, block);
 
         Ok(())
     }
@@ -122,7 +119,7 @@ impl<E: EthSpec> Store<E> for HotColdDB<E> {
         metrics::inc_counter(&metrics::BEACON_BLOCK_GET_COUNT);
 
         // Check the cache.
-        if let Some(block) = self.block_cache.lock().get(block_root) {
+        if let Some(block) = self.block_cache.write().get(block_root) {
             metrics::inc_counter(&metrics::BEACON_BLOCK_CACHE_HIT_COUNT);
             return Ok(Some(block.clone()));
         }
@@ -131,7 +128,7 @@ impl<E: EthSpec> Store<E> for HotColdDB<E> {
         match self.get::<SignedBeaconBlock<E>>(block_root)? {
             Some(block) => {
                 // Add to cache.
-                self.block_cache.lock().put(*block_root, block.clone());
+                self.block_cache.write().put(*block_root, block.clone());
                 Ok(Some(block))
             }
             None => Ok(None),
@@ -140,12 +137,12 @@ impl<E: EthSpec> Store<E> for HotColdDB<E> {
 
     /// Delete a block from the store and the block cache.
     fn delete_block(&self, block_root: &Hash256) -> Result<(), Error> {
-        self.block_cache.lock().pop(block_root);
+        self.block_cache.write().pop(block_root);
         self.delete::<SignedBeaconBlock<E>>(block_root)
     }
 
     /// Store a state in the store.
-    fn put_state(&self, state_root: &Hash256, state: BeaconState<E>) -> Result<(), Error> {
+    fn put_state(&self, state_root: &Hash256, state: &BeaconState<E>) -> Result<(), Error> {
         if state.slot < self.get_split_slot() {
             self.store_cold_state(state_root, &state)
         } else {
@@ -159,7 +156,7 @@ impl<E: EthSpec> Store<E> for HotColdDB<E> {
         state_root: &Hash256,
         slot: Option<Slot>,
     ) -> Result<Option<BeaconState<E>>, Error> {
-        self.get_state_with(state_root, slot, CloneConfig::all())
+        self.get_state_with(state_root, slot)
     }
 
     /// Get a state from the store.
@@ -169,7 +166,6 @@ impl<E: EthSpec> Store<E> for HotColdDB<E> {
         &self,
         state_root: &Hash256,
         slot: Option<Slot>,
-        clone_config: CloneConfig,
     ) -> Result<Option<BeaconState<E>>, Error> {
         metrics::inc_counter(&metrics::BEACON_STATE_GET_COUNT);
 
@@ -177,10 +173,10 @@ impl<E: EthSpec> Store<E> for HotColdDB<E> {
             if slot < self.get_split_slot() {
                 self.load_cold_state_by_slot(slot).map(Some)
             } else {
-                self.load_hot_state(state_root, clone_config)
+                self.load_hot_state(state_root)
             }
         } else {
-            match self.load_hot_state(state_root, clone_config)? {
+            match self.load_hot_state(state_root)? {
                 Some(state) => Ok(Some(state)),
                 None => self.load_cold_state(state_root),
             }
@@ -204,9 +200,6 @@ impl<E: EthSpec> Store<E> for HotColdDB<E> {
                 .key_delete(DBColumn::BeaconState.into(), state_root.as_bytes())?;
         }
 
-        // Delete from the cache.
-        self.state_cache.lock().pop(state_root);
-
         Ok(())
     }
 
@@ -309,10 +302,7 @@ impl<E: EthSpec> Store<E> for HotColdDB<E> {
         {
             // NOTE: minor inefficiency here because we load an unnecessary hot state summary
             let state = self
-                .load_hot_state(
-                    &epoch_boundary_state_root,
-                    CloneConfig::committee_caches_only(),
-                )?
+                .load_hot_state(&epoch_boundary_state_root)?
                 .ok_or_else(|| {
                     HotColdDBError::MissingEpochBoundaryState(epoch_boundary_state_root)
                 })?;
@@ -348,8 +338,7 @@ impl<E: EthSpec> HotColdDB<E> {
             split: RwLock::new(Split::default()),
             cold_db: LevelDB::open(cold_path)?,
             hot_db: LevelDB::open(hot_path)?,
-            block_cache: Mutex::new(LruCache::new(config.block_cache_size)),
-            state_cache: Mutex::new(LruCache::new(config.state_cache_size)),
+            block_cache: RwLock::new(LruCache::new(config.block_cache_size)),
             config,
             spec,
             log,
@@ -371,7 +360,7 @@ impl<E: EthSpec> HotColdDB<E> {
     pub fn store_hot_state(
         &self,
         state_root: &Hash256,
-        state: BeaconState<E>,
+        state: &BeaconState<E>,
     ) -> Result<(), Error> {
         // On the epoch boundary, store the full state.
         if state.slot % E::slots_per_epoch() == 0 {
@@ -387,10 +376,7 @@ impl<E: EthSpec> HotColdDB<E> {
         // Store a summary of the state.
         // We store one even for the epoch boundary states, as we may need their slots
         // when doing a look up by state root.
-        self.put_state_summary(state_root, HotStateSummary::new(state_root, &state)?)?;
-
-        // Store the state in the cache.
-        self.state_cache.lock().put(*state_root, state);
+        self.put_state_summary(state_root, HotStateSummary::new(state_root, state)?)?;
 
         Ok(())
     }
@@ -398,24 +384,9 @@ impl<E: EthSpec> HotColdDB<E> {
     /// Load a post-finalization state from the hot database.
     ///
     /// Will replay blocks from the nearest epoch boundary.
-    pub fn load_hot_state(
-        &self,
-        state_root: &Hash256,
-        clone_config: CloneConfig,
-    ) -> Result<Option<BeaconState<E>>, Error> {
+    pub fn load_hot_state(&self, state_root: &Hash256) -> Result<Option<BeaconState<E>>, Error> {
         metrics::inc_counter(&metrics::BEACON_STATE_HOT_GET_COUNT);
 
-        // Check the cache.
-        if let Some(state) = self.state_cache.lock().get(state_root) {
-            metrics::inc_counter(&metrics::BEACON_STATE_CACHE_HIT_COUNT);
-
-            let timer = metrics::start_timer(&metrics::BEACON_STATE_CACHE_CLONE_TIME);
-            let state = state.clone_with(clone_config);
-            metrics::stop_timer(timer);
-
-            return Ok(Some(state));
-        }
-
         if let Some(HotStateSummary {
             slot,
             latest_block_root,
@@ -439,9 +410,6 @@ impl<E: EthSpec> HotColdDB<E> {
                 self.replay_blocks(boundary_state, blocks, slot)?
             };
 
-            // Update the LRU cache.
-            self.state_cache.lock().put(*state_root, state.clone());
-
             Ok(Some(state))
         } else {
             Ok(None)

beacon_node/store/src/iter.rs

@@ -345,7 +345,7 @@ mod test {
 
         let state_a_root = hashes.next().unwrap();
         state_b.state_roots[0] = state_a_root;
-        store.put_state(&state_a_root, state_a).unwrap();
+        store.put_state(&state_a_root, &state_a).unwrap();
 
         let iter = BlockRootsIterator::new(store, &state_b);
 
@@ -393,8 +393,8 @@ mod test {
         let state_a_root = Hash256::from_low_u64_be(slots_per_historical_root as u64);
         let state_b_root = Hash256::from_low_u64_be(slots_per_historical_root as u64 * 2);
 
-        store.put_state(&state_a_root, state_a).unwrap();
-        store.put_state(&state_b_root, state_b.clone()).unwrap();
+        store.put_state(&state_a_root, &state_a).unwrap();
+        store.put_state(&state_b_root, &state_b.clone()).unwrap();
 
         let iter = StateRootsIterator::new(store, &state_b);
 

beacon_node/store/src/leveldb_store.rs

@@ -123,7 +123,7 @@ impl<E: EthSpec> Store<E> for LevelDB<E> {
     }
 
     /// Store a state in the store.
-    fn put_state(&self, state_root: &Hash256, state: BeaconState<E>) -> Result<(), Error> {
+    fn put_state(&self, state_root: &Hash256, state: &BeaconState<E>) -> Result<(), Error> {
         store_full_state(self, state_root, &state)
     }
 

beacon_node/store/src/lib.rs

@@ -38,7 +38,6 @@ pub use errors::Error;
 pub use impls::beacon_state::StorageContainer as BeaconStateStorageContainer;
 pub use metrics::scrape_for_metrics;
 pub use state_batch::StateBatch;
-pub use types::beacon_state::CloneConfig;
 pub use types::*;
 
 /// An object capable of storing and retrieving objects implementing `StoreItem`.
@@ -97,7 +96,7 @@ pub trait Store<E: EthSpec>: Sync + Send + Sized + 'static {
     }
 
     /// Store a state in the store.
-    fn put_state(&self, state_root: &Hash256, state: BeaconState<E>) -> Result<(), Error>;
+    fn put_state(&self, state_root: &Hash256, state: &BeaconState<E>) -> Result<(), Error>;
 
     /// Store a state summary in the store.
     // NOTE: this is a hack for the HotColdDb, we could consider splitting this
@@ -122,7 +121,6 @@ pub trait Store<E: EthSpec>: Sync + Send + Sized + 'static {
         &self,
         state_root: &Hash256,
         slot: Option<Slot>,
-        _clone_config: CloneConfig,
     ) -> Result<Option<BeaconState<E>>, Error> {
         // Default impl ignores config. Overriden in `HotColdDb`.
         self.get_state(state_root, slot)

beacon_node/store/src/memory_store.rs

@@ -76,7 +76,7 @@ impl<E: EthSpec> Store<E> for MemoryStore<E> {
     }
 
     /// Store a state in the store.
-    fn put_state(&self, state_root: &Hash256, state: BeaconState<E>) -> Result<(), Error> {
+    fn put_state(&self, state_root: &Hash256, state: &BeaconState<E>) -> Result<(), Error> {
         store_full_state(self, state_root, &state)
     }
 

beacon_node/store/src/state_batch.rs

@@ -38,7 +38,7 @@ impl<E: EthSpec> StateBatch<E> {
     /// May fail to write the full batch if any of the items error (i.e. not atomic!)
     pub fn commit<S: Store<E>>(self, store: &S) -> Result<(), Error> {
         self.items.into_iter().try_for_each(|item| match item {
-            BatchItem::Full(state_root, state) => store.put_state(&state_root, state),
+            BatchItem::Full(state_root, state) => store.put_state(&state_root, &state),
             BatchItem::Summary(state_root, summary) => {
                 store.put_state_summary(&state_root, summary)
             }