cerc-io/lighthouse
16
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Bootstrap (#501)

Browse Source 
* Renamed fork_choice::process_attestation_from_block

* Processing attestation in fork choice

* Retrieving state from store and checking signature

* Looser check on beacon state validity.

* Cleaned up get_attestation_state

* Expanded fork choice api to provide latest validator message.

* Checking if the an attestation contains a latest message

* Correct process_attestation error handling.

* Copy paste error in comment fixed.

* Tidy ancestor iterators

* Getting attestation slot via helper method

* Refactored attestation creation in test utils

* Revert "Refactored attestation creation in test utils"

This reverts commit 4d277fe4239a7194758b18fb5c00dfe0b8231306.

* Integration tests for free attestation processing

* Implicit conflicts resolved.

* formatting

* Do first pass on Grants code

* Add another attestation processing test

* Tidy attestation processing

* Remove old code fragment

* Add non-compiling half finished changes

* Simplify, fix bugs, add tests for chain iters

* Remove attestation processing from op pool

* Fix bug with fork choice, tidy

* Fix overly restrictive check in fork choice.

* Ensure committee cache is build during attn proc

* Ignore unknown blocks at fork choice

* Various minor fixes

* Make fork choice write lock in to read lock

* Remove unused method

* Tidy comments

* Fix attestation prod. target roots change

* Fix compile error in store iters

* Reject any attestation prior to finalization

* Begin metrics refactor

* Move beacon_chain to new metrics structure.

* Make metrics not panic if already defined

* Use global prometheus gather at rest api

* Unify common metric fns into a crate

* Add heavy metering to block processing

* Remove hypen from prometheus metric name

* Add more beacon chain metrics

* Add beacon chain persistence metric

* Prune op pool on finalization

* Add extra prom beacon chain metrics

* Prefix BeaconChain metrics with "beacon_"

* Add more store metrics

* Add basic metrics to libp2p

* Add metrics to HTTP server

* Remove old `http_server` crate

* Update metrics names to be more like standard

* Fix broken beacon chain metrics, add slot clock metrics

* Add lighthouse_metrics gather fn

* Remove http args

* Fix wrong state given to op pool prune

* Make prom metric names more consistent

* Add more metrics, tidy existing metrics

* Fix store block read metrics

* Tidy attestation metrics

* Fix minor PR comments

* Fix minor PR comments

* Remove duplicated attestation finalization check

* Remove awkward `let` statement

* Add first attempts at HTTP bootstrap

* Add beacon_block methods to rest api

* Fix serde for block.body.grafitti

* Allow travis failures on beta (see desc)

There's a non-backward compatible change in `cargo fmt`. Stable and beta
do not agree.

* Add network routes to API

* Fix rustc warnings

* Add best_slot method

* Add --bootstrap arg to beacon node

* Get bootstrapper working for ENR address

* Store intermediate states during block processing

* Allow bootstrapper to scrape libp2p address

* Update bootstrapper libp2p address finding

* Add comments

* Tidy API to be more consistent with recent decisions

* Address some review comments

* Make BeaconChainTypes Send + Sync + 'static

* Add `/network/listen_port` API endpoint

* Abandon starting the node if libp2p doesn't start

* Update bootstrapper for API changes

* Remove unnecessary trait bounds
This commit is contained in:
Paul Hauner 2019-08-23 15:53:53 +10:00 committed by GitHub
parent c4ced3e0d2
commit 0c3fdcd57c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
23 changed files with 742 additions and 49 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
beacon_node/beacon_chain/src/beacon_chain.rs
View File

@ -77,7 +77,7 @@ pub enum AttestationProcessingOutcome {
Invalid(AttestationValidationError),
}
pub trait BeaconChainTypes {
pub trait BeaconChainTypes: Send + Sync + 'static {
type Store: store::Store;
type SlotClock: slot_clock::SlotClock;
type LmdGhost: LmdGhost<Self::Store, Self::EthSpec>;
@ -870,9 +870,16 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
let catchup_timer = metrics::start_timer(&metrics::BLOCK_PROCESSING_CATCHUP_STATE);
// Keep a list of any states that were "skipped" (block-less) in between the parent state
// slot and the block slot. These will need to be stored in the database.
let mut intermediate_states = vec![];
// Transition the parent state to the block slot.
let mut state: BeaconState<T::EthSpec> = parent_state;
for _ in state.slot.as_u64()..block.slot.as_u64() {
for i in state.slot.as_u64()..block.slot.as_u64() {
if i > 0 {
intermediate_states.push(state.clone());
}
per_slot_processing(&mut state, &self.spec)?;
}
@ -911,6 +918,22 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
let db_write_timer = metrics::start_timer(&metrics::BLOCK_PROCESSING_DB_WRITE);
// Store all the states between the parent block state and this blocks slot before storing
// the final state.
for (i, intermediate_state) in intermediate_states.iter().enumerate() {
// To avoid doing an unnecessary tree hash, use the following (slot + 1) state's
// state_roots field to find the root.
let following_state = match intermediate_states.get(i + 1) {
Some(following_state) => following_state,
None => &state,
};
let intermediate_state_root =
following_state.get_state_root(intermediate_state.slot)?;
self.store
.put(&intermediate_state_root, intermediate_state)?;
}
// Store the block and state.
self.store.put(&block_root, &block)?;
self.store.put(&state_root, &state)?;

4
beacon_node/beacon_chain/src/test_utils.rs
View File

@ -54,7 +54,7 @@ where
impl<L, E> BeaconChainTypes for CommonTypes<L, E>
where
L: LmdGhost<MemoryStore, E>,
L: LmdGhost<MemoryStore, E> + 'static,
E: EthSpec,
{
type Store = MemoryStore;
@ -69,7 +69,7 @@ where
/// Used for testing.
pub struct BeaconChainHarness<L, E>
where
L: LmdGhost<MemoryStore, E>,
L: LmdGhost<MemoryStore, E> + 'static,
E: EthSpec,
{
pub chain: BeaconChain<CommonTypes<L, E>>,

3
beacon_node/client/Cargo.toml
View File

@ -7,6 +7,7 @@ edition = "2018"
[dependencies]
beacon_chain = { path = "../beacon_chain" }
network = { path = "../network" }
eth2-libp2p = { path = "../eth2-libp2p" }
rpc = { path = "../rpc" }
rest_api = { path = "../rest_api" }
prometheus = "^0.6"
@ -26,3 +27,5 @@ clap = "2.32.0"
dirs = "1.0.3"
exit-future = "0.1.3"
futures = "0.1.25"
reqwest = "0.9"
url = "1.2"

17
beacon_node/client/src/beacon_chain_types.rs
View File

@ -1,3 +1,4 @@
use crate::bootstrapper::Bootstrapper;
use crate::error::Result;
use crate::{config::GenesisState, ClientConfig};
use beacon_chain::{
@ -35,7 +36,11 @@ pub struct ClientType<S: Store, E: EthSpec> {
_phantom_u: PhantomData<E>,
}
impl<S: Store, E: EthSpec + Clone> BeaconChainTypes for ClientType<S, E> {
impl<S, E> BeaconChainTypes for ClientType<S, E>
where
S: Store + 'static,
E: EthSpec,
{
type Store = S;
type SlotClock = SystemTimeSlotClock;
type LmdGhost = ThreadSafeReducedTree<S, E>;
@ -74,6 +79,16 @@ where
serde_yaml::from_reader(file)
.map_err(|e| format!("Unable to parse YAML genesis state file: {:?}", e))?
}
GenesisState::HttpBootstrap { server } => {
let bootstrapper = Bootstrapper::from_server_string(server.to_string())
.map_err(|e| format!("Failed to initialize bootstrap client: {}", e))?;
let (state, _block) = bootstrapper
.genesis()
.map_err(|e| format!("Failed to bootstrap genesis state: {}", e))?;
state
}
};
let mut genesis_block = BeaconBlock::empty(&spec);

210
beacon_node/client/src/bootstrapper.rs Normal file
View File

@ -0,0 +1,210 @@
use eth2_libp2p::{
multiaddr::{Multiaddr, Protocol},
Enr,
};
use reqwest::{Error as HttpError, Url};
use serde::Deserialize;
use std::borrow::Cow;
use std::net::Ipv4Addr;
use types::{BeaconBlock, BeaconState, Checkpoint, EthSpec, Hash256, Slot};
use url::Host;
#[derive(Debug)]
enum Error {
InvalidUrl,
HttpError(HttpError),
}
impl From<HttpError> for Error {
fn from(e: HttpError) -> Error {
Error::HttpError(e)
}
}
/// Used to load "bootstrap" information from the HTTP API of another Lighthouse beacon node.
///
/// Bootstrapping information includes things like genesis and finalized states and blocks, and
/// libp2p connection details.
pub struct Bootstrapper {
url: Url,
}
impl Bootstrapper {
/// Parses the given `server` as a URL, instantiating `Self`.
pub fn from_server_string(server: String) -> Result<Self, String> {
Ok(Self {
url: Url::parse(&server).map_err(|e| format!("Invalid bootstrap server url: {}", e))?,
})
}
/// Build a multiaddr using the HTTP server URL that is not guaranteed to be correct.
///
/// The address is created by querying the HTTP server for its listening libp2p addresses.
/// Then, we find the first TCP port in those addresses and combine the port with the URL of
/// the server.
///
/// For example, the server `http://192.168.0.1` might end up with a `best_effort_multiaddr` of
/// `/ipv4/192.168.0.1/tcp/9000` if the server advertises a listening address of
/// `/ipv4/172.0.0.1/tcp/9000`.
pub fn best_effort_multiaddr(&self) -> Option<Multiaddr> {
let tcp_port = self.listen_port().ok()?;
let mut multiaddr = Multiaddr::with_capacity(2);
match self.url.host()? {
Host::Ipv4(addr) => multiaddr.push(Protocol::Ip4(addr)),
Host::Domain(s) => multiaddr.push(Protocol::Dns4(Cow::Borrowed(s))),
_ => return None,
};
multiaddr.push(Protocol::Tcp(tcp_port));
Some(multiaddr)
}
/// Returns the IPv4 address of the server URL, unless it contains a FQDN.
pub fn server_ipv4_addr(&self) -> Option<Ipv4Addr> {
match self.url.host()? {
Host::Ipv4(addr) => Some(addr),
_ => None,
}
}
/// Returns the servers ENR address.
pub fn enr(&self) -> Result<Enr, String> {
get_enr(self.url.clone()).map_err(|e| format!("Unable to get ENR: {:?}", e))
}
/// Returns the servers listening libp2p addresses.
pub fn listen_port(&self) -> Result<u16, String> {
get_listen_port(self.url.clone()).map_err(|e| format!("Unable to get listen port: {:?}", e))
}
/// Returns the genesis block and state.
pub fn genesis<T: EthSpec>(&self) -> Result<(BeaconState<T>, BeaconBlock<T>), String> {
let genesis_slot = Slot::new(0);
let block = get_block(self.url.clone(), genesis_slot)
.map_err(|e| format!("Unable to get genesis block: {:?}", e))?
.beacon_block;
let state = get_state(self.url.clone(), genesis_slot)
.map_err(|e| format!("Unable to get genesis state: {:?}", e))?
.beacon_state;
Ok((state, block))
}
/// Returns the most recent finalized state and block.
pub fn finalized<T: EthSpec>(&self) -> Result<(BeaconState<T>, BeaconBlock<T>), String> {
let slots_per_epoch = get_slots_per_epoch(self.url.clone())
.map_err(|e| format!("Unable to get slots per epoch: {:?}", e))?;
let finalized_slot = get_finalized_slot(self.url.clone(), slots_per_epoch.as_u64())
.map_err(|e| format!("Unable to get finalized slot: {:?}", e))?;
let block = get_block(self.url.clone(), finalized_slot)
.map_err(|e| format!("Unable to get finalized block: {:?}", e))?
.beacon_block;
let state = get_state(self.url.clone(), finalized_slot)
.map_err(|e| format!("Unable to get finalized state: {:?}", e))?
.beacon_state;
Ok((state, block))
}
}
fn get_slots_per_epoch(mut url: Url) -> Result<Slot, Error> {
url.path_segments_mut()
.map(|mut url| {
url.push("spec").push("slots_per_epoch");
})
.map_err(|_| Error::InvalidUrl)?;
reqwest::get(url)?
.error_for_status()?
.json()
.map_err(Into::into)
}
fn get_finalized_slot(mut url: Url, slots_per_epoch: u64) -> Result<Slot, Error> {
url.path_segments_mut()
.map(|mut url| {
url.push("beacon").push("latest_finalized_checkpoint");
})
.map_err(|_| Error::InvalidUrl)?;
let checkpoint: Checkpoint = reqwest::get(url)?.error_for_status()?.json()?;
Ok(checkpoint.epoch.start_slot(slots_per_epoch))
}
#[derive(Deserialize)]
#[serde(bound = "T: EthSpec")]
pub struct StateResponse<T: EthSpec> {
pub root: Hash256,
pub beacon_state: BeaconState<T>,
}
fn get_state<T: EthSpec>(mut url: Url, slot: Slot) -> Result<StateResponse<T>, Error> {
url.path_segments_mut()
.map(|mut url| {
url.push("beacon").push("state");
})
.map_err(|_| Error::InvalidUrl)?;
url.query_pairs_mut()
.append_pair("slot", &format!("{}", slot.as_u64()));
reqwest::get(url)?
.error_for_status()?
.json()
.map_err(Into::into)
}
#[derive(Deserialize)]
#[serde(bound = "T: EthSpec")]
pub struct BlockResponse<T: EthSpec> {
pub root: Hash256,
pub beacon_block: BeaconBlock<T>,
}
fn get_block<T: EthSpec>(mut url: Url, slot: Slot) -> Result<BlockResponse<T>, Error> {
url.path_segments_mut()
.map(|mut url| {
url.push("beacon").push("block");
})
.map_err(|_| Error::InvalidUrl)?;
url.query_pairs_mut()
.append_pair("slot", &format!("{}", slot.as_u64()));
reqwest::get(url)?
.error_for_status()?
.json()
.map_err(Into::into)
}
fn get_enr(mut url: Url) -> Result<Enr, Error> {
url.path_segments_mut()
.map(|mut url| {
url.push("network").push("enr");
})
.map_err(|_| Error::InvalidUrl)?;
reqwest::get(url)?
.error_for_status()?
.json()
.map_err(Into::into)
}
fn get_listen_port(mut url: Url) -> Result<u16, Error> {
url.path_segments_mut()
.map(|mut url| {
url.push("network").push("listen_port");
})
.map_err(|_| Error::InvalidUrl)?;
reqwest::get(url)?
.error_for_status()?
.json()
.map_err(Into::into)
}

40
beacon_node/client/src/config.rs
View File

@ -1,8 +1,8 @@
use crate::Eth2Config;
use crate::{Bootstrapper, Eth2Config};
use clap::ArgMatches;
use network::NetworkConfig;
use serde_derive::{Deserialize, Serialize};
use slog::{info, o, Drain};
use slog::{info, o, warn, Drain};
use std::fs::{self, OpenOptions};
use std::path::PathBuf;
use std::sync::Mutex;
@ -46,6 +46,8 @@ pub enum GenesisState {
},
/// Load a YAML-encoded genesis state from a file.
Yaml { file: PathBuf },
/// Use a HTTP server (running our REST-API) to load genesis and finalized states and blocks.
HttpBootstrap { server: String },
}
impl Default for Config {
@ -147,6 +149,40 @@ impl Config {
self.update_logger(log)?;
};
// If the `--bootstrap` flag is provided, overwrite the default configuration.
if let Some(server) = args.value_of("bootstrap") {
do_bootstrapping(self, server.to_string(), &log)?;
}
Ok(())
}
}
/// Perform the HTTP bootstrapping procedure, reading an ENR and multiaddr from the HTTP server and
/// adding them to the `config`.
fn do_bootstrapping(config: &mut Config, server: String, log: &slog::Logger) -> Result<(), String> {
// Set the genesis state source.
config.genesis_state = GenesisState::HttpBootstrap {
server: server.to_string(),
};
let bootstrapper = Bootstrapper::from_server_string(server.to_string())?;
config.network.boot_nodes.push(bootstrapper.enr()?);
if let Some(server_multiaddr) = bootstrapper.best_effort_multiaddr() {
info!(
log,
"Estimated bootstrapper libp2p address";
"multiaddr" => format!("{:?}", server_multiaddr)
);
config.network.libp2p_nodes.push(server_multiaddr);
} else {
warn!(
log,
"Unable to estimate a bootstrapper libp2p address, this node may not find any peers."
);
}
Ok(())
}

7
beacon_node/client/src/lib.rs
View File

@ -1,6 +1,7 @@
extern crate slog;
mod beacon_chain_types;
mod bootstrapper;
mod config;
pub mod error;
@ -21,7 +22,8 @@ use tokio::timer::Interval;
pub use beacon_chain::BeaconChainTypes;
pub use beacon_chain_types::ClientType;
pub use beacon_chain_types::InitialiseBeaconChain;
pub use config::Config as ClientConfig;
pub use bootstrapper::Bootstrapper;
pub use config::{Config as ClientConfig, GenesisState};
pub use eth2_config::Eth2Config;
/// Main beacon node client service. This provides the connection and initialisation of the clients
@ -47,7 +49,7 @@ pub struct Client<T: BeaconChainTypes> {
impl<T> Client<T>
where
T: BeaconChainTypes + InitialiseBeaconChain<T> + Clone + 'static,
T: BeaconChainTypes + InitialiseBeaconChain<T> + Clone,
{
/// Generate an instance of the client. Spawn and link all internal sub-processes.
pub fn new(
@ -121,6 +123,7 @@ where
&client_config.rest_api,
executor,
beacon_chain.clone(),
network.clone(),
client_config.db_path().expect("unable to read datadir"),
&log,
) {

6
beacon_node/client/src/notifier.rs
View File

@ -17,11 +17,7 @@ pub const WARN_PEER_COUNT: usize = 1;
/// durations.
///
/// Presently unused, but remains for future use.
pub fn run<T: BeaconChainTypes + Send + Sync + 'static>(
client: &Client<T>,
executor: TaskExecutor,
exit: Exit,
) {
pub fn run<T: BeaconChainTypes>(client: &Client<T>, executor: TaskExecutor, exit: Exit) {
// notification heartbeat
let interval = Interval::new(
Instant::now(),

4
beacon_node/eth2-libp2p/src/behaviour.rs
View File

@ -78,6 +78,10 @@ impl<TSubstream: AsyncRead + AsyncWrite> Behaviour<TSubstream> {
log: behaviour_log,
})
}
pub fn discovery(&self) -> &Discovery<TSubstream> {
&self.discovery
}
}
// Implement the NetworkBehaviourEventProcess trait so that we can derive NetworkBehaviour for Behaviour

9
beacon_node/eth2-libp2p/src/discovery.rs
View File

@ -103,6 +103,10 @@ impl<TSubstream> Discovery<TSubstream> {
})
}
pub fn local_enr(&self) -> &Enr {
self.discovery.local_enr()
}
/// Manually search for peers. This restarts the discovery round, sparking multiple rapid
/// queries.
pub fn discover_peers(&mut self) {
@ -120,6 +124,11 @@ impl<TSubstream> Discovery<TSubstream> {
self.connected_peers.len()
}
/// The current number of connected libp2p peers.
pub fn connected_peer_set(&self) -> &HashSet<PeerId> {
&self.connected_peers
}
/// Search for new peers using the underlying discovery mechanism.
fn find_peers(&mut self) {
// pick a random NodeId

3
beacon_node/eth2-libp2p/src/lib.rs
View File

@ -17,12 +17,13 @@ pub use behaviour::PubsubMessage;
pub use config::{
Config as NetworkConfig, BEACON_ATTESTATION_TOPIC, BEACON_BLOCK_TOPIC, SHARD_TOPIC_PREFIX,
};
pub use libp2p::enr::Enr;
pub use libp2p::gossipsub::{Topic, TopicHash};
pub use libp2p::multiaddr;
pub use libp2p::Multiaddr;
pub use libp2p::{
gossipsub::{GossipsubConfig, GossipsubConfigBuilder},
PeerId,
PeerId, Swarm,
};
pub use rpc::RPCEvent;
pub use service::Libp2pEvent;

17
beacon_node/eth2-libp2p/src/service.rs
View File

@ -16,7 +16,7 @@ use libp2p::core::{
upgrade::{InboundUpgradeExt, OutboundUpgradeExt},
};
use libp2p::{core, secio, PeerId, Swarm, Transport};
use slog::{debug, info, trace, warn};
use slog::{crit, debug, info, trace, warn};
use std::fs::File;
use std::io::prelude::*;
use std::io::{Error, ErrorKind};
@ -33,7 +33,7 @@ pub struct Service {
//TODO: Make this private
pub swarm: Swarm<Libp2pStream, Libp2pBehaviour>,
/// This node's PeerId.
_local_peer_id: PeerId,
pub local_peer_id: PeerId,
/// The libp2p logger handle.
pub log: slog::Logger,
}
@ -69,10 +69,15 @@ impl Service {
log_address.push(Protocol::P2p(local_peer_id.clone().into()));
info!(log, "Listening on: {}", log_address);
}
Err(err) => warn!(
Err(err) => {
crit!(
log,
"Cannot listen on: {} because: {:?}", listen_multiaddr, err
),
"Unable to listen on libp2p address";
"error" => format!("{:?}", err),
"listen_multiaddr" => format!("{}", listen_multiaddr),
);
return Err("Libp2p was unable to listen on the given listen address.".into());
}
};
// attempt to connect to user-input libp2p nodes
@ -113,7 +118,7 @@ impl Service {
info!(log, "Subscribed to topics: {:?}", subscribed_topics);
Ok(Service {
_local_peer_id: local_peer_id,
local_peer_id,
swarm,
log,
})

50
beacon_node/network/src/service.rs
View File

@ -5,7 +5,7 @@ use beacon_chain::{BeaconChain, BeaconChainTypes};
use core::marker::PhantomData;
use eth2_libp2p::Service as LibP2PService;
use eth2_libp2p::Topic;
use eth2_libp2p::{Libp2pEvent, PeerId};
use eth2_libp2p::{Enr, Libp2pEvent, Multiaddr, PeerId, Swarm};
use eth2_libp2p::{PubsubMessage, RPCEvent};
use futures::prelude::*;
use futures::Stream;
@ -18,6 +18,7 @@ use tokio::sync::{mpsc, oneshot};
/// Service that handles communication between internal services and the eth2_libp2p network service.
pub struct Service<T: BeaconChainTypes> {
libp2p_service: Arc<Mutex<LibP2PService>>,
libp2p_port: u16,
_libp2p_exit: oneshot::Sender<()>,
_network_send: mpsc::UnboundedSender<NetworkMessage>,
_phantom: PhantomData<T>, //message_handler: MessageHandler,
@ -56,6 +57,7 @@ impl<T: BeaconChainTypes + 'static> Service<T> {
)?;
let network_service = Service {
libp2p_service,
libp2p_port: config.libp2p_port,
_libp2p_exit: libp2p_exit,
_network_send: network_send.clone(),
_phantom: PhantomData,
@ -64,6 +66,52 @@ impl<T: BeaconChainTypes + 'static> Service<T> {
Ok((Arc::new(network_service), network_send))
}
/// Returns the local ENR from the underlying Discv5 behaviour that external peers may connect
/// to.
pub fn local_enr(&self) -> Enr {
self.libp2p_service
.lock()
.swarm
.discovery()
.local_enr()
.clone()
}
/// Returns the local libp2p PeerID.
pub fn local_peer_id(&self) -> PeerId {
self.libp2p_service.lock().local_peer_id.clone()
}
/// Returns the list of `Multiaddr` that the underlying libp2p instance is listening on.
pub fn listen_multiaddrs(&self) -> Vec<Multiaddr> {
Swarm::listeners(&self.libp2p_service.lock().swarm)
.cloned()
.collect()
}
/// Returns the libp2p port that this node has been configured to listen using.
pub fn listen_port(&self) -> u16 {
self.libp2p_port
}
/// Returns the number of libp2p connected peers.
pub fn connected_peers(&self) -> usize {
self.libp2p_service.lock().swarm.connected_peers()
}
/// Returns the set of `PeerId` that are connected via libp2p.
pub fn connected_peer_set(&self) -> Vec<PeerId> {
self.libp2p_service
.lock()
.swarm
.discovery()
.connected_peer_set()
.iter()
.cloned()
.collect()
}
/// Provides a reference to the underlying libp2p service.
pub fn libp2p_service(&self) -> Arc<Mutex<LibP2PService>> {
self.libp2p_service.clone()
}

2
beacon_node/rest_api/Cargo.toml
View File

@ -7,6 +7,8 @@ edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
beacon_chain = { path = "../beacon_chain" }
network = { path = "../network" }
eth2-libp2p = { path = "../eth2-libp2p" }
store = { path = "../store" }
version = { path = "../version" }
serde = { version = "1.0", features = ["derive"] }

150
beacon_node/rest_api/src/beacon.rs
View File

@ -2,9 +2,114 @@ use super::{success_response, ApiResult};
use crate::{helpers::*, ApiError, UrlQuery};
use beacon_chain::{BeaconChain, BeaconChainTypes};
use hyper::{Body, Request};
use serde::Serialize;
use std::sync::Arc;
use store::Store;
use types::BeaconState;
use types::{BeaconBlock, BeaconState, EthSpec, Hash256, Slot};
#[derive(Serialize)]
pub struct HeadResponse {
pub slot: Slot,
pub block_root: Hash256,
pub state_root: Hash256,
}
/// HTTP handler to return a `BeaconBlock` at a given `root` or `slot`.
pub fn get_head<T: BeaconChainTypes + 'static>(req: Request<Body>) -> ApiResult {
let beacon_chain = req
.extensions()
.get::<Arc<BeaconChain<T>>>()
.ok_or_else(|| ApiError::ServerError("Beacon chain extension missing".to_string()))?;
let head = HeadResponse {
slot: beacon_chain.head().beacon_state.slot,
block_root: beacon_chain.head().beacon_block_root,
state_root: beacon_chain.head().beacon_state_root,
};
let json: String = serde_json::to_string(&head)
.map_err(|e| ApiError::ServerError(format!("Unable to serialize HeadResponse: {:?}", e)))?;
Ok(success_response(Body::from(json)))
}
#[derive(Serialize)]
#[serde(bound = "T: EthSpec")]
pub struct BlockResponse<T: EthSpec> {
pub root: Hash256,
pub beacon_block: BeaconBlock<T>,
}
/// HTTP handler to return a `BeaconBlock` at a given `root` or `slot`.
pub fn get_block<T: BeaconChainTypes + 'static>(req: Request<Body>) -> ApiResult {
let beacon_chain = req
.extensions()
.get::<Arc<BeaconChain<T>>>()
.ok_or_else(|| ApiError::ServerError("Beacon chain extension missing".to_string()))?;
let query_params = ["root", "slot"];
let (key, value) = UrlQuery::from_request(&req)?.first_of(&query_params)?;
let block_root = match (key.as_ref(), value) {
("slot", value) => {
let target = parse_slot(&value)?;
block_root_at_slot(&beacon_chain, target).ok_or_else(|| {
ApiError::NotFound(format!("Unable to find BeaconBlock for slot {}", target))
})?
}
("root", value) => parse_root(&value)?,
_ => return Err(ApiError::ServerError("Unexpected query parameter".into())),
};
let block = beacon_chain
.store
.get::<BeaconBlock<T::EthSpec>>(&block_root)?
.ok_or_else(|| {
ApiError::NotFound(format!(
"Unable to find BeaconBlock for root {}",
block_root
))
})?;
let response = BlockResponse {
root: block_root,
beacon_block: block,
};
let json: String = serde_json::to_string(&response).map_err(|e| {
ApiError::ServerError(format!("Unable to serialize BlockResponse: {:?}", e))
})?;
Ok(success_response(Body::from(json)))
}
/// HTTP handler to return a `BeaconBlock` root at a given `slot`.
pub fn get_block_root<T: BeaconChainTypes + 'static>(req: Request<Body>) -> ApiResult {
let beacon_chain = req
.extensions()
.get::<Arc<BeaconChain<T>>>()
.ok_or_else(|| ApiError::ServerError("Beacon chain extension missing".to_string()))?;
let slot_string = UrlQuery::from_request(&req)?.only_one("slot")?;
let target = parse_slot(&slot_string)?;
let root = block_root_at_slot(&beacon_chain, target).ok_or_else(|| {
ApiError::NotFound(format!("Unable to find BeaconBlock for slot {}", target))
})?;
let json: String = serde_json::to_string(&root)
.map_err(|e| ApiError::ServerError(format!("Unable to serialize root: {:?}", e)))?;
Ok(success_response(Body::from(json)))
}
#[derive(Serialize)]
#[serde(bound = "T: EthSpec")]
pub struct StateResponse<T: EthSpec> {
pub root: Hash256,
pub beacon_state: BeaconState<T>,
}
/// HTTP handler to return a `BeaconState` at a given `root` or `slot`.
///
@ -19,26 +124,34 @@ pub fn get_state<T: BeaconChainTypes + 'static>(req: Request<Body>) -> ApiResult
let query_params = ["root", "slot"];
let (key, value) = UrlQuery::from_request(&req)?.first_of(&query_params)?;
let state: BeaconState<T::EthSpec> = match (key.as_ref(), value) {
let (root, state): (Hash256, BeaconState<T::EthSpec>) = match (key.as_ref(), value) {
("slot", value) => state_at_slot(&beacon_chain, parse_slot(&value)?)?,
("root", value) => {
let root = &parse_root(&value)?;
beacon_chain
let state = beacon_chain
.store
.get(root)?
.ok_or_else(|| ApiError::NotFound(format!("No state for root: {}", root)))?
.ok_or_else(|| ApiError::NotFound(format!("No state for root: {}", root)))?;
(*root, state)
}
_ => unreachable!("Guarded by UrlQuery::from_request()"),
_ => return Err(ApiError::ServerError("Unexpected query parameter".into())),
};
let json: String = serde_json::to_string(&state)
.map_err(|e| ApiError::ServerError(format!("Unable to serialize BeaconState: {:?}", e)))?;
let response = StateResponse {
root,
beacon_state: state,
};
let json: String = serde_json::to_string(&response).map_err(|e| {
ApiError::ServerError(format!("Unable to serialize StateResponse: {:?}", e))
})?;
Ok(success_response(Body::from(json)))
}
/// HTTP handler to return a `BeaconState` root at a given or `slot`.
/// HTTP handler to return a `BeaconState` root at a given `slot`.
///
/// Will not return a state if the request slot is in the future. Will return states higher than
/// the current head by skipping slots.
@ -58,3 +171,24 @@ pub fn get_state_root<T: BeaconChainTypes + 'static>(req: Request<Body>) -> ApiR
Ok(success_response(Body::from(json)))
}
/// HTTP handler to return the highest finalized slot.
pub fn get_latest_finalized_checkpoint<T: BeaconChainTypes + 'static>(
req: Request<Body>,
) -> ApiResult {
let beacon_chain = req
.extensions()
.get::<Arc<BeaconChain<T>>>()
.ok_or_else(|| ApiError::ServerError("Beacon chain extension missing".to_string()))?;
let checkpoint = beacon_chain
.head()
.beacon_state
.finalized_checkpoint
.clone();
let json: String = serde_json::to_string(&checkpoint)
.map_err(|e| ApiError::ServerError(format!("Unable to serialize checkpoint: {:?}", e)))?;
Ok(success_response(Body::from(json)))
}

28
beacon_node/rest_api/src/helpers.rs
View File

@ -31,22 +31,40 @@ pub fn parse_root(string: &str) -> Result<Hash256, ApiError> {
}
}
/// Returns a `BeaconState` in the canonical chain of `beacon_chain` at the given `slot`, if
/// possible.
/// Returns the root of the `BeaconBlock` in the canonical chain of `beacon_chain` at the given
/// `slot`, if possible.
///
/// May return a root for a previous slot, in the case of skip slots.
pub fn block_root_at_slot<T: BeaconChainTypes>(
beacon_chain: &BeaconChain<T>,
target: Slot,
) -> Option<Hash256> {
beacon_chain
.rev_iter_block_roots()
.take_while(|(_root, slot)| *slot >= target)
.find(|(_root, slot)| *slot == target)
.map(|(root, _slot)| root)
}
/// Returns a `BeaconState` and it's root in the canonical chain of `beacon_chain` at the given
/// `slot`, if possible.
///
/// Will not return a state if the request slot is in the future. Will return states higher than
/// the current head by skipping slots.
pub fn state_at_slot<T: BeaconChainTypes>(
beacon_chain: &BeaconChain<T>,
slot: Slot,
) -> Result<BeaconState<T::EthSpec>, ApiError> {
) -> Result<(Hash256, BeaconState<T::EthSpec>), ApiError> {
let head_state = &beacon_chain.head().beacon_state;
if head_state.slot == slot {
// The request slot is the same as the best block (head) slot.
// I'm not sure if this `.clone()` will be optimized out. If not, it seems unnecessary.
Ok(beacon_chain.head().beacon_state.clone())
Ok((
beacon_chain.head().beacon_state_root,
beacon_chain.head().beacon_state.clone(),
))
} else {
let root = state_root_at_slot(beacon_chain, slot)?;
@ -55,7 +73,7 @@ pub fn state_at_slot<T: BeaconChainTypes>(
.get(&root)?
.ok_or_else(|| ApiError::NotFound(format!("Unable to find state at root {}", root)))?;
Ok(state)
Ok((root, state))
}
}

30
beacon_node/rest_api/src/lib.rs
View File

@ -1,15 +1,18 @@
#[macro_use]
extern crate lazy_static;
extern crate network as client_network;
mod beacon;
mod config;
mod helpers;
mod metrics;
mod network;
mod node;
mod spec;
mod url_query;
use beacon_chain::{BeaconChain, BeaconChainTypes};
pub use config::Config as ApiConfig;
use client_network::Service as NetworkService;
use hyper::rt::Future;
use hyper::service::service_fn_ok;
use hyper::{Body, Method, Response, Server, StatusCode};
@ -20,6 +23,9 @@ use std::sync::Arc;
use tokio::runtime::TaskExecutor;
use url_query::UrlQuery;
pub use beacon::{BlockResponse, HeadResponse, StateResponse};
pub use config::Config as ApiConfig;
#[derive(PartialEq, Debug)]
pub enum ApiError {
MethodNotAllowed(String),
@ -67,10 +73,11 @@ impl From<state_processing::per_slot_processing::Error> for ApiError {
}
}
pub fn start_server<T: BeaconChainTypes + Clone + 'static>(
pub fn start_server<T: BeaconChainTypes>(
config: &ApiConfig,
executor: &TaskExecutor,
beacon_chain: Arc<BeaconChain<T>>,
network_service: Arc<NetworkService<T>>,
db_path: PathBuf,
log: &slog::Logger,
) -> Result<exit_future::Signal, hyper::Error> {
@ -98,6 +105,7 @@ pub fn start_server<T: BeaconChainTypes + Clone + 'static>(
let log = server_log.clone();
let beacon_chain = server_bc.clone();
let db_path = db_path.clone();
let network_service = network_service.clone();
// Create a simple handler for the router, inject our stateful objects into the request.
service_fn_ok(move |mut req| {
@ -108,16 +116,34 @@ pub fn start_server<T: BeaconChainTypes + Clone + 'static>(
req.extensions_mut()
.insert::<Arc<BeaconChain<T>>>(beacon_chain.clone());
req.extensions_mut().insert::<DBPath>(db_path.clone());
req.extensions_mut()
.insert::<Arc<NetworkService<T>>>(network_service.clone());
let path = req.uri().path().to_string();
// Route the request to the correct handler.
let result = match (req.method(), path.as_ref()) {
(&Method::GET, "/beacon/head") => beacon::get_head::<T>(req),
(&Method::GET, "/beacon/block") => beacon::get_block::<T>(req),
(&Method::GET, "/beacon/block_root") => beacon::get_block_root::<T>(req),
(&Method::GET, "/beacon/latest_finalized_checkpoint") => {
beacon::get_latest_finalized_checkpoint::<T>(req)
}
(&Method::GET, "/beacon/state") => beacon::get_state::<T>(req),
(&Method::GET, "/beacon/state_root") => beacon::get_state_root::<T>(req),
(&Method::GET, "/metrics") => metrics::get_prometheus::<T>(req),
(&Method::GET, "/network/enr") => network::get_enr::<T>(req),
(&Method::GET, "/network/peer_count") => network::get_peer_count::<T>(req),
(&Method::GET, "/network/peer_id") => network::get_peer_id::<T>(req),
(&Method::GET, "/network/peers") => network::get_peer_list::<T>(req),
(&Method::GET, "/network/listen_port") => network::get_listen_port::<T>(req),
(&Method::GET, "/network/listen_addresses") => {
network::get_listen_addresses::<T>(req)
}
(&Method::GET, "/node/version") => node::get_version(req),
(&Method::GET, "/node/genesis_time") => node::get_genesis_time::<T>(req),
(&Method::GET, "/spec") => spec::get_spec::<T>(req),
(&Method::GET, "/spec/slots_per_epoch") => spec::get_slots_per_epoch::<T>(req),
_ => Err(ApiError::MethodNotAllowed(path.clone())),
};

108
beacon_node/rest_api/src/network.rs Normal file
View File

@ -0,0 +1,108 @@
use crate::{success_response, ApiError, ApiResult, NetworkService};
use beacon_chain::BeaconChainTypes;
use eth2_libp2p::{Enr, Multiaddr, PeerId};
use hyper::{Body, Request};
use std::sync::Arc;
/// HTTP handle to return the list of libp2p multiaddr the client is listening on.
///
/// Returns a list of `Multiaddr`, serialized according to their `serde` impl.
pub fn get_listen_addresses<T: BeaconChainTypes>(req: Request<Body>) -> ApiResult {
let network = req
.extensions()
.get::<Arc<NetworkService<T>>>()
.ok_or_else(|| ApiError::ServerError("NetworkService extension missing".to_string()))?;
let multiaddresses: Vec<Multiaddr> = network.listen_multiaddrs();
Ok(success_response(Body::from(
serde_json::to_string(&multiaddresses)
.map_err(|e| ApiError::ServerError(format!("Unable to serialize Enr: {:?}", e)))?,
)))
}
/// HTTP handle to return the list of libp2p multiaddr the client is listening on.
///
/// Returns a list of `Multiaddr`, serialized according to their `serde` impl.
pub fn get_listen_port<T: BeaconChainTypes>(req: Request<Body>) -> ApiResult {
let network = req
.extensions()
.get::<Arc<NetworkService<T>>>()
.ok_or_else(|| ApiError::ServerError("NetworkService extension missing".to_string()))?;
Ok(success_response(Body::from(
serde_json::to_string(&network.listen_port())
.map_err(|e| ApiError::ServerError(format!("Unable to serialize port: {:?}", e)))?,
)))
}
/// HTTP handle to return the Discv5 ENR from the client's libp2p service.
///
/// ENR is encoded as base64 string.
pub fn get_enr<T: BeaconChainTypes>(req: Request<Body>) -> ApiResult {
let network = req
.extensions()
.get::<Arc<NetworkService<T>>>()
.ok_or_else(|| ApiError::ServerError("NetworkService extension missing".to_string()))?;
let enr: Enr = network.local_enr();
Ok(success_response(Body::from(
serde_json::to_string(&enr.to_base64())
.map_err(|e| ApiError::ServerError(format!("Unable to serialize Enr: {:?}", e)))?,
)))
}
/// HTTP handle to return the `PeerId` from the client's libp2p service.
///
/// PeerId is encoded as base58 string.
pub fn get_peer_id<T: BeaconChainTypes>(req: Request<Body>) -> ApiResult {
let network = req
.extensions()
.get::<Arc<NetworkService<T>>>()
.ok_or_else(|| ApiError::ServerError("NetworkService extension missing".to_string()))?;
let peer_id: PeerId = network.local_peer_id();
Ok(success_response(Body::from(
serde_json::to_string(&peer_id.to_base58())
.map_err(|e| ApiError::ServerError(format!("Unable to serialize Enr: {:?}", e)))?,
)))
}
/// HTTP handle to return the number of peers connected in the client's libp2p service.
pub fn get_peer_count<T: BeaconChainTypes>(req: Request<Body>) -> ApiResult {
let network = req
.extensions()
.get::<Arc<NetworkService<T>>>()
.ok_or_else(|| ApiError::ServerError("NetworkService extension missing".to_string()))?;
let connected_peers: usize = network.connected_peers();
Ok(success_response(Body::from(
serde_json::to_string(&connected_peers)
.map_err(|e| ApiError::ServerError(format!("Unable to serialize Enr: {:?}", e)))?,
)))
}
/// HTTP handle to return the list of peers connected to the client's libp2p service.
///
/// Peers are presented as a list of `PeerId::to_string()`.
pub fn get_peer_list<T: BeaconChainTypes>(req: Request<Body>) -> ApiResult {
let network = req
.extensions()
.get::<Arc<NetworkService<T>>>()
.ok_or_else(|| ApiError::ServerError("NetworkService extension missing".to_string()))?;
let connected_peers: Vec<String> = network
.connected_peer_set()
.iter()
.map(PeerId::to_string)
.collect();
Ok(success_response(Body::from(
serde_json::to_string(&connected_peers).map_err(|e| {
ApiError::ServerError(format!("Unable to serialize Vec<PeerId>: {:?}", e))
})?,
)))
}

27
beacon_node/rest_api/src/spec.rs Normal file
View File

@ -0,0 +1,27 @@
use super::{success_response, ApiResult};
use crate::ApiError;
use beacon_chain::{BeaconChain, BeaconChainTypes};
use hyper::{Body, Request};
use std::sync::Arc;
use types::EthSpec;
/// HTTP handler to return the full spec object.
pub fn get_spec<T: BeaconChainTypes + 'static>(req: Request<Body>) -> ApiResult {
let beacon_chain = req
.extensions()
.get::<Arc<BeaconChain<T>>>()
.ok_or_else(|| ApiError::ServerError("Beacon chain extension missing".to_string()))?;
let json: String = serde_json::to_string(&beacon_chain.spec)
.map_err(|e| ApiError::ServerError(format!("Unable to serialize spec: {:?}", e)))?;
Ok(success_response(Body::from(json)))
}
/// HTTP handler to return the full spec object.
pub fn get_slots_per_epoch<T: BeaconChainTypes + 'static>(_req: Request<Body>) -> ApiResult {
let json: String = serde_json::to_string(&T::EthSpec::slots_per_epoch())
.map_err(|e| ApiError::ServerError(format!("Unable to serialize epoch: {:?}", e)))?;
Ok(success_response(Body::from(json)))
}

15
beacon_node/src/main.rs
View File

@ -200,6 +200,16 @@ fn main() {
.help("Sets the verbosity level")
.takes_value(true),
)
/*
* Bootstrap.
*/
.arg(
Arg::with_name("bootstrap")
.long("bootstrap")
.value_name("HTTP_SERVER")
.help("Load the genesis state and libp2p address from the HTTP API of another Lighthouse node.")
.takes_value(true)
)
.get_matches();
// build the initial logger
@ -227,6 +237,11 @@ fn main() {
let mut log = slog::Logger::root(drain.fuse(), o!());
warn!(
log,
"Ethereum 2.0 is pre-release. This software is experimental."
);
let data_dir = match matches
.value_of("datadir")
.and_then(|v| Some(PathBuf::from(v)))

9
beacon_node/src/run.rs
View File

@ -4,7 +4,7 @@ use client::{
};
use futures::sync::oneshot;
use futures::Future;
use slog::{error, info, warn};
use slog::{error, info};
use std::cell::RefCell;
use std::path::Path;
use std::path::PathBuf;
@ -42,11 +42,6 @@ pub fn run_beacon_node(
let other_client_config = client_config.clone();
warn!(
log,
"Ethereum 2.0 is pre-release. This software is experimental."
);
info!(
log,
"BeaconNode init";
@ -123,7 +118,7 @@ fn run<T>(
log: &slog::Logger,
) -> error::Result<()>
where
T: BeaconChainTypes + InitialiseBeaconChain<T> + Clone + Send + Sync + 'static,
T: BeaconChainTypes + InitialiseBeaconChain<T> + Clone,
T::Store: OpenDatabase,
{
let store = T::Store::open_database(&db_path)?;

7
eth2/types/src/beacon_block_body.rs
View File

@ -1,5 +1,5 @@
use crate::test_utils::TestRandom;
use crate::utils::graffiti_from_hex_str;
use crate::utils::{graffiti_from_hex_str, graffiti_to_hex_str};
use crate::*;
use serde_derive::{Deserialize, Serialize};
@ -16,7 +16,10 @@ use tree_hash_derive::TreeHash;
pub struct BeaconBlockBody<T: EthSpec> {
pub randao_reveal: Signature,
pub eth1_data: Eth1Data,
#[serde(deserialize_with = "graffiti_from_hex_str")]
#[serde(
serialize_with = "graffiti_to_hex_str",
deserialize_with = "graffiti_from_hex_str"
)]
pub graffiti: [u8; 32],
pub proposer_slashings: VariableList<ProposerSlashing, T::MaxProposerSlashings>,
pub attester_slashings: VariableList<AttesterSlashing<T>, T::MaxAttesterSlashings>,

16
eth2/types/src/utils/serde_utils.rs
View File

@ -46,8 +46,20 @@ where
Ok(array)
}
// #[allow(clippy::trivially_copy_pass_by_ref)] // Serde requires the `byte` to be a ref.
pub fn fork_to_hex_str<S>(bytes: &[u8; 4], serializer: S) -> Result<S::Ok, S::Error>
pub fn fork_to_hex_str<S>(bytes: &[u8; FORK_BYTES_LEN], serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
let mut hex_string: String = "0x".to_string();
hex_string.push_str(&hex::encode(&bytes));
serializer.serialize_str(&hex_string)
}
pub fn graffiti_to_hex_str<S>(
bytes: &[u8; GRAFFITI_BYTES_LEN],
serializer: S,
) -> Result<S::Ok, S::Error>
where
S: Serializer,
{