From 44fa54004c86fe819647a5573540019a8ab38517 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Tue, 31 Aug 2021 04:48:21 +0000 Subject: [PATCH] Persist to DB after setting canonical head (#2547) ## Issue Addressed NA ## Proposed Changes Missed head votes on attestations is a well-known issue. The primary cause is a block getting set as the head *after* the attestation deadline. This PR aims to shorten the overall time between "block received" and "block set as head" by: 1. Persisting the head and fork choice *after* setting the canonical head - Informal measurements show this takes ~200ms 1. Pruning the op pool *after* setting the canonical head. 1. No longer persisting the op pool to disk during `BeaconChain::fork_choice` - Informal measurements show this can take up to 1.2s. I also add some metrics to help measure the effect of these changes. Persistence changes like this run the risk of breaking assumptions downstream. However, I have considered these risks and I think we're fine here. I will describe my reasoning for each change. ## Reasoning ### Change 1: Persisting the head and fork choice *after* setting the canonical head For (1), although the function is called `persist_head_and_fork_choice`, it only persists: - Fork choice - Head tracker - Genesis block root Since `BeaconChain::fork_choice_internal` does not modify these values between the original time we were persisting it and the current time, I assert that the change I've made is non-substantial in terms of what ends up on-disk. There's the possibility that some *other* thread has modified fork choice in the extra time we've given it, but that's totally fine. Since the only time we *read* those values from disk is during startup, I assert that this has no impact during runtime. ### Change 2: Pruning the op pool after setting the canonical head Similar to the argument above, we don't modify the op pool during `BeaconChain::fork_choice_internal` so it shouldn't matter when we prune. This change should be non-substantial. ### Change 3: No longer persisting the op pool to disk during `BeaconChain::fork_choice` This change *is* substantial. With the proposed changes, we'll only be persisting the op pool to disk when we shut down cleanly (i.e., the `BeaconChain` gets dropped). This means we'll save disk IO and time during usual operation, but a `kill -9` or similar "crash" will probably result in an out-of-date op pool when we reboot. An out-of-date op pool can only have an impact when producing blocks or aggregate attestations/sync committees. I think it's pretty reasonable that a crash might result in an out-of-date op pool, since: - Crashes are fairly rare. Practically the only time I see LH suffer a full crash is when the OOM killer shows up, and that's a very serious event. - It's generally quite rare to produce a block/aggregate immediately after a reboot. Just a few slots of runtime is probably enough to have a decent-enough op pool again. ## Additional Info Credits to @macladson for the timings referenced here. --- beacon_node/beacon_chain/src/beacon_chain.rs | 15 +++++++++------ beacon_node/beacon_chain/src/metrics.rs | 4 ++++ 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/beacon_node/beacon_chain/src/beacon_chain.rs b/beacon_node/beacon_chain/src/beacon_chain.rs index 2103542de..6b514d569 100644 --- a/beacon_node/beacon_chain/src/beacon_chain.rs +++ b/beacon_node/beacon_chain/src/beacon_chain.rs @@ -2810,6 +2810,8 @@ impl BeaconChain { return Ok(()); } + let lag_timer = metrics::start_timer(&metrics::FORK_CHOICE_SET_HEAD_LAG_TIMES); + // At this point we know that the new head block is not the same as the previous one metrics::inc_counter(&metrics::FORK_CHOICE_CHANGED_HEAD); @@ -2913,12 +2915,6 @@ impl BeaconChain { .slot() .epoch(T::EthSpec::slots_per_epoch()); - if is_epoch_transition || is_reorg { - self.persist_head_and_fork_choice()?; - self.op_pool.prune_attestations(self.epoch()?); - self.persist_op_pool()?; - } - let update_head_timer = metrics::start_timer(&metrics::UPDATE_HEAD_TIMES); // These fields are used for server-sent events @@ -2934,6 +2930,8 @@ impl BeaconChain { .start_slot(T::EthSpec::slots_per_epoch()); let head_proposer_index = new_head.beacon_block.message().proposer_index(); + drop(lag_timer); + // Update the snapshot that stores the head of the chain at the time it received the // block. *self @@ -2984,6 +2982,11 @@ impl BeaconChain { ); }); + if is_epoch_transition || is_reorg { + self.persist_head_and_fork_choice()?; + self.op_pool.prune_attestations(self.epoch()?); + } + if new_finalized_checkpoint.epoch != old_finalized_checkpoint.epoch { // Due to race conditions, it's technically possible that the head we load here is // different to the one earlier in this function. diff --git a/beacon_node/beacon_chain/src/metrics.rs b/beacon_node/beacon_chain/src/metrics.rs index 6b27dfcfc..ebedef992 100644 --- a/beacon_node/beacon_chain/src/metrics.rs +++ b/beacon_node/beacon_chain/src/metrics.rs @@ -261,6 +261,10 @@ lazy_static! { "beacon_fork_choice_process_attestation_seconds", "Time taken to add an attestation to fork choice" ); + pub static ref FORK_CHOICE_SET_HEAD_LAG_TIMES: Result = try_create_histogram( + "beacon_fork_choice_set_head_lag_times", + "Time taken between finding the head and setting the canonical head value" + ); pub static ref BALANCES_CACHE_HITS: Result = try_create_int_counter("beacon_balances_cache_hits_total", "Count of times balances cache fulfils request"); pub static ref BALANCES_CACHE_MISSES: Result =