54cf94ea59
## Issue Addressed
Fixes a timing issue that results in spurious fork choice notifier failures:
```
WARN Error signalling fork choice waiter slot: 3962270, error: ForkChoiceSignalOutOfOrder { current: Slot(3962271), latest: Slot(3962270) }, service: beacon
```
There’s a fork choice run that is scheduled to run at the start of every slot by the `timer`, which creates a 12s interval timer when the beacon node starts up. The problem is that if there’s a bit of clock drift that gets corrected via NTP (or a leap second for that matter) then these 12s intervals will cease to line up with the start of the slot. This then creates the mismatch in slot number that we see above.
Lighthouse also runs fork choice 500ms before the slot begins, and these runs are what is conflicting with the start-of-slot runs. This means that the warning in current versions of Lighthouse is mostly cosmetic because fork choice is up to date with all but the most recent 500ms of attestations (which usually isn’t many).
## Proposed Changes
Fix the per-slot timer so that it continually re-calculates the duration to the start of the next slot and waits for that.
A side-effect of this change is that we may skip slots if the per-slot task takes >12s to run, but I think this is an unlikely scenario and an acceptable compromise.
59 lines
1.8 KiB
Rust
59 lines
1.8 KiB
Rust
//! A timer service for the beacon node.
|
|
//!
|
|
//! This service allows task execution on the beacon node for various functionality.
|
|
|
|
use beacon_chain::{BeaconChain, BeaconChainTypes};
|
|
use slog::{debug, info, warn};
|
|
use slot_clock::SlotClock;
|
|
use std::sync::Arc;
|
|
use tokio::time::sleep;
|
|
|
|
/// Spawns a timer service which periodically executes tasks for the beacon chain
|
|
pub fn spawn_timer<T: BeaconChainTypes>(
|
|
executor: task_executor::TaskExecutor,
|
|
beacon_chain: Arc<BeaconChain<T>>,
|
|
) -> Result<(), &'static str> {
|
|
let log = executor.log();
|
|
let per_slot_executor = executor.clone();
|
|
|
|
let timer_future = async move {
|
|
let log = per_slot_executor.log().clone();
|
|
loop {
|
|
let duration_to_next_slot = match beacon_chain.slot_clock.duration_to_next_slot() {
|
|
Some(duration) => duration,
|
|
None => {
|
|
warn!(log, "Unable to determine duration to next slot");
|
|
return;
|
|
}
|
|
};
|
|
|
|
sleep(duration_to_next_slot).await;
|
|
|
|
let chain = beacon_chain.clone();
|
|
if let Some(handle) = per_slot_executor
|
|
.spawn_blocking_handle(move || chain.per_slot_task(), "timer_per_slot_task")
|
|
{
|
|
if let Err(e) = handle.await {
|
|
warn!(
|
|
log,
|
|
"Per slot task failed";
|
|
"info" => ?e
|
|
);
|
|
}
|
|
} else {
|
|
debug!(
|
|
log,
|
|
"Per slot task timer stopped";
|
|
"info" => "shutting down"
|
|
);
|
|
break;
|
|
}
|
|
}
|
|
};
|
|
|
|
executor.spawn(timer_future, "timer");
|
|
info!(log, "Timer service started");
|
|
|
|
Ok(())
|
|
}
|