lighthouse/beacon_node/beacon_chain/src
Michael Sproul 01556f6f01 Optimise payload attributes calculation and add SSE (#4027)
## Issue Addressed

Closes #3896
Closes #3998
Closes #3700

## Proposed Changes

- Optimise the calculation of withdrawals for payload attributes by avoiding state clones, avoiding unnecessary state advances and reading from the snapshot cache if possible.
- Use the execution layer's payload attributes cache to avoid re-calculating payload attributes. I actually implemented a new LRU cache just for withdrawals but it had the exact same key and most of the same data as the existing payload attributes cache, so I deleted it.
- Add a new SSE event that fires when payloadAttributes are calculated. This is useful for block builders, a la https://github.com/ethereum/beacon-APIs/issues/244.
- Add a new CLI flag `--always-prepare-payload` which forces payload attributes to be sent with every fcU regardless of connected proposers. This is intended for use by builders/relays.

For maximum effect, the flags I've been using to run Lighthouse in "payload builder mode" are:

```
--always-prepare-payload \
--prepare-payload-lookahead 12000 \
--suggested-fee-recipient 0x0000000000000000000000000000000000000000
```

The fee recipient is required so Lighthouse has something to pack in the payload attributes (it can be ignored by the builder). The lookahead causes fcU to be sent at the start of every slot rather than at 8s. As usual, fcU will also be sent after each change of head block. I think this combination is sufficient for builders to build on all viable heads. Often there will be two fcU (and two payload attributes) sent for the same slot: one sent at the start of the slot with the head from `n - 1` as the parent, and one sent after the block arrives with `n` as the parent.

Example usage of the new event stream:

```bash
curl -N "http://localhost:5052/eth/v1/events?topics=payload_attributes"
```

## Additional Info

- [x] Tests added by updating the proposer re-org tests. This has the benefit of testing the proposer re-org code paths with withdrawals too, confirming that the new changes don't interact poorly.
- [ ] Benchmarking with `blockdreamer` on devnet-7 showed promising results but I'm yet to do a comparison to `unstable`.


Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-03-05 23:43:30 +00:00
..
attestation_verification Use async code when interacting with EL (#3244) 2022-07-03 05:36:50 +00:00
schema_change Fix Capella schema downgrades (#4004) 2023-02-20 17:50:42 +11:00
attestation_rewards.rs Tweaks to reward APIs (#3957) 2023-02-10 06:19:42 +00:00
attestation_verification.rs Clippy 1.67 (#3916) 2023-01-27 09:48:42 +00:00
attester_cache.rs Add early attester cache (#2872) 2022-01-11 01:35:55 +00:00
beacon_block_reward.rs Update block rewards API for Capella 2023-02-14 12:09:40 +11:00
beacon_chain.rs Optimise payload attributes calculation and add SSE (#4027) 2023-03-05 23:43:30 +00:00
beacon_fork_choice_store.rs Merge remote-tracking branch 'origin/unstable' into capella-update 2022-12-14 13:00:41 +11:00
beacon_proposer_cache.rs Use async code when interacting with EL (#3244) 2022-07-03 05:36:50 +00:00
beacon_snapshot.rs Capella eip 4844 cleanup (#3652) 2022-10-26 15:15:26 -04:00
block_reward.rs Capella eip 4844 cleanup (#3652) 2022-10-26 15:15:26 -04:00
block_times_cache.rs Add BlockTimesCache to allow additional block delay metrics (#2546) 2021-09-30 04:31:41 +00:00
block_verification.rs Merge remote-tracking branch 'origin/unstable' into capella-merge 2023-02-14 12:07:27 +11:00
builder.rs Merge remote-tracking branch 'origin/unstable' into capella 2023-01-12 16:22:00 +11:00
canonical_head.rs Merge remote-tracking branch 'origin/unstable' into capella-update 2022-12-14 13:00:41 +11:00
capella_readiness.rs Suggestions for Capella beacon_chain (#3999) 2023-02-21 11:05:36 +11:00
chain_config.rs Optimise payload attributes calculation and add SSE (#4027) 2023-03-05 23:43:30 +00:00
early_attester_cache.rs Fix some typos (#3376) 2022-07-27 00:51:06 +00:00
errors.rs Clean capella (#4019) 2023-03-01 03:19:02 +00:00
eth1_chain.rs Deposit Cache Finalization & Fast WS Sync (#2915) 2022-10-30 04:04:24 +00:00
eth1_finalization_cache.rs Deposit Cache Finalization & Fast WS Sync (#2915) 2022-10-30 04:04:24 +00:00
events.rs Optimise payload attributes calculation and add SSE (#4027) 2023-03-05 23:43:30 +00:00
execution_payload.rs Permit a null LVH from an INVALID response to newPayload (#4037) 2023-03-03 04:12:50 +00:00
fork_choice_signal.rs Clippy 1.67 (#3916) 2023-01-27 09:48:42 +00:00
fork_revert.rs Enable proposer boost re-orging (#2860) 2022-12-13 09:57:26 +00:00
head_tracker.rs Fix rust 1.65 lints (#3682) 2022-11-04 07:43:43 +00:00
historical_blocks.rs Use async code when interacting with EL (#3244) 2022-07-03 05:36:50 +00:00
lib.rs Clean capella (#4019) 2023-03-01 03:19:02 +00:00
light_client_finality_update_verification.rs Adding light_client gossip topics (#3693) 2022-12-13 06:24:51 +00:00
light_client_optimistic_update_verification.rs light client optimistic update reprocessing (#3799) 2023-01-24 22:17:50 +00:00
merge_readiness.rs exchangeCapabilities & Capella Readiness Logging (#3918) 2023-01-31 18:26:23 +01:00
metrics.rs Clean capella (#4019) 2023-03-01 03:19:02 +00:00
migrate.rs Fix rust 1.65 lints (#3682) 2022-11-04 07:43:43 +00:00
naive_aggregation_pool.rs Clippy lints for rust 1.66 (#3810) 2022-12-16 04:04:00 +00:00
observed_aggregates.rs v2.2.0 (#3139) 2022-04-05 02:53:09 +00:00
observed_attesters.rs Ensure doppelganger detects attestations in blocks (#2495) 2021-08-09 02:43:03 +00:00
observed_block_producers.rs Doppelganger detection (#2230) 2021-07-31 03:50:52 +00:00
observed_operations.rs Isolate withdrawals-processing Feature (#3854) 2023-01-09 11:05:28 +11:00
otb_verification_service.rs Initial Commit of Retrospective OTB Verification (#3372) 2022-07-30 00:22:38 +00:00
persisted_beacon_chain.rs Fix head tracker concurrency bugs (#1771) 2020-10-19 05:58:39 +00:00
persisted_fork_choice.rs Merge 'upstream/unstable' into capella (#3773) 2022-12-03 14:05:25 -06:00
pre_finalization_cache.rs Separate execution payloads in the DB (#3157) 2022-05-12 00:42:17 +00:00
proposer_prep_service.rs Enable proposer boost re-orging (#2860) 2022-12-13 09:57:26 +00:00
schema_change.rs Broadcast address changes at Capella (#3919) 2023-02-07 17:13:49 +11:00
shuffling_cache.rs Impl oneshot_broadcast for committee promises (#3595) 2022-09-21 01:01:50 +00:00
snapshot_cache.rs Enable proposer boost re-orging (#2860) 2022-12-13 09:57:26 +00:00
state_advance_timer.rs Enable proposer boost re-orging (#2860) 2022-12-13 09:57:26 +00:00
sync_committee_rewards.rs Update sync rewards API for abstract exec payload 2023-01-25 15:46:47 +11:00
sync_committee_verification.rs New rust lints for rustc 1.64.0 (#3602) 2022-09-23 03:52:46 +00:00
test_utils.rs Optimise payload attributes calculation and add SSE (#4027) 2023-03-05 23:43:30 +00:00
timeout_rw_lock.rs Add flag to disable lock timeouts (#2714) 2021-10-19 00:30:40 +00:00
validator_monitor.rs Cache validator balances and allow them to be served over the HTTP API (#3863) 2023-02-21 20:54:55 +00:00
validator_pubkey_cache.rs Fix regression in DB write atomicity (#3931) 2023-02-13 03:32:01 +00:00