lighthouse/common
Michael Sproul 01556f6f01 Optimise payload attributes calculation and add SSE (#4027)
## Issue Addressed

Closes #3896
Closes #3998
Closes #3700

## Proposed Changes

- Optimise the calculation of withdrawals for payload attributes by avoiding state clones, avoiding unnecessary state advances and reading from the snapshot cache if possible.
- Use the execution layer's payload attributes cache to avoid re-calculating payload attributes. I actually implemented a new LRU cache just for withdrawals but it had the exact same key and most of the same data as the existing payload attributes cache, so I deleted it.
- Add a new SSE event that fires when payloadAttributes are calculated. This is useful for block builders, a la https://github.com/ethereum/beacon-APIs/issues/244.
- Add a new CLI flag `--always-prepare-payload` which forces payload attributes to be sent with every fcU regardless of connected proposers. This is intended for use by builders/relays.

For maximum effect, the flags I've been using to run Lighthouse in "payload builder mode" are:

```
--always-prepare-payload \
--prepare-payload-lookahead 12000 \
--suggested-fee-recipient 0x0000000000000000000000000000000000000000
```

The fee recipient is required so Lighthouse has something to pack in the payload attributes (it can be ignored by the builder). The lookahead causes fcU to be sent at the start of every slot rather than at 8s. As usual, fcU will also be sent after each change of head block. I think this combination is sufficient for builders to build on all viable heads. Often there will be two fcU (and two payload attributes) sent for the same slot: one sent at the start of the slot with the head from `n - 1` as the parent, and one sent after the block arrives with `n` as the parent.

Example usage of the new event stream:

```bash
curl -N "http://localhost:5052/eth/v1/events?topics=payload_attributes"
```

## Additional Info

- [x] Tests added by updating the proposer re-org tests. This has the benefit of testing the proposer re-org code paths with withdrawals too, confirming that the new changes don't interact poorly.
- [ ] Benchmarking with `blockdreamer` on devnet-7 showed promising results but I'm yet to do a comparison to `unstable`.


Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-03-05 23:43:30 +00:00
..
account_utils Clippy lints for rust 1.66 (#3810) 2022-12-16 04:04:00 +00:00
clap_utils Verify execution block hashes during finalized sync (#3794) 2023-01-09 03:11:59 +00:00
compare_fields Clippy 1.67 (#3916) 2023-01-27 09:48:42 +00:00
compare_fields_derive Clippy 1.67 (#3916) 2023-01-27 09:48:42 +00:00
deposit_contract Update discv5 (#3171) 2022-10-28 05:40:06 +00:00
directory Update to Rust 1.59 and 2021 edition (#3038) 2022-02-25 00:10:17 +00:00
eth2 Optimise payload attributes calculation and add SSE (#4027) 2023-03-05 23:43:30 +00:00
eth2_config Delete Kiln and Ropsten configs (#4038) 2023-03-01 06:16:14 +00:00
eth2_interop_keypairs Update and consolidate dependencies (#3136) 2022-04-04 00:26:16 +00:00
eth2_network_config Delete Kiln and Ropsten configs (#4038) 2023-03-01 06:16:14 +00:00
eth2_wallet_manager Replace OpenOptions::new with File::options to be readable (#3059) 2022-03-07 06:30:18 +00:00
filesystem Fix rust 1.65 lints (#3682) 2022-11-04 07:43:43 +00:00
lighthouse_metrics Support histogram buckets (#3391) 2022-09-13 01:57:44 +00:00
lighthouse_version v3.5.0 (#3996) 2023-02-22 06:00:49 +00:00
lockfile Replace OpenOptions::new with File::options to be readable (#3059) 2022-03-07 06:30:18 +00:00
logging Add log debounce to work processor (#3045) 2022-03-07 06:30:17 +00:00
lru_cache Enforce a timeout on peer disconnect (#3757) 2023-02-14 03:25:42 +00:00
malloc_utils Switch allocator to jemalloc (#3697) 2023-01-20 04:19:29 +00:00
monitoring_api Remove fallback support from eth1 service (#3594) 2022-10-04 08:33:39 +00:00
oneshot_broadcast Fix concurrency issue with oneshot_broadcast (#3596) 2022-09-21 10:52:14 +00:00
sensitive_url Fix lints for Rust 1.63 (#3459) 2022-08-12 00:56:39 +00:00
slot_clock Improve testing slot clock to allow manipulation of time in tests (#3974) 2023-02-16 23:34:32 +00:00
system_health Health Endpoints for UI (#3668) 2022-11-15 05:21:26 +00:00
target_check Update to Rust 1.59 and 2021 edition (#3038) 2022-02-25 00:10:17 +00:00
task_executor Enable proposer boost re-orging (#2860) 2022-12-13 09:57:26 +00:00
test_random_derive Update to Rust 1.59 and 2021 edition (#3038) 2022-02-25 00:10:17 +00:00
unused_port Engine API v1.0.0.alpha.6 + interop tests (#3024) 2022-02-17 21:47:06 +00:00
validator_dir Clippy lints for rust 1.66 (#3810) 2022-12-16 04:04:00 +00:00
warp_utils Implement feerecipient API for keymanager (#3213) 2022-07-06 03:51:08 +00:00
README.md Directory Restructure (#1163) 2020-05-18 21:24:23 +10:00

eth2

Common crates containing eth2-specific logic.