lighthouse/testing/ef_tests
Michael Sproul bcdd960ab1 Separate execution payloads in the DB (#3157)
## Proposed Changes

Reduce post-merge disk usage by not storing finalized execution payloads in Lighthouse's database.

⚠️ **This is achieved in a backwards-incompatible way for networks that have already merged** ⚠️. Kiln users and shadow fork enjoyers will be unable to downgrade after running the code from this PR. The upgrade migration may take several minutes to run, and can't be aborted after it begins.

The main changes are:

- New column in the database called `ExecPayload`, keyed by beacon block root.
- The `BeaconBlock` column now stores blinded blocks only.
- Lots of places that previously used full blocks now use blinded blocks, e.g. analytics APIs, block replay in the DB, etc.
- On finalization:
    - `prune_abanonded_forks` deletes non-canonical payloads whilst deleting non-canonical blocks.
    - `migrate_db` deletes finalized canonical payloads whilst deleting finalized states.
- Conversions between blinded and full blocks are implemented in a compositional way, duplicating some work from Sean's PR #3134.
- The execution layer has a new `get_payload_by_block_hash` method that reconstructs a payload using the EE's `eth_getBlockByHash` call.
   - I've tested manually that it works on Kiln, using Geth and Nethermind.
   - This isn't necessarily the most efficient method, and new engine APIs are being discussed to improve this: https://github.com/ethereum/execution-apis/pull/146.
   - We're depending on the `ethers` master branch, due to lots of recent changes. We're also using a workaround for https://github.com/gakonst/ethers-rs/issues/1134.
- Payload reconstruction is used in the HTTP API via `BeaconChain::get_block`, which is now `async`. Due to the `async` fn, the `blocking_json` wrapper has been removed.
- Payload reconstruction is used in network RPC to serve blocks-by-{root,range} responses. Here the `async` adjustment is messier, although I think I've managed to come up with a reasonable compromise: the handlers take the `SendOnDrop` by value so that they can drop it on _task completion_ (after the `fn` returns). Still, this is introducing disk reads onto core executor threads, which may have a negative performance impact (thoughts appreciated).

## Additional Info

- [x] For performance it would be great to remove the cloning of full blocks when converting them to blinded blocks to write to disk. I'm going to experiment with a `put_block` API that takes the block by value, breaks it into a blinded block and a payload, stores the blinded block, and then re-assembles the full block for the caller.
- [x] We should measure the latency of blocks-by-root and blocks-by-range responses.
- [x] We should add integration tests that stress the payload reconstruction (basic tests done, issue for more extensive tests: https://github.com/sigp/lighthouse/issues/3159)
- [x] We should (manually) test the schema v9 migration from several prior versions, particularly as blocks have changed on disk and some migrations rely on being able to load blocks.

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2022-05-12 00:42:17 +00:00
..
src Separate execution payloads in the DB (#3157) 2022-05-12 00:42:17 +00:00
tests Kiln mev boost (#3062) 2022-03-31 07:52:23 +00:00
.gitignore Update spec tests to v1.1.0-beta.4 (#2548) 2021-09-25 05:58:35 +00:00
Cargo.toml Update to Rust 1.59 and 2021 edition (#3038) 2022-02-25 00:10:17 +00:00
check_all_files_accessed.py Update to spec v1.1.8 (#2893) 2022-01-19 00:24:19 +00:00
Makefile Rename random to prev_randao (#3040) 2022-03-03 02:10:57 +00:00
README.md Update spec tests to v1.1.0-beta.4 (#2548) 2021-09-25 05:58:35 +00:00

Consensus Specification Tests

This crate parses and executes the test vectors at ethereum/consensus-spec-tests.

Functionality is achieved only via the $ cargo test --features ef_tests command.

Running the Tests

Because the test vectors are very large, we do not download or run them by default. To download them, run (in this directory):

$ make

Note: this may download hundreds of MB of compressed archives from the ethereum/consensus-spec-tests, which may expand into several GB of files.

If successful, you should now have the extracted tests in ./consensus-spec-tests.

Run them with:

$ cargo test --features ef_tests

The tests won't run without the ef_tests feature enabled (this is to ensure that a top-level cargo test --all won't fail on missing files).

Saving Space

When you download the tests, the downloaded archives will be kept in addition to the extracted files. You have several options for saving space:

  1. Delete the archives (make clean-archives), and keep the extracted files. Suitable for everyday use, just don't re-run make or it will redownload the archives.
  2. Delete the extracted files (make clean-test-files), and keep the archives. Suitable for CI, or temporarily saving space. If you re-run make it will extract the archives rather than redownloading them.
  3. Delete everything (make clean). Good for updating to a new version, or if you no longer wish to run the EF tests.