cerc-io/lighthouse
16
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity
931daa40d7
lighthouse/testing/ef_tests
History
Paul Hauner 931daa40d7 Add fork choice EF tests (#2737) 
## Issue Addressed

Resolves #2545

## Proposed Changes

Adds the long-overdue EF tests for fork choice. Although we had pretty good coverage via other implementations that closely followed our approach, it is nonetheless important for us to implement these tests too.

During testing I found that we were using a hard-coded `SAFE_SLOTS_TO_UPDATE_JUSTIFIED` value rather than one from the `ChainSpec`. This caused a failure during a minimal preset test. This doesn't represent a risk to mainnet or testnets, since the hard-coded value matched the mainnet preset.

## Failing Cases

There is one failing case which is presently marked as `SkippedKnownFailure`:

```
case 4 ("new_finalized_slot_is_justified_checkpoint_ancestor") from /home/paul/development/lighthouse/testing/ef_tests/consensus-spec-tests/tests/minimal/phase0/fork_choice/on_block/pyspec_tests/new_finalized_slot_is_justified_checkpoint_ancestor failed with NotEqual:
head check failed: Got Head { slot: Slot(40), root: 0x9183dbaed4191a862bd307d476e687277fc08469fc38618699863333487703e7 } | Expected Head { slot: Slot(24), root: 0x105b49b51bf7103c182aa58860b039550a89c05a4675992e2af703bd02c84570 }
```

This failure is due to #2741. It's not a particularly high priority issue at the moment, so we fix it after merging this PR.
2021-11-08 07:29:04 +00:00
..
src Add fork choice EF tests (#2737) 2021-11-08 07:29:04 +00:00
tests Add fork choice EF tests (#2737) 2021-11-08 07:29:04 +00:00
.gitignore Update spec tests to v1.1.0-beta.4 (#2548) 2021-09-25 05:58:35 +00:00
Cargo.toml Add fork choice EF tests (#2737) 2021-11-08 07:29:04 +00:00
check_all_files_accessed.py Add fork choice EF tests (#2737) 2021-11-08 07:29:04 +00:00
Makefile Update to consensus-spec-tests v1.1.3 (#2722) 2021-10-16 05:07:21 +00:00
README.md Update spec tests to v1.1.0-beta.4 (#2548) 2021-09-25 05:58:35 +00:00

README.md

Consensus Specification Tests

This crate parses and executes the test vectors at ethereum/consensus-spec-tests.

Functionality is achieved only via the $ cargo test --features ef_tests command.

Running the Tests

Because the test vectors are very large, we do not download or run them by default. To download them, run (in this directory):

$ make

Note: this may download hundreds of MB of compressed archives from the ethereum/consensus-spec-tests, which may expand into several GB of files.

If successful, you should now have the extracted tests in ./consensus-spec-tests.

Run them with:

$ cargo test --features ef_tests

The tests won't run without the ef_tests feature enabled (this is to ensure that a top-level cargo test --all won't fail on missing files).

Saving Space

When you download the tests, the downloaded archives will be kept in addition to the extracted files. You have several options for saving space:

  1. Delete the archives (make clean-archives), and keep the extracted files. Suitable for everyday use, just don't re-run make or it will redownload the archives.
  2. Delete the extracted files (make clean-test-files), and keep the archives. Suitable for CI, or temporarily saving space. If you re-run make it will extract the archives rather than redownloading them.
  3. Delete everything (make clean). Good for updating to a new version, or if you no longer wish to run the EF tests.