lighthouse/testing/state_transition_vectors
Paul Hauner fe52322088 Implement SSZ union type (#2579)
## Issue Addressed

NA

## Proposed Changes

Implements the "union" type from the SSZ spec for `ssz`, `ssz_derive`, `tree_hash` and `tree_hash_derive` so it may be derived for `enums`:

https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.3/ssz/simple-serialize.md#union

The union type is required for the merge, since the `Transaction` type is defined as a single-variant union `Union[OpaqueTransaction]`.

### Crate Updates

This PR will (hopefully) cause CI to publish new versions for the following crates:

- `eth2_ssz_derive`: `0.2.1` -> `0.3.0`
- `eth2_ssz`: `0.3.0` -> `0.4.0`
- `eth2_ssz_types`: `0.2.0` -> `0.2.1`
- `tree_hash`: `0.3.0` -> `0.4.0`
- `tree_hash_derive`: `0.3.0` -> `0.4.0`

These these crates depend on each other, I've had to add a workspace-level `[patch]` for these crates. A follow-up PR will need to remove this patch, ones the new versions are published.

### Union Behaviors

We already had SSZ `Encode` and `TreeHash` derive for enums, however it just did a "transparent" pass-through of the inner value. Since the "union" decoding from the spec is in conflict with the transparent method, I've required that all `enum` have exactly one of the following enum-level attributes:

#### SSZ

-  `#[ssz(enum_behaviour = "union")]`
    - matches the spec used for the merge
-  `#[ssz(enum_behaviour = "transparent")]`
    - maintains existing functionality
    - not supported for `Decode` (never was)
    
#### TreeHash

-  `#[tree_hash(enum_behaviour = "union")]`
    - matches the spec used for the merge
-  `#[tree_hash(enum_behaviour = "transparent")]`
    - maintains existing functionality

This means that we can maintain the existing transparent behaviour, but all existing users will get a compile-time error until they explicitly opt-in to being transparent.

### Legacy Option Encoding

Before this PR, we already had a union-esque encoding for `Option<T>`. However, this was with the *old* SSZ spec where the union selector was 4 bytes. During merge specification, the spec was changed to use 1 byte for the selector.

Whilst the 4-byte `Option` encoding was never used in the spec, we used it in our database. Writing a migrate script for all occurrences of `Option` in the database would be painful, especially since it's used in the `CommitteeCache`. To avoid the migrate script, I added a serde-esque `#[ssz(with = "module")]` field-level attribute to `ssz_derive` so that we can opt into the 4-byte encoding on a field-by-field basis.

The `ssz::legacy::four_byte_impl!` macro allows a one-liner to define the module required for the `#[ssz(with = "module")]` for some `Option<T> where T: Encode + Decode`.

Notably, **I have removed `Encode` and `Decode` impls for `Option`**. I've done this to force a break on downstream users. Like I mentioned, `Option` isn't used in the spec so I don't think it'll be *that* annoying. I think it's nicer than quietly having two different union implementations or quietly breaking the existing `Option` impl.

### Crate Publish Ordering

I've modified the order in which CI publishes crates to ensure that we don't publish a crate without ensuring we already published a crate that it depends upon.

## TODO

- [ ] Queue a follow-up `[patch]`-removing PR.
2021-09-25 05:58:36 +00:00
..
src Altair consensus changes and refactors (#2279) 2021-07-09 06:15:32 +00:00
.gitignore Directory Restructure (#1163) 2020-05-18 21:24:23 +10:00
Cargo.toml Implement SSZ union type (#2579) 2021-09-25 05:58:36 +00:00
Makefile Directory Restructure (#1163) 2020-05-18 21:24:23 +10:00
README.md Directory Restructure (#1163) 2020-05-18 21:24:23 +10:00

state_transition_vectors

This crate contains test vectors for Lighthouse state transition functions.

This crate serves two purposes:

  • Outputting the test vectors to disk via make.
  • Running the vectors against our code via make test.

Outputting vectors to disk

Whilst we don't actually need to write the vectors to disk to test them, we provide this functionality so we can generate corpra for the fuzzer and also so they can be of use to other clients.

To create the files in ./vectors (directory relative to this crate), run:

make

This will produce a directory structure that looks roughly like this:

vectors
└── exit
    ├── invalid_bad_signature
    │   ├── block.ssz
    │   ├── error.txt
    │   └── pre.ssz
    ├── invalid_duplicate
    │   ├── block.ssz
    │   ├── error.txt
    │   └── pre.ssz
    ├── invalid_exit_already_initiated
    │   ├── block.ssz
    │   ├── error.txt
    │   └── pre.ssz
    ├── invalid_future_exit_epoch
    │   ├── block.ssz
    │   ├── error.txt
    │   └── pre.ssz
    ├── invalid_not_active_after_exit_epoch
    │   ├── block.ssz
    │   ├── error.txt
    │   └── pre.ssz
    ├── invalid_not_active_before_activation_epoch
    │   ├── block.ssz
    │   ├── error.txt
    │   └── pre.ssz
    ├── invalid_too_young_by_a_lot
    │   ├── block.ssz
    │   ├── error.txt
    │   └── pre.ssz
    ├── invalid_too_young_by_one_epoch
    │   ├── block.ssz
    │   ├── error.txt
    │   └── pre.ssz
    ├── invalid_validator_unknown
    │   ├── block.ssz
    │   ├── error.txt
    │   └── pre.ssz
    ├── valid_genesis_epoch
    │   ├── block.ssz
    │   ├── post.ssz
    │   └── pre.ssz
    └── valid_previous_epoch
        ├── block.ssz
        ├── post.ssz
        └── pre.ssz