Commit Graph

4368 Commits

Author SHA1 Message Date
Divma
f2b1e096b2 Code quality improvents to the network service (#2932)
Checking how to priorize the polling of the network I moved most of the service code to functions. This change I think it's worth on it's own for code quality since inside the `tokio::select` many tools don't work (cargo fmt, sometimes clippy, and sometimes even the compiler's errors get wack). This is functionally equivalent to the previous code, just better organized
2022-01-26 23:14:23 +00:00
Divma
9964f5afe5 Document why we hash downloaded blocks for both sync algs (#2927)
## Proposed Changes
Initially the idea was to remove hashing of blocks in backfill sync. After considering it more, we conclude that we need to do it in both (forward and backfill) anyway. But since we forgot why we were doing it in the first place, this PR documents this logic. 

Future us should find it useful 


Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
2022-01-26 23:14:22 +00:00
Akihito Nakano
150931950d Fix errors from local testnet scripts on MacOS (#2919)
## Issue Addressed

Resolves https://github.com/sigp/lighthouse/issues/2763

## Proposed Changes

- Add a workflow which tests that local testnet starts successfully
  - Added `set` option into the scripts in order to fail fast so that we can notice errors during starting local testnet.
- Fix errors on MacOS
  - The redirect `&>>` is supported since bash v4 but the version bundled in macOS(11.6.1) is v3. a54f119c9b
2022-01-26 23:14:20 +00:00
Paul Hauner
5f628a71d4 v2.1.1 (#2951)
## Issue Addressed

NA

## Proposed Changes

- Bump Lighthouse version to v2.1.1
- Update `thread_local` from v1.1.3 to v1.1.4 to address https://rustsec.org/advisories/RUSTSEC-2022-0006

## Additional Info

- ~~Blocked on #2950~~
- ~~Blocked on #2952~~
2022-01-25 00:46:24 +00:00
Michael Sproul
69288f6164 VC: don't warn if BN config doesn't match exactly (#2952)
## Proposed Changes

Remove the check for exact equality on the beacon node spec when polling `/config/spec` from the VC. This check was always overzealous, and mostly served to check that the BN was configured for upcoming forks. I've replaced it by explicit checks of the `altair_fork_epoch` and `bellatrix_fork_epoch` instead.

## Additional Info

We should come back to this and clean it up so that we can retain compatibility while removing the field `default`s we installed.
2022-01-24 22:33:04 +00:00
Pawan Dhananjay
b9b3ea70de Fix metric name for monitoring (#2950)
## Issue Addressed

Resolves #2949 

## Proposed Changes

Fix metric naming for libp2p peer count.
2022-01-24 22:33:02 +00:00
Age Manning
ca29b580a2 Increase target subnet peers (#2948)
In the latest release we decreased the target number of subnet peers. 

It appears this could be causing issues in some cases and so reverting it back to the previous number it wise. A larger PR that follows this will address some other related discovery issues and peer management around subnet peer discovery.
2022-01-24 12:08:00 +00:00
Michael Sproul
f35a33716b Quote validator indices when posting duties (#2943)
## Proposed Changes

This PR establishes compatibility between Lighthouse's VC and Nimbus's BN.

Lighthouse was previously `POST`ing unquoted lists of validator indices to the attester and sync duties endpoints which were (correctly) not accepted by Nimbus. These lists had slipped through the cracks because they didn't have an explicit wrapper type to add `serde` annotations to. I've added the `ValidatorIndexDataRef` newtype in order to implement the modified serialisation behaviour.

## Testing

Combined with https://github.com/sigp/lighthouse/pull/2940, I've confirmed that this PR allows my Lighthouse VC on Prater to validate with the public Nimbus BN listed here: https://github.com/status-im/nimbus-eth2#quickly-test-your-tooling-against-nimbus. I haven't had a block proposal yet, but attestations and sync committee messages are working.

## Additional Info

This may also provide compatibility with Prysm BNs but I haven't had a chance to test that yet.
2022-01-21 06:07:20 +00:00
Pawan Dhananjay
799aedd631 Add default config options for transition constants (#2940)
## Issue Addressed

Continuation to #2934 

## Proposed Changes

Currently, we have the transition fields in the config (`TERMINAL_TOTAL_DIFFICULTY`, `TERMINAL_BLOCK_HASH` and `TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH`) as mandatory fields. 

This is causing compatibility issues with other client BN's (nimbus and teku v22.1.0) which don't return these fields on a `eth/v1/config/spec` api call. Since we don't use this values until the merge, I think it's okay to have default values set for these fields as well to ensure compatibility.
2022-01-20 23:05:42 +00:00
Rishi Kumar Ray
f0f327af0c Removed all disable_forks (#2925)
#2923 

Which issue # does this PR address?
There's a redundant field on the BeaconChain called disabled_forks that was once part of our fork-aware networking (#953) but which is no longer used and could be deleted. so Removed all references to disabled_forks so that the code compiles and git grep disabled_forks returns no results.

## Proposed Changes

Please list or describe the changes introduced by this PR.
Removed all references of disabled_forks


Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
2022-01-20 09:14:26 +00:00
Age Manning
fc7a1a7dc7 Allow disconnected states to introduce new peers without warning (#2922)
## Issue Addressed

We emit a warning to verify that all peer connection state information is consistent. A warning is given under one edge case;
We try to dial a peer with peer-id X and multiaddr Y. The peer responds to multiaddr Y with a different peer-id, Z. The dialing to the peer fails, but libp2p injects the failed attempt as peer-id Z. 

In this instance, our PeerDB tries to add a new peer in the disconnected state under a previously unknown peer-id. This is harmless and so this PR permits this behaviour without logging a warning.
2022-01-20 09:14:25 +00:00
Akihito Nakano
a8ae9c8418 Add linkcheck workflow (#2918)
## Issue Addressed

Resolves #2889

## Additional Info

I have checked that the `linkcheck` workflow runs and detects broken links as expected, in https://github.com/ackintosh/lighthouse/pull/1.
2022-01-20 09:14:23 +00:00
eklm
0116c8d464 Change type of extra fields in ConfigAndPreset (#2913)
## Issue Addressed

#2900

## Proposed Changes

Change type of extra_fields in ConfigAndPreset so it can contain non string values (inside serde_json::Value)
2022-01-20 09:14:21 +00:00
Mac L
d06f87486a Support duplicate keys in HTTP API query strings (#2908)
## Issues Addressed

Closes #2739
Closes #2812

## Proposed Changes

Support the deserialization of query strings containing duplicate keys into their corresponding types.
As `warp` does not support this feature natively (as discussed in #2739), it relies on the external library [`serde_array_query`](https://github.com/sigp/serde_array_query) (written by @michaelsproul)

This is backwards compatible meaning that both of the following requests will produce the same output:
```
curl "http://localhost:5052/eth/v1/events?topics=head,block"
```
```
curl "http://localhost:5052/eth/v1/events?topics=head&topics=block"
```

## Additional Info

Certain error messages have changed slightly.  This only affects endpoints which accept multiple values.
For example:
```
{"code":400,"message":"BAD_REQUEST: invalid query: Invalid query string","stacktraces":[]}
```
is now
```
{"code":400,"message":"BAD_REQUEST: unable to parse query","stacktraces":[]}
```


The serve order of the endpoints `get_beacon_state_validators` and `get_beacon_state_validators_id` have flipped:
```rust
.or(get_beacon_state_validators_id.boxed())
.or(get_beacon_state_validators.boxed())
``` 
This is to ensure proper error messages when filter fallback occurs due to the use of the `and_then` filter.

## Future Work
- Cleanup / remove filter fallback behaviour by substituting `and_then` with `then` where appropriate.
- Add regression tests for HTTP API error messages.

## Credits
- @mooori for doing the ground work of investigating possible solutions within the existing Rust ecosystem.
- @michaelsproul for writing [`serde_array_query`](https://github.com/sigp/serde_array_query) and for helping debug the behaviour of the `warp` filter fallback leading to incorrect error messages.
2022-01-20 09:14:19 +00:00
Paul Hauner
79db2d4deb v2.1.0 (#2928)
## Issue Addressed

NA

## Proposed Changes

Bump to `v2.1.0`.

## Additional Info

NA
2022-01-20 03:39:41 +00:00
Michael Sproul
95b3183cb4 Make /config/spec backwards compat for VC (#2934)
## Proposed Changes

Restore compatibility with beacon nodes using the `MERGE` naming by:

1. Adding defaults for the Bellatrix `Config` fields
2. Not attempting to read (or serve) the Bellatrix preset on `/config/spec`.

I've confirmed that this works with Infura, and just logs a warning:

```
Jan 20 10:51:31.078 INFO Connected to beacon node                endpoint: https://eth2-beacon-mainnet.infura.io/, version: teku/v22.1.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17
Jan 20 10:51:31.344 WARN Beacon node config does not match exactly, advice: check that the BN is updated and configured for any upcoming forks, endpoint: https://eth2-beacon-mainnet.infura.io/
Jan 20 10:51:31.344 INFO Initialized beacon node connections     available: 1, total: 1
```
2022-01-20 01:31:53 +00:00
Michael Sproul
ef7351ddfe Update to spec v1.1.8 (#2893)
## Proposed Changes

Change the canonical fork name for the merge to Bellatrix. Keep other merge naming the same to avoid churn.

I've also fixed and enabled the `fork` and `transition` tests for Bellatrix, and the v1.1.7 fork choice tests.

Additionally, the `BellatrixPreset` has been added with tests. It gets served via the `/config/spec` API endpoint along with the other presets.
2022-01-19 00:24:19 +00:00
Paul Hauner
9ed92d6e78 Use "release candidate" in book (#2920)
## Issue Addressed

NA

## Proposed Changes

Since we use the `rc` (release candidate) tag in our version strings, it seems consistent if we also use "release candidate" in the book rather than "pre-release".

Notably, Github adds a "pre-release" tag to release when we request. I think it's OK that Github uses that term whilst we consistently use "release candidate". Our docs indicate that the terms are interchangeable. 

## Additional Info

I hope to use the new docs link in the `v2.1.0` release, so it would be nice if we can merge this soon 🙏
2022-01-17 06:09:58 +00:00
Michael Sproul
a836e180f9 Release v2.1.0-rc.1 (#2921)
## Proposed Changes

New release candidate to address Windows build failure for rc.0
2022-01-17 03:25:30 +00:00
Paul Hauner
a26b8802da Release v2.1.0-rc.0 (#2905)
## Issue Addressed

NA

## Proposed Changes

Bump version tags to `v2.1.0-rc.0`.

## Additional Info

NA
2022-01-16 23:25:25 +00:00
Paul Hauner
c11253a82f Remove grandparents from snapshot cache (#2917)
## Issue Addressed

NA

## Proposed Changes

In https://github.com/sigp/lighthouse/pull/2832 we made some changes to the `SnapshotCache` to help deal with the one-block reorgs seen on mainnet (and testnets).

I believe the change in #2832 is good and we should keep it, but I think that in its present form it is causing the `SnapshotCache` to hold onto states that it doesn't need anymore. For example, a skip slot will result in one more `BeaconSnapshot` being stored in the cache.

This PR adds a new type of pruning that happens after a block is inserted to the cache. We will remove any snapshot from the cache that is a *grandparent* of the block being imported. Since we know the grandparent has two valid blocks built atop it, it is not at risk from a one-block re-org. 

## Additional Info

NA
2022-01-14 07:20:55 +00:00
Michael Sproul
ceeab02e3a Lazy hashing for SignedBeaconBlock in sync (#2916)
## Proposed Changes

Allocate less memory in sync by hashing the `SignedBeaconBlock`s in a batch directly, rather than going via SSZ bytes.

Credit to @paulhauner for finding this source of temporary allocations.
2022-01-14 07:20:54 +00:00
Age Manning
1c667ad3ca PeerDB Status unknown bug fix (#2907)
## Issue Addressed

The PeerDB was getting out of sync with the number of disconnected peers compared to the actual count. As this value determines how many we store in our cache, over time the cache was depleting and we were removing peers immediately resulting in errors that manifest as unknown peers for some operations.

The error occurs when dialing a peer fails, we were not correctly updating the peerdb counter because the increment to the counter was placed in the wrong order and was therefore not incrementing the count. 

This PR corrects this.
2022-01-14 05:42:48 +00:00
Age Manning
6f4102aab6 Network performance tuning (#2608)
There is a pretty significant tradeoff between bandwidth and speed of gossipsub messages. 

We can reduce our bandwidth usage considerably at the cost of minimally delaying gossipsub messages. The impact of delaying messages has not been analyzed thoroughly yet, however this PR in conjunction with some gossipsub updates show considerable bandwidth reduction. 

This PR allows the user to set a CLI value (`network-load`) which is an integer in the range of 1 of 5 depending on their bandwidth appetite. 1 represents the least bandwidth but slowest message recieving and 5 represents the most bandwidth and fastest received message time. 

For low-bandwidth users it is likely to be more efficient to use a lower value. The default is set to 3, which currently represents a reduced bandwidth usage compared to previous version of this PR. The previous lighthouse versions are equivalent to setting the `network-load` CLI to 4.

This PR is awaiting a few gossipsub updates before we can get it into lighthouse.
2022-01-14 05:42:47 +00:00
Age Manning
db95255aeb Remove gitter from readme (#2914)
We dont check gitter, so shouldn't refer users to it.
2022-01-14 04:07:17 +00:00
Akihito Nakano
6883e1bfb6 Fix broken links in book (#2912)
## Issue Addressed

https://github.com/sigp/lighthouse/issues/2889

## Additional Info

I have checked that linkcheck has succeeded on the book built locally. 👌 

```shell
$ cd book
$ mdbook serve --open
...
2022-01-14 01:13:40 [INFO] (mdbook::book): Book building has started
2022-01-14 01:13:40 [INFO] (mdbook::book): Running the html backend

$ linkcheck http://localhost:3000
Perfect. Checked 4495 links, 80 destination URLs (76 ignored).
```

Also I'll tackle running linkcheck on CI in another pull request.
2022-01-14 00:38:04 +00:00
Michael Sproul
e8887ffea0 Rust 1.58 lints (#2906)
## Issue Addressed

Closes #2616

## Proposed Changes

* Fixes for new Rust 1.58.0 lints
* Enable the `fn_to_numeric_cast_any` (#2616)
2022-01-13 22:39:58 +00:00
Paul Hauner
2ce2ec9b62 Remove penalty for attesting to unknown head (#2903)
## Issue Addressed

- Resolves https://github.com/sigp/lighthouse/issues/2902

## Proposed Changes

As documented in https://github.com/sigp/lighthouse/issues/2902, there are some cases where we will score peers very harshly for sending attestations to an unknown head.

This PR removes the penalty when an attestation for an unknown head is received, queued for block look-up, then popped from the queue without the head block being known. This prevents peers from being penalized for an unknown block when that peer was never actually asked for the block.

Peer penalties should still be applied to the peers who *do* get the request for the block and fail to respond with a valid block. As such, peers who send us attestations to non-existent heads should eventually be booted.

## Additional Info

- [ ] Need to confirm that a timeout for a bbroot request will incur a penalty.
2022-01-13 03:08:38 +00:00
realbigsean
f13e9c3d10 Antithesis docker workflow (#2877)
## Issue Addressed

Automates a build and push to antithesis servers on merges to unstable. They run tests against lighthouse daily and have requested more frequent pushes. Currently we are just manually pushing stable images when we have a new release.

## Proposed Changes

- Add a `Dockerfile.libvoidstar`
- Add the `libvoidstar.so` binary
- Add a new workflow to autmatically build and push on merges to unstable

## Additional Info
Requires adding the following secrets
-`ANTITHESIS_USERNAME`
-`ANTITHESIS_PASSWORD`
-`ANTITHESIS_REPOSITORY`
-`ANTITHESIS_SERVER`

Tested here: https://github.com/realbigsean/lighthouse/actions/runs/1612821446

Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <sean@sigmaprime.io>
2022-01-12 20:58:41 +00:00
Paul Hauner
aaa5344eab Add peer score adjustment msgs (#2901)
## Issue Addressed

N/A

## Proposed Changes

This PR adds the `msg` field to `Peer score adjusted` log messages. These `msg` fields help identify *why* a peer was banned.

Example:

```
Jan 11 04:18:48.096 DEBG Peer score adjusted                     score: -100.00, peer_id: 16Uiu2HAmQskxKWWGYfginwZ51n5uDbhvjHYnvASK7PZ5gBdLmzWj, msg: attn_unknown_head, service: libp2p
Jan 11 04:18:48.096 DEBG Peer score adjusted                     score: -27.86, peer_id: 16Uiu2HAmA7cCb3MemVDbK3MHZoSb7VN3cFUG3vuSZgnGesuVhPDE, msg: sync_past_slot, service: libp2p
Jan 11 04:18:48.096 DEBG Peer score adjusted                     score: -100.00, peer_id: 16Uiu2HAmQskxKWWGYfginwZ51n5uDbhvjHYnvASK7PZ5gBdLmzWj, msg: attn_unknown_head, service: libp2p
Jan 11 04:18:48.096 DEBG Peer score adjusted                     score: -28.86, peer_id: 16Uiu2HAmA7cCb3MemVDbK3MHZoSb7VN3cFUG3vuSZgnGesuVhPDE, msg: sync_past_slot, service: libp2p
Jan 11 04:18:48.096 DEBG Peer score adjusted                     score: -29.86, peer_id: 16Uiu2HAmA7cCb3MemVDbK3MHZoSb7VN3cFUG3vuSZgnGesuVhPDE, msg: sync_past_slot, service: libp2p
```

There is also a `libp2p_report_peer_msgs_total` metrics which allows us to see count of reports per `msg` tag. 

## Additional Info

NA
2022-01-12 05:32:14 +00:00
Paul Hauner
61f60bdf03 Avoid penalizing peers for delays during processing (#2894)
## Issue Addressed

NA

## Proposed Changes

We have observed occasions were under-resourced nodes will receive messages that were valid *at the time*, but later become invalidated due to long waits for a `BeaconProcessor` worker.

In this PR, we will check to see if the message was valid *at the time of receipt*. If it was initially valid but invalid now, we just ignore the message without penalizing the peer.

## Additional Info

NA
2022-01-12 02:36:24 +00:00
Michael Sproul
b656007963 Skip serializing proposer boost if null (#2899)
## Issue Addressed

Restore compatibility between Lighthouse v2.0.1 VC and `unstable` BN in preparation for the next release.

## Proposed Changes

* Don't serialize the `PROPOSER_SCORE_BOOST` as `null` because it breaks the `extra_fields: HashMap<String, String>` used by the v2.0.1 VC.
2022-01-11 05:33:29 +00:00
Paul Hauner
4848e53155 Avoid peer penalties on internal errors for batch block import (#2898)
## Issue Addressed

NA

## Proposed Changes

I've observed some Prater nodes (and potentially some mainnet nodes) banning peers due to validator pubkey cache lock timeouts. For the `BeaconChainError`-type of errors, they're caused by internal faults and we can't necessarily tell if the peer is bad or not. I think this is causing us to ban peers unnecessarily when running on under-resourced machines.

## Additional Info

NA
2022-01-11 05:33:28 +00:00
Michael Sproul
6976796162 Update dependencies including sha2 (#2896)
## Proposed Changes

Although the [security advisory](https://rustsec.org/advisories/RUSTSEC-2021-0100.html) only lists `sha2` 0.9.7 as vulnerable, the [changelog](https://github.com/RustCrypto/hashes/blob/master/sha2/CHANGELOG.md#099-2022-01-06) states that 0.9.8 is also vulnerable, and has been yanked.
2022-01-11 01:35:56 +00:00
Paul Hauner
02e2fd2fb8 Add early attester cache (#2872)
## Issue Addressed

NA

## Proposed Changes

Introduces a cache to attestation to produce atop blocks which will become the head, but are not fully imported (e.g., not inserted into the database).

Whilst attesting to a block before it's imported is rather easy, if we're going to produce that attestation then we also need to be able to:

1. Verify that attestation.
1. Respond to RPC requests for the `beacon_block_root`.

Attestation verification (1) is *partially* covered. Since we prime the shuffling cache before we insert the block into the early attester cache, we should be fine for all typical use-cases. However, it is possible that the cache is washed out before we've managed to insert the state into the database and then attestation verification will fail with a "missing beacon state"-type error.

Providing the block via RPC (2) is also partially covered, since we'll check the database *and* the early attester cache when responding a blocks-by-root request. However, we'll still omit the block from blocks-by-range requests (until the block lands in the DB). I *think* this is fine, since there's no guarantee that we return all blocks for those responses.

Another important consideration is whether or not the *parent* of the early attester block is available in the databse. If it were not, we might fail to respond to blocks-by-root request that are iterating backwards to collect a chain of blocks. I argue that *we will always have the parent of the early attester block in the database.* This is because we are holding the fork-choice write-lock when inserting the block into the early attester cache and we do not drop that until the block is in the database.
2022-01-11 01:35:55 +00:00
Richard Patel
65b1374b58 Document Homebrew package (#2885)
## Issue Addressed

Resolves #2329

## Proposed Changes

Documents the recently added `lighthouse` Homebrew formula.

## Additional Info

NA


Co-authored-by: Richard Patel <richard@blockdaemon.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2022-01-08 01:15:07 +00:00
Fredrik Svantes
ccdc10c288 Adjusting ARCHIVE_URL (#2892)
Was renamed from eth2-clients to eth-clients
2022-01-07 05:32:34 +00:00
Akihito Nakano
daa3da3758 Add tests for flags enable-enr-auto-update and disable-packet-filter (#2887)
Resolves https://github.com/sigp/lighthouse/issues/2602

## Issue Addressed

https://github.com/sigp/lighthouse/pull/2749#issue-1037552417
> ## Open TODO
> Add tests for boot_node flags `enable-enr-auto-update` and `disable-packet-filter`. They end up in [Discv5Config](9ed2cba6bc/boot_node/src/config.rs (L29)), which doesn't support serde (de)serialization.

## Proposed Changes

- Added tests for flags `enable-enr-auto-update` and `disable-packet-filter`
- Instead of (de)serialize Discv5Config, added the two fields copied from Discv5Config to BootNodeConfigSerialization.
2022-01-07 05:32:33 +00:00
Mac L
20941bc0f7 Fix off-by-one in block packing lcli (#2878)
## Issue Addressed

The current `lcli` block packing code has an off-by-one where it would include an extra slot (the oldest slot) of attestations as "available" (this means there would be 33 slots of "available" attestations instead of 32).
There is typically only single-digit attestations remaining from that slot and as such does not cause a significant change to the results although every efficiency will have been very slightly under-reported.

## Proposed Changes

Prune the `available_attestation_set` before writing out the data instead of after.

## Additional Info

This `lcli` code will soon be deprecated by a Lighthouse API (#2879)  which will run significantly faster and will be used to hook into our upcoming monitoring platform #2873.
2022-01-07 05:32:29 +00:00
Philipp K
668477872e Allow value for beacon_node fee-recipient argument (#2884)
## Issue Addressed

The fee-recipient argument of the beacon node does not allow a value to be specified:

> $ lighthouse beacon_node --merge --fee-recipient "0x332E43696A505EF45b9319973785F837ce5267b9"
> error: Found argument '0x332E43696A505EF45b9319973785F837ce5267b9' which wasn't expected, or isn't valid in this context
> 
> USAGE:
>    lighthouse beacon_node --fee-recipient --merge
>
> For more information try --help

## Proposed Changes

Allow specifying a value for the fee-recipient argument in beacon_node/src/cli.rs

## Additional Info

I've added .takes_value(true) and successfully proposed a block in the kintsugi testnet with my own fee-recipient address instead of the hardcoded default. I think that was just missed as the argument does not make sense without a value :)


Co-authored-by: pk910 <philipp@pk910.de>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2022-01-07 01:21:42 +00:00
Paul Hauner
f6b5b1a8be Use ? debug formatting for block roots in beacon_chain.rs (#2890)
## Issue Addressed

NA

## Proposed Changes

Ensures full roots are printed, rather than shortened versions like `0x935b…d376`.

For example, it would be nice if we could do API queries based upon the roots shown in the `Beacon chain re-org` event:

```
Jan 05 12:36:52.224 WARN Beacon chain re-org                     reorg_distance: 2, new_slot: 2073184, new_head: 0x8a97…2dec, new_head_parent: 0xa985…7688, previous_slot: 2073183, previous_head: 0x935b…d376, service: beacon
Jan 05 13:35:05.832 WARN Beacon chain re-org                     reorg_distance: 1, new_slot: 2073475, new_head: 0x9207…c6b9, new_head_parent: 0xb2ce…839b, previous_slot: 2073474, previous_head: 0x8066…92f7, service: beacon
```

## Additional Info

We should eventually fix this project-wide, however this is a short-term patch.
2022-01-06 05:16:50 +00:00
Michael Sproul
fac117667b Update to superstruct v0.4.1 (#2886)
## Proposed Changes

Update `superstruct` to bring in @realbigsean's fixes necessary for MEV-compatible private beacon block types (a la #2795).

The refactoring is due to another change in superstruct that allows partial getters to be auto-generated.
2022-01-06 03:14:58 +00:00
Michael Sproul
0b54ff17f2 Fix assert in slashing protection import (#2881)
## Issue Addressed

There was an overeager assert in the import of slashing protection data here:

fff01b24dd/validator_client/slashing_protection/src/slashing_database.rs (L939)

We were asserting that if the import contained any blocks for a validator, then the database should contain only a single block for that validator due to pruning/consolidation. However, we would only prune if the import contained _relevant blocks_ (that would actually change the maximum slot):

fff01b24dd/validator_client/slashing_protection/src/slashing_database.rs (L629-L633)

This lead to spurious failures (in the form of `ConsistencyError`s) when importing an interchange containing no new blocks for any of the validators. This wasn't hard to trigger, e.g. export and then immediately re-import the same file.

## Proposed Changes

This PR fixes the issue by simplifying the import so that it's more like the import for attestations. I.e. we make the assert true by always pruning when the imported file contains blocks.

In practice this doesn't have any downsides: if we import a new block then the behaviour is as before, except that we drop the `signing_root`. If we import an existing block or an old block then we prune the database to a single block. The only time this would be relevant is during extreme clock drift locally _plus_ import of a non-drifted interchange, which should occur infrequently.

## Additional Info

I've also added `Arbitrary` implementations to the slashing protection types so that we can fuzz them. I have a fuzzer sitting in a separate directory which I may or may not commit in a subsequent PR.

There's a new test in the standard interchange tests v5.2.1 that checks for this issue: https://github.com/eth-clients/slashing-protection-interchange-tests/pull/12
2022-01-04 20:46:44 +00:00
realbigsean
dfc8968201 Update rust version in lcli Dockerfile (#2876)
The `lcli` docker build was no longer working on the old rust version

Co-authored-by: realbigsean <seananderson33@gmail.com>
2021-12-22 18:55:42 +00:00
Pawan Dhananjay
a0c5701e36 Only import blocks with valid execution payloads (#2869)
## Issue Addressed

N/A

## Proposed Changes

We are currently treating errors from the EL on `engine_executePayload` as `PayloadVerificationStatus::NotVerified`. This adds the block as a candidate head block in fork choice even if the EL explicitly rejected the block as invalid. 

`PayloadVerificationStatus::NotVerified` should be only returned when the EL explicitly returns "syncing" imo. This PR propagates an error instead of returning `NotVerified` on EL all EL errors.
2021-12-22 08:15:37 +00:00
Age Manning
81c667b58e Additional networking metrics (#2549)
Adds additional metrics for network monitoring and evaluation.


Co-authored-by: Mark Mackey <mark@sigmaprime.io>
2021-12-22 06:17:14 +00:00
eklm
60d917d9e9 Allow to set validator password via reimport (#2868)
## Issue Addressed

Resolves #2854 

## Proposed Changes

If validator was imported first without entering password and then imported again with valid password update the password in validator_definitions.yml

## Additional Info

There can be other cases for updating existing validator during import. They are not covered here.

Co-authored-by: Michael Sproul <micsproul@gmail.com>
2021-12-21 08:23:18 +00:00
Michael Sproul
3b61ac9cbf Optimise slasher DB layout and switch to MDBX (#2776)
## Issue Addressed

Closes #2286
Closes #2538
Closes #2342

## Proposed Changes

Part II of major slasher optimisations after #2767

These changes will be backwards-incompatible due to the move to MDBX (and the schema change) 😱 

* [x] Shrink attester keys from 16 bytes to 7 bytes.
* [x] Shrink attester records from 64 bytes to 6 bytes.
* [x] Separate `DiskConfig` from regular `Config`.
* [x] Add configuration for the LRU cache size.
* [x] Add a "migration" that deletes any legacy LMDB database.
2021-12-21 08:23:17 +00:00
Michael Sproul
a290a3c537 Add configurable block replayer (#2863)
## Issue Addressed

Successor to #2431

## Proposed Changes

* Add a `BlockReplayer` struct to abstract over the intricacies of calling `per_slot_processing` and `per_block_processing` while avoiding unnecessary tree hashing.
* Add a variant of the forwards state root iterator that does not require an `end_state`.
* Use the `BlockReplayer` when reconstructing states in the database. Use the efficient forwards iterator for frozen states.
* Refactor the iterators to remove `Arc<HotColdDB>` (this seems to be neater than making _everything_ an `Arc<HotColdDB>` as I did in #2431).

Supplying the state roots allow us to avoid building a tree hash cache at all when reconstructing historic states, which saves around 1 second flat (regardless of `slots-per-restore-point`). This is a small percentage of worst-case state load times with 200K validators and SPRP=2048 (~15s vs ~16s) but a significant speed-up for more frequent restore points: state loads with SPRP=32 should be now consistently <500ms instead of 1.5s (a ~3x speedup).

## Additional Info

Required by https://github.com/sigp/lighthouse/pull/2628
2021-12-21 06:30:52 +00:00
Divma
56d596ee42 Unban peers at the swarm level when purged (#2855)
## Issue Addressed
#2840
2021-12-20 23:45:21 +00:00