Commit Graph

4873 Commits

Author SHA1 Message Date
realbigsean
35c9e2407b
bump ef-tests 2023-01-13 15:11:46 -05:00
realbigsean
f47fc9ddf3
Merge pull request #3880 from ethDreamer/capella
Sign BlsToExecutionChange w/ GENESIS_FORK_VERSION
2023-01-13 15:11:11 -05:00
realbigsean
1319683736
Update gossip_methods.rs 2023-01-13 14:59:03 -05:00
Mark Mackey
05c1291d8a Don't Penalize Early bls_to_execution_change 2023-01-13 12:53:25 -06:00
Mark Mackey
d9dd9b43ee Sign BlsToExecutionChange w/ GENESIS_FORK_VERSION 2023-01-13 10:47:19 -06:00
realbigsean
d0368b0459
Merge pull request #3874 from michaelsproul/beacon_chain_tests
Fix some beacon_chain tests
2023-01-12 10:59:04 -05:00
Michael Sproul
aa896decc1 Fix some beacon_chain tests 2023-01-12 19:13:01 +11:00
Michael Sproul
b2c2d31b69
Merge pull request #3871 from michaelsproul/capella-v3.4.0
Back-merge unstable to Capella
2023-01-12 16:44:00 +11:00
Michael Sproul
2af8110529
Merge remote-tracking branch 'origin/unstable' into capella
Fixing the conflicts involved patching up some of the `block_hash` verification,
the rest will be done as part of https://github.com/sigp/lighthouse/issues/3870
2023-01-12 16:22:00 +11:00
Michael Sproul
56e6b3557a
Fix Arbitrary implementations (#3867)
* Fix Arbitrary implementations

* Remove remaining vestiges of arbitrary-fuzz

* Remove FIXME

* Clippy
2023-01-12 15:17:03 +11:00
ethDreamer
52c1055fdc
Remove withdrawals-processing feature (#3864)
* Use spec to Determine Supported Engine APIs

* Remove `withdrawals-processing` feature

* Fixed Tests

* Missed Some Spots

* Fixed Another Test

* Stupid Clippy
2023-01-12 15:15:08 +11:00
Paul Hauner
38514c07f2 Release v3.4.0 (#3862)
## Issue Addressed

NA

## Proposed Changes

Bump versions

## Additional Info

- [x] ~~Blocked on #3728, #3801~~
- [x] ~~Blocked on #3866~~
- [x] Requires additional testing
2023-01-11 03:27:08 +00:00
realbigsean
98b11bbd3f
add historical summaries (#3865)
* add historical summaries

* fix tree hash caching, disable the sanity slots test with fake crypto

* add ssz static HistoricalSummary

* only store historical summaries after capella

* Teach `UpdatePattern` about Capella

* Tidy EF tests

* Clippy

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-01-11 12:40:21 +11:00
Michael Sproul
0c74cd4696 Update dependencies incl Tokio (#3866)
## Proposed Changes

Update all dependencies to new semver-compatible releases with `cargo update`. Importantly this patches a Tokio vuln: https://rustsec.org/advisories/RUSTSEC-2023-0001. I don't think we were affected by the vuln because it only applies to named pipes on Windows, but it's still good hygiene to patch.
2023-01-09 23:29:23 +00:00
realbigsean
c85cd87c10 Web3 signer validator definitions reloading on any request (#3801)
## Issue Addressed

https://github.com/sigp/lighthouse/issues/3795


Co-authored-by: realbigsean <sean@sigmaprime.io>
2023-01-09 08:18:56 +00:00
Paul Hauner
830efdb5c2 Improve validator monitor experience for high validator counts (#3728)
## Issue Addressed

NA

## Proposed Changes

Myself and others (#3678) have observed  that when running with lots of validators (e.g., 1000s) the cardinality is too much for Prometheus. I've seen Prometheus instances just grind to a halt when we turn the validator monitor on for our testnet validators (we have 10,000s of Goerli validators). Additionally, the debug log volume can get very high with one log per validator, per attestation.

To address this, the `bn --validator-monitor-individual-tracking-threshold <INTEGER>` flag has been added to *disable* per-validator (i.e., non-aggregated) metrics/logging once the validator monitor exceeds the threshold of validators. The default value is `64`, which is a finger-to-the-wind value. I don't actually know the value at which Prometheus starts to become overwhelmed, but I've seen it work with ~64 validators and I've seen it *not* work with 1000s of validators. A default of `64` seems like it will result in a breaking change to users who are running millions of dollars worth of validators whilst resulting in a no-op for low-validator-count users. I'm open to changing this number, though.

Additionally, this PR starts collecting aggregated Prometheus metrics (e.g., total count of head hits across all validators), so that high-validator-count validators still have some interesting metrics. We already had logging for aggregated values, so nothing has been added there.

I've opted to make this a breaking change since it can be rather damaging to your Prometheus instance to accidentally enable the validator monitor with large numbers of validators. I've crashed a Prometheus instance myself and had a report from another user who's done the same thing.

## Additional Info

NA

## Breaking Changes Note

A new label has been added to the validator monitor Prometheus metrics: `total`. This label tracks the aggregated metrics of all validators in the validator monitor (as opposed to each validator being tracking individually using its pubkey as the label).

Additionally, a new flag has been added to the Beacon Node: `--validator-monitor-individual-tracking-threshold`. The default value is `64`, which means that when the validator monitor is tracking more than 64 validators then it will stop tracking per-validator metrics and only track the `all_validators` metric. It will also stop logging per-validator logs and only emit aggregated logs (the exception being that exit and slashing logs are always emitted).

These changes were introduced in #3728 to address issues with untenable Prometheus cardinality and log volume when using the validator monitor with high validator counts (e.g., 1000s of validators). Users with less than 65 validators will see no change in behavior (apart from the added `all_validators` metric). Users with more than 65 validators who wish to maintain the previous behavior can set something like `--validator-monitor-individual-tracking-threshold 999999`.
2023-01-09 08:18:55 +00:00
Michael Sproul
168a7805c3 Add more Gnosis bootnodes (#3855)
## Proposed Changes

Add the latest long-running Gnosis chain bootnodes provided to us by the Gnosis team.
2023-01-09 05:12:31 +00:00
Michael Sproul
87c44697d0
Bump MSRV to 1.65 (#3860) 2023-01-09 14:40:09 +11:00
Michael Sproul
4bd2b777ec Verify execution block hashes during finalized sync (#3794)
## Issue Addressed

Recent discussions with other client devs about optimistic sync have revealed a conceptual issue with the optimisation implemented in #3738. In designing that feature I failed to consider that the execution node checks the `blockHash` of the execution payload before responding with `SYNCING`, and that omitting this check entirely results in a degradation of the full node's validation. A node omitting the `blockHash` checks could be tricked by a supermajority of validators into following an invalid chain, something which is ordinarily impossible.

## Proposed Changes

I've added verification of the `payload.block_hash` in Lighthouse. In case of failure we log a warning and fall back to verifying the payload with the execution client.

I've used our existing dependency on `ethers_core` for RLP support, and a new dependency on Parity's `triehash` crate for the Merkle patricia trie. Although the `triehash` crate is currently unmaintained it seems like our best option at the moment (it is also used by Reth, and requires vastly less boilerplate than Parity's generic `trie-root` library).

Block hash verification is pretty quick, about 500us per block on my machine (mainnet).

The optimistic finalized sync feature can be disabled using `--disable-optimistic-finalized-sync` which forces full verification with the EL.

## Additional Info

This PR also introduces a new dependency on our [`metastruct`](https://github.com/sigp/metastruct) library, which was perfectly suited to the RLP serialization method. There will likely be changes as `metastruct` grows, but I think this is a good way to start dogfooding it.

I took inspiration from some Parity and Reth code while writing this, and have preserved the relevant license headers on the files containing code that was copied and modified.
2023-01-09 03:11:59 +00:00
ethDreamer
11f4784ae6
Added bls_to_execution_changes to PersistedOpPool (#3857)
* Added bls_to_execution_changes to PersistedOpPool
2023-01-09 12:38:02 +11:00
ethDreamer
cb94f639b0
Isolate withdrawals-processing Feature (#3854) 2023-01-09 11:05:28 +11:00
ethDreamer
6b72f45cad
Merge pull request #3845 from realbigsean/capella-cleanup
Capella cleanup
2023-01-06 13:26:41 -06:00
Age Manning
1d9a2022b4 Upgrade to libp2p v0.50.0 (#3764)
I've needed to do this work in order to do some episub testing. 

This version of libp2p has not yet been released, so this is left as a draft for when we wish to update.

Co-authored-by: Diva M <divma@protonmail.com>
2023-01-06 15:59:33 +00:00
ethDreamer
a7c79ab5a4
Merge pull request #3847 from ethDreamer/capella
Update Execution Layer Tests for Capella
2023-01-05 15:50:26 -06:00
Mark Mackey
2ac609b64e Fixing Moar Failing Tests 2023-01-05 13:00:44 -06:00
Age Manning
4e5e7ee1fc Restructure code for libp2p upgrade (#3850)
Our custom RPC implementation is lagging from the libp2p v50 version. 

We are going to need to change a bunch of function names and would be nice to have consistent ordering of function names inside the handlers. 

This is a precursor to the libp2p upgrade to minimize merge conflicts in function ordering.
2023-01-05 17:18:24 +00:00
Mark Mackey
8711db2f3b Fix EF Tests 2023-01-04 15:14:43 -06:00
Mark Mackey
933772dd06
Fixed Operation Pool Tests 2023-01-03 18:40:35 -06:00
Mark Mackey
be232c4587
Update Execution Layer Tests for Capella 2023-01-03 16:58:15 -06:00
realbigsean
4353c49855
Update beacon_node/execution_layer/src/engine_api/json_structures.rs 2023-01-03 08:55:19 -05:00
realbigsean
d8f7277beb
cleanup 2022-12-30 11:00:14 -05:00
Michael Sproul
222a514506
Merge pull request #3842 from ethDreamer/capella
Merge with `unstable` and fix clippy
2022-12-29 09:35:33 +11:00
Mark Mackey
986ae4360a
Fix clippy complaints 2022-12-28 14:47:16 -06:00
Mark Mackey
c188cde034
merge upstream/unstable 2022-12-28 14:43:25 -06:00
ethDreamer
890cd6d810
Merge pull request #3838 from ethDreamer/capella
Fixed Some Tests
2022-12-27 17:35:37 -06:00
Mark Mackey
c922566fbc
Fixed Some Tests 2022-12-27 15:59:34 -06:00
Mark Mackey
96da8b9383
Feature Guard V2 Engine API Methods 2022-12-27 15:55:43 -06:00
ethDreamer
bfce794777
Merge pull request #3826 from ethDreamer/capella
Fixed spec serialization bug
2022-12-21 11:43:10 -06:00
Mark Mackey
3d253abadc Fixed spec serialization bug 2022-12-21 11:41:14 -06:00
ethDreamer
bf468692f7
Merge pull request #3821 from ethDreamer/capella
Removed `withdrawals` feature flag
2022-12-19 19:36:05 -06:00
ethDreamer
b224ed8151
Update consensus/state_processing/src/upgrade/eip4844.rs
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2022-12-19 19:35:17 -06:00
ethDreamer
0c22d69e15
Update consensus/state_processing/src/upgrade/eip4844.rs
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2022-12-19 19:35:08 -06:00
Michael Sproul
59a7a4703c Various CI fixes (#3813)
## Issue Addressed

Closes #3812
Closes #3750
Closes #3705
2022-12-20 01:34:52 +00:00
John Adler
53aad18da3 docs: remove mention of phases in voluntary exits (#3776)
The notion of "phases" doesn't exist anymore in the Ethereum roadmap. Also fix dead link to roadmap.

Co-authored-by: Michael Sproul <micsproul@gmail.com>
2022-12-20 01:34:51 +00:00
Mark Mackey
b75ca74222 Removed withdrawals feature flag 2022-12-19 15:38:46 -06:00
ethDreamer
ab11f8c71f
Merge pull request #3817 from ethDreamer/capella
Make engine_getPayloadV2 accept local block value
2022-12-16 16:05:37 -06:00
Mark Mackey
3a08c7634e Make engine_getPayloadV2 accept local block value 2022-12-16 15:44:55 -06:00
Divma
ffbf70e2d9 Clippy lints for rust 1.66 (#3810)
## Issue Addressed
Fixes the new clippy lints for rust 1.66

## Proposed Changes

Most of the changes come from:
- [unnecessary_cast](https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_cast)
- [iter_kv_map](https://rust-lang.github.io/rust-clippy/master/index.html#iter_kv_map)
- [needless_borrow](https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow)

## Additional Info

na
2022-12-16 04:04:00 +00:00
Michael Sproul
558367ab8c
Bounded withdrawals and spec v1.3.0-alpha.2 (#3802) 2022-12-16 09:20:45 +11:00
ethDreamer
37f735058a
Merge pull request #3809 from ethDreamer/capella
Merge branch `unstable` into `capella`
2022-12-15 12:25:46 -06:00