Commit Graph

6087 Commits

Author SHA1 Message Date
Lucas Saldanha
f7daf82430 Removed old Teku mainnet bootnode ENRs (#4786)
## Issue Addressed

N/A

## Proposed Changes

Removing the two Teku mainnet bootnodes that are being sunset.

## Additional Info

We are leaving only these two bootnodes: https://github.com/eth-clients/eth2-networks/blob/master/shared/mainnet/bootstrap_nodes.txt#L10-L11
2023-10-03 23:59:34 +00:00
Divma
f11884ccdb enforce non zero enr ports (#4776)
## Issue Addressed

Right now lighthouse accepts zero as enr ports. Since enr ports should be reachable, zero ports should be rejected here

## Proposed Changes

- update the config to use `NonZerou16` as an ENR port for all enr-related fields.
- the enr builder from config now sets the enr to the listening port only if the enr port is not already set (prev behaviour) and the listening port is not zero (new behaviour)
- reject zero listening ports when used with `enr-match`. 
- boot node now rejects listening port as zero, since those are advertised.
- generate-bootnode-enr also rejected zero listening ports for the same reason.
- update local network scripts

## Additional Info

Unrelated, but why do we overwrite `enr-x-port` values with listening ports if `enr-match` is present? we prob should only do this for enr values that are not already set.
2023-10-03 23:59:34 +00:00
João Oliveira
0dc95a1d37 PeerManager: move the check for banned peers from connection_established (#4569)
## Issue Addressed
https://github.com/sigp/lighthouse/issues/4543
## Proposed Changes
- Removes `NotBanned` from `BanResult`, implements `Display` and `std::error::Error` for `BanResult` and changes `ban_result` return type to `Option<BanResult>` which helps returning `BanResult` on `handle_established_inbound_connection`  
- moves the check from for banned peers from `on_connection_established` to `handle_established_inbound_connection` to start addressing #4543.
- Removes `allow_block_list` as it's now redundant? Not sure about this one but if `PeerManager` keeps track of the banned peers, no need to send a `Swarm` event for `alow_block_list` to also keep that list right? 
 
## Questions

-  #4543 refers:
>  More specifically, implement the connection limit behaviour inside the peer manager.

@AgeManning do you mean copying `libp2p::connection_limits::Behaviour`'s code into `PeerManager`/ having it as an inner `NetworkBehaviour` of `PeerManager`/other? If it's the first two, I think it probably makes more sense to have it as it is as it's less code to maintain.

> Also implement the banning of peers inside the behaviour, rather than passing messages back up to the swarm.

I tried to achieve this, but we still need to pass the `PeerManagerEvent::Banned` swarm event as `DiscV5` handles it's node and ip management internally and I did not find a method to query if a peer is banned. Is there anything else we can do from here?

3397612160/beacon_node/lighthouse_network/src/discovery/mod.rs (L931-L940)

Same as the question above, I did not find a way to check if `DiscV5` has the peer banned, so that we could check here and avoid sending `Swarm` events

3397612160/beacon_node/lighthouse_network/src/peer_manager/network_behaviour.rs (L168-L178)

Is there a chance we try to dial a peer that has been banned previously? 

Thanks!
2023-10-03 23:59:32 +00:00
realbigsean
7605494791 Use only lighthouse types in the mock builder (#4793)
## Proposed Changes

- only use LH types to avoid build issues
- use warp instead of axum for the server to avoid importing the dep

## Additional Info

- wondering if we can move the `execution_layer/test_utils` to its own crate and import it as a dev dependency
- this would be made easier by separating out our engine API types into their own crate so we can use them in the test crate
- or maybe we can look into using reth types for the engine api if they are in their own crate


Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-03 17:59:28 +00:00
realbigsean
c7ddf1f0b1
add processing and processed caching to the DA checker (#4732)
* add processing and processed caching to the DA checker

* move processing cache out of critical cache

* get it compiling

* fix lints

* add docs to `AvailabilityView`

* some self review

* fix lints

* fix beacon chain tests

* cargo fmt

* make availability view easier to implement, start on testing

* move child component cache and finish test

* cargo fix

* cargo fix

* cargo fix

* fmt and lint

* make blob commitments not optional, rename some caches, add missing blobs struct

* Update beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* marks review feedback and other general cleanup

* cargo fix

* improve availability view docs

* some renames

* some renames and docs

* fix should delay lookup logic

* get rid of some wrapper methods

* fix up single lookup changes

* add a couple docs

* add single blob merge method and improve process_... docs

* update some names

* lints

* fix merge

* remove blob indices from lookup creation log

* remove blob indices from lookup creation log

* delayed lookup logging improvement

* check fork choice before doing any blob processing

* remove unused dep

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/network/src/sync/block_lookups/delayed_lookup.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* remove duplicate deps

* use gen range in random blobs geneartor

* rename processing cache fields

* require block root in rpc block construction and check block root consistency

* send peers as vec in single message

* spawn delayed lookup service from network beacon processor

* fix tests

---------

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-10-03 09:59:33 -04:00
Age Manning
8a1b77bf89 Ultra Fast Super Slick CI (#4755)
Attempting to improve our CI speeds as its recently been a pain point.

Major changes:

 - Use a github action to pull stable/nightly rust rather than building it each run
 - Shift test suite to `nexttest` https://github.com/nextest-rs/nextest for CI
 
 UPDATE:

So I've iterated on some changes, and although I think its still not optimal I think this is a good base to start from. Some extra things in this PR:
- Shifted where we pull rust from. We're now using this thing: https://github.com/moonrepo/setup-rust . It's got some interesting cache's built in, but was not seeing the gains that Jimmy managed to get. In either case tho, it can pull rust, cargofmt, clippy, cargo nexttest all in < 5s. So I think it's worthwhile. 
- I've grouped a few of the check-like tests into a single test called `code-test`. Although we were using github runners in parallel which may be faster, it just seems wasteful. There were like 4-5 tests, where we would pull lighthouse, compile it, then run an action, like clippy, cargo-audit or fmt. I've grouped these into a single action, so we only compile lighthouse once, then in each step we run the checks. This avoids compiling lighthouse like 5 times.
- Ive made doppelganger tests run on our local machines to avoid pulling foundry, building and making lcli which are all now baked into the images. 
- We have sccache and do not incremental compile lighthouse

Misc bonus things:
- Cargo update
- Fix web3 signer openssl keys which is required after a cargo update
- Use mock_instant in an LRU cache test to avoid non-deterministic test
- Remove race condition in building web3signer tests

There's still some things we could improve on. Such as downloading the EF tests every run and the web3-signer binary, but I've left these to be out of scope of this PR. I think the above are meaningful improvements.



Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: antondlr <anton@delaruelle.net>
2023-10-03 06:33:15 +00:00
Jack McPherson
1c98806b6f Allow libp2p to determine listening addresses (#4700)
## Issue Addressed

#4675 

## Proposed Changes

 - Update local ENR (**only port numbers**) with local addresses received from libp2p (via `SwarmEvent::NewListenAddr`)
 - Only use the zero port for CLI tests

## Additional Info

### See Also ###

 - #4705 
 - #4402 
 - #4745
2023-10-03 04:57:20 +00:00
realbigsean
a935daebd5 Clean bors.toml (#4795)
unblock https://github.com/sigp/lighthouse/pull/4755

Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-03 03:37:12 +00:00
realbigsean
67aeb6bf6b
insert cached child at the front of a chain of parent lookups (#4780)
* insert cached child at the front of a chain of parent lookups

* use vecdeque in parent lookup chain of blocks
2023-09-29 12:55:12 -04:00
Jimmy Chen
57edc0f3ce
Add serde(default) to max_per_epoch_activation_churn_limit in spec config so that VC is compatible to older BN versions. (#4783) 2023-09-26 08:48:32 -04:00
realbigsean
9f37d6df77
reduce blob prune logging in forward sync (#4779) 2023-09-26 08:22:26 -04:00
realbigsean
a642bd7de7
Merge pull request #4781 from jimmygchen/merge-unstable-to-deneb-20230926
Merge `unstable` into `deneb-free-blobs`
2023-09-26 08:19:37 -04:00
Jimmy Chen
8f07a96b88
Fix failing tests. 2023-09-26 12:39:58 +10:00
Jimmy Chen
1458394cd9
Fix compilation issues after merging unstable. 2023-09-26 11:46:20 +10:00
Jimmy Chen
7a3cb135d4
Fix tests and add BlockContents decoding. Remove unused builder_threshold field in ApiTesterConfig. 2023-09-26 10:57:21 +10:00
Jimmy Chen
c0b6b92f27
Merge unstable 20230925 into deneb-free-blobs. 2023-09-26 10:32:18 +10:00
Michael Sproul
9244f7f7bc
Improvements to Deneb store upon review (#4693)
* Start testing blob pruning

* Get rid of unnecessary orphaned blob column

* Make random blob tests deterministic

* Test for pruning being blocked by finality

* Fix bugs and test fork boundary

* A few more tweaks to pruning conditions

* Tweak oldest_blob_slot semantics

* Test margin pruning

* Clean up some terminology and lints

* Schema migrations for v18

* Remove FIXME

* Prune blobs on finalization not every slot

* Fix more bugs + tests

* Address review comments
2023-09-25 14:21:54 -04:00
Paul Hauner
441fc1691b Release v4.5.0 (#4768)
## Issue Addressed

NA

## Proposed Changes

Bump versions from v4.4.1 to v4.5.0.

## Additional Info

NA
2023-09-25 05:14:01 +00:00
Lion - dapplion
5c5afafc0d
Update Deneb to 1.4.0-beta.2 (devnet-9) (#4735)
* Add MAX_PER_EPOCH_ACTIVATION_CHURN_LIMIT

* Update tests to 1.4.0-beta.2

* Implement equivocation check for proposer boost

* Use hotfix tests and fix minimal config

* Start updating fork choice tests for Deneb

* Finish implementing fork choice blob handling

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-09-25 15:05:31 +10:00
João Oliveira
0f05499e30 Fix cli options (#4772)
## Issue Addressed

Fixes breaking change introduced on https://github.com/sigp/lighthouse/pull/4674/  that doesn't allow multiple `http_enabled` `ArgGroup` flags
2023-09-22 12:00:51 +00:00
Paul Hauner
fbb6997309 Fix release CI for self-hosted runners (#4770)
## Issue Addressed

NA

## Proposed Changes

Disables some commands for self-hosted runners to prevent failures.

## Additional Info

NA
2023-09-22 11:04:47 +00:00
João Oliveira
dcd69dfc62 Move dependencies to workspace (#4650)
## Issue Addressed

Synchronize dependencies and edition on the workspace `Cargo.toml`

## Proposed Changes

with https://github.com/rust-lang/cargo/issues/8415 merged it's now possible to synchronize details on the workspace `Cargo.toml` like the metadata and dependencies.
By only having dependencies that are shared between multiple crates aligned on the workspace `Cargo.toml` it's easier to not miss duplicate versions of the same dependency and therefore ease on the compile times.

## Additional Info
this PR also removes the no longer required direct dependency of the `serde_derive` crate.

should be reviewed after https://github.com/sigp/lighthouse/pull/4639 get's merged.
closes https://github.com/sigp/lighthouse/issues/4651


Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-09-22 04:30:56 +00:00
antondlr
69c39ad1e5 Use release workflow runners (#4765)
## Issue Addressed

Build releases on self-hosted hardware to speed the process up
2023-09-22 02:33:13 +00:00
Age Manning
6b02e8525a Add new teku bootnodes (#4724)
Adds new Teku bootnodes

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-09-22 02:33:12 +00:00
Jimmy Chen
c4e907de9f Update the voluntary exit endpoint to comply with the key manager specification (#4679)
## Issue Addressed

#4635 

## Proposed Changes

Wrap the `SignedVoluntaryExit` object in a `GenericResponse` container, adding an additional `data` layer, to ensure compliance with the key manager API specification.

The new response would look like this:

```json
{"data":{"message":{"epoch":"196868","validator_index":"505597"},"signature":"0xhexsig"}}
```

This is a backward incompatible change and will affect Siren as well.
2023-09-22 02:33:11 +00:00
João Oliveira
c5588eb66e require http and metrics for respective flags (#4674)
## Issue Addressed

following discussion on https://github.com/sigp/lighthouse/pull/4639#discussion_r1305183750 this PR makes the `http` and `metrics` sub-flags to require those main flags enabled
2023-09-22 02:33:10 +00:00
Paul Hauner
2441a247ab Bump quinn-proto to address rustsec vuln (#4767)
## Issue Addressed

NA

## Proposed Changes

Bumps `quinn-proto` to address a QUIC-related vulnerability: https://rustsec.org/advisories/RUSTSEC-2023-0063

Fixes a `cargo audit` failure.

## Additional Info

NA
2023-09-21 22:37:00 +00:00
antondlr
d0b1abc6fa Update Holesky boot ENR (#4763)
## Issue Addressed

update boot ENR for Holesky relaunch
2023-09-21 22:36:59 +00:00
Michael Sproul
0074a3b5f5 Fix block & state queries prior to genesis (#4761)
## Issue Addressed

Closes #4751

## Proposed Changes

Prevent `state_root_at_slot` and `block_root_at_slot` from erroring out due to a call to `self.slot()?` that fails before genesis. This fixes pre-genesis queries for:

- block at slot 0
- block by genesis block root
- state at slot 0
- state by genesis state root
- state at `finalized` tag
- state at `justified` tag
2023-09-21 06:38:33 +00:00
Jimmy Chen
d3fe3ad337 Update holesky config for relaunch (#4760)
## Issue Addressed

#4759 

Note: Sigma Prime ENR hasn't been updated, tracking it in #4759
2023-09-21 06:38:32 +00:00
Eitan Seri-Levi
992b476eac Add SSZ support to validator block production endpoints (#4534)
## Issue Addressed

#4531 

## Proposed Changes

add SSZ support to the following block production endpoints:

GET /eth/v2/validator/blocks/{slot}
GET /eth/v1/validator/blinded_blocks/{slot}

## Additional Info

i updated a few existing tests to use ssz instead of writing completely new tests
2023-09-21 06:38:31 +00:00
Jimmy Chen
a0478da990 Fix genesis state download panic when running in debug mode (#4753)
## Issue Addressed

#4738 

## Proposed Changes

See the above issue for details. Went with option #2 to use the async reqwest client in `Eth2NetworkConfig` and propagate the async-ness.
2023-09-21 04:17:25 +00:00
realbigsean
082bb2d638 Self hosted docker builds (#4592)
## Issue Addressed

We're OOM'ing on Docker builds on the Deneb branch https://github.com/sigp/lighthouse/issues/3929

Are we ok to self host automated docker builds?


Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <sean@sigmaprime.io>
Co-authored-by: antondlr <anton@delaruelle.net>
2023-09-21 04:17:24 +00:00
Jimmy Chen
fe3bd03234 Fix local testnet to generate keys in the correct folders (#4752)
Fix local testnet to generate keys in the correct folders when `BN_COUNT` and `VC_COUNT` don't match.

The current script place the generated validator keys in validator folders based on the `BN_COUNT` config, e.g. `node_1/validators`, `node_2/validators`..etc. We should be using `VC_COUNT` here instead, otherwise the number of validator clients may not match the number of directories generated, and would result in either:
1. a VC not having any keys  (when `BN_COUNT` < `VC_COUNT`)
2. a validator key directory not being used (when `BN_COUNT` > `VC_COUNT`).
2023-09-21 00:26:56 +00:00
chonghe
f9a3c00518 Update local testnet script (#4733)
There is an issue with the file `scripts/local_testnet/start_local_testnet.sh` - when we use a non-default `$SPEC-PRESET` in `vars.env` it runs into an error: 
```
executing: ./setup.sh >> /home/ck/.lighthouse/local-testnet/testnet/setup.log
parse error: Invalid numeric literal at line 1, column 7
```
@jimmygchen found the issue and the updated script includes the flag `--spec $SPEC-PRESET`
2023-09-21 00:26:55 +00:00
Michael Sproul
5a35278aea Add more checks and logging before genesis (#4730)
## Proposed Changes

This PR adds more logging prior to genesis, particularly on networks that start with execution enabled.

There are new checks using `eth_getBlockByHash/Number` to verify that the genesis state's `latest_execution_payload_header` matches the execution node's genesis block.

The first commit also runs the merge-readiness/Capella-readiness checks prior to genesis. This has two effects:

- Give more information on the execution node's status and its readiness for genesis.
- Prevent the `el_offline` status from being set on `/eth/v1/node/syncing`, which previously caused the VC to complain loudly.

I would like to include this for the Holesky reboot. It would have caught the misconfig that doomed the first Holesky.

## Additional Info

- Geth doesn't serve payload bodies prior to genesis, which is why we use the legacy methods. I haven't checked with other ELs yet.
- Currently this is logging errors with _Capella_ genesis states generated by `ethereum-genesis-generator` because the `withdrawals_root` is not set correctly (it is 0x0). This is not a blocker for Holesky, as it starts from Bellatrix (Pari is investigating).
2023-09-21 00:26:53 +00:00
Jimmy Chen
1e9925435e Reuse fork choice read lock instead of re-acquiring it immediately (#4688)
## Issue Addressed

I went through the code base and look for places where we acquire fork choice locks (after the deadlock bug was found and fixed in #4687), and discovered an instance where we re-acquire a lock immediately after dropping it. This shouldn't cause deadlock like the other issue, but is slightly less efficient.
2023-09-21 00:26:52 +00:00
Michael Sproul
4b6cb3db2c Prevent port re-use in HTTP API tests (#4745)
## Issue Addressed

CI is plagued by `AddrAlreadyInUse` failures, which are caused by race conditions in allocating free ports.

This PR removes all usages of the `unused_port` crate for Lighthouse's HTTP API, in favour of passing `:0` as the listen address. As a result, the listen address isn't known ahead of time and must be read from the listening socket after it binds. This requires tying some self-referential knots, which is a little disruptive, but hopefully doesn't clash too much with Deneb 🤞

There are still a few usages of `unused_tcp4_port` left in cases where we start external processes, like the `watch` Postgres DB, Anvil, Geth, Nethermind, etc. Removing these usages is non-trivial because it's hard to read the port back from an external process after starting it with `--port 0`. We might be able to do something on Linux where we read from `/proc/`, but I'll leave that for future work.
2023-09-20 01:19:03 +00:00
Jimmy Chen
665334e936
Implement SSZ decoding for SignedBlockContents (#4744)
* Implement `SignedBlockContent` decoding and fixed bug in `SignedBlockContent::new`

* Update Cargo.lock file

* Use `make_genesis_spec` to simplify test setup.

* Fix syntax errors.
2023-09-19 18:18:05 -04:00
João Oliveira
d386a07b0c validator client: start http api before genesis (#4714)
## Issue Addressed

On a new network a user might require importing validators before waiting until genesis has occurred.

## Proposed Changes

Starts the validator client http api before waiting for genesis 

## Additional Info

cc @antondlr
2023-09-15 10:08:30 +00:00
Jimmy Chen
b88e57c989 Update Java runtime requirement to 17 for Web3Signer tests (#4681)
Web3Signer now requires Java runtime v17, see [v23.8.0 release](https://github.com/Consensys/web3signer/releases/tag/23.8.0).

We have some Web3Signer tests that requires a compatible Java runtime to be installed on dev machines. This PR updates `setup` documentation in Lighthouse book, and also fixes a small typo.
2023-09-15 08:49:14 +00:00
realbigsean
5f98a7b8ad
Deneb PR feedback updates (#4716)
* move length update outside of if let in LRU cache

* add comment and use hex for G1_POINT_AT_INFINITY

* remove some misleading comments from `ssz_snappy`

* make sure we can't overflow on blobs by range requests with large counts

* downgrade gossip verification internal availability check error

* change blob rpc responses from BlockingFnWithManualSendOnIdle to BlockingFn

* remove unnecessary collect in blobs by range response

* add a comment to blobs by range response start slot logic

* typo persist_data_availabilty_checker -> persist_data_availability_checker

* unify cheap_state_advance_to_obtain_committees
2023-09-15 15:05:48 +10:00
Age Manning
e4ed317b76 Add Experimental QUIC support (#4577)
## Issue Addressed

#4402 

## Proposed Changes

This PR adds QUIC support to Lighthouse. As this is not officially spec'd this will only work between lighthouse <-> lighthouse connections. We attempt a QUIC connection (if the node advertises it) and if it fails we fallback to TCP. 

This should be a backwards compatible modification. We want to test this functionality on live networks to observe any improvements in bandwidth/latency.

NOTE: This also removes the websockets transport as I believe no one is really using it. It should be mentioned in our release however.


Co-authored-by: João Oliveira <hello@jxs.pt>
2023-09-15 03:07:24 +00:00
Pawan Dhananjay
2cfcb51207
Blob references in ckzg (#4723)
* Move to using references in ckzg functions

* cleanup TrustedSetup a bit

* Remove BYTES_PER_FIELD_ELEMENT from KzgPreset
2023-09-14 12:14:30 +10:00
Stefan
4a31e369bf
increase the max topic subscriptions #4581 (#4588)
* increase the max topic subscriptions #4581

* make the max_subscription limitation based off constants / configuration

* format

* wording & add deneb topic array

* reduce max_subscriptions_per_request to 2x

* format

* update comment
2023-09-13 17:06:33 +10:00
realbigsean
8e7b57a794
Merge pull request #4719 from jimmygchen/deneb-merge-from-unstable-20230911
Deneb merge from unstable 20230911
2023-09-12 17:41:42 -04:00
realbigsean
58cd50681c
fix the missing components response code for block publish (#4721) 2023-09-11 09:54:14 -04:00
Jack McPherson
35f47f454f Await listening address from libp2p in RPC tests setup (#4705)
## Issue Addressed

#4704 

## Proposed Changes

 - Receive multiaddr from libp2p by awaiting listener setup

## Additional Info

See also: #4675
2023-09-11 06:14:56 +00:00
Jimmy Chen
d4aedab21f
Fix some compilation errors in tests 2023-09-11 14:07:08 +10:00
Jimmy Chen
6771954c5f
Merge unstable 20230911 into deneb-free-blobs. 2023-09-11 12:09:58 +10:00