Commit Graph

2820 Commits

Author SHA1 Message Date
Michael Sproul
c574f8136e Fix block backfill with genesis skip slots (#4820)
## Issue Addressed

Closes #4817.

## Proposed Changes

- Fill in the linear block roots array between 0 and the slot of the first block (e.g. slots 0 and 1 on Holesky).
- Backport the `--freezer`, `--skip` and `--limit` options for `lighthouse db inspect` from tree-states. This allows us to easily view the database corruption of 4817 using `lighthouse db inspect --network holesky --freezer --column bbr --output values --limit 2`.
- Backport the `iter_column_from` change and `MemoryStore` overhaul from tree-states. These are required to enable `lighthouse db inspect`.
- Rework `freezer_upper_limit` to allow state lookups for slots below the `state_lower_limit`. Currently state lookups will fail until state reconstruction completes entirely.

There is a new regression test for the main bug, but no test for the `freezer_upper_limit` fix because we don't currently support running state reconstruction partially (see #3026). This will be fixed once we merge `tree-states`! In lieu of an automated test, I've tested manually on a Holesky node while it was reconstructing.

## Additional Info

Users who backfilled Holesky to slot 0 (e.g. using `--reconstruct-historic-states`) need to either:

- Re-sync from genesis.
- Re-sync using checkpoint sync and the changes from this PR.

Due to the recency of the Holesky genesis, writing a custom pass to fix up broken databases (which would require its own thorough testing) was deemed unnecessary. This is the primary reason for this PR being marked `backwards-incompat`.

This will create few conflicts with Deneb, which I've already resolved on `tree-states-deneb` and will be happy to backport to Deneb once this PR is merged to unstable.
2023-10-27 05:08:49 +00:00
realbigsean
ba891e1fed deneb related logging improvements (#4859)
1. Add commitments to logs and update the `Display` implementation of `KzgCommitment` to become truncated similarly to block root.

I've been finding it difficult to debug scenarios involving multiple blobs for the same `(index, block_root)`. Logging the commitment will help with this, we can match it to what exists in the block. 

Example output:

```
Oct 20 21:13:36.700 DEBG Successfully verified gossip blob       commitment: 0xa3c1…1cd8, index: 0, root: 0xf31e…f9de, slot: 154568
Oct 20 21:13:36.785 DEBG Successfully verified gossip block      commitments: [0xa3c1…1cd8, 0x8655…02ff, 0x8d6a…955a, 0x84ac…3a1b, 0x9752…629b, 0xb9fc…20fb], root: 0xf31eeb732702e429e89057b15e1c0c631e8452e09e03cb1924353f536ef4f9de, slot: 154568, graffiti: teku/besu, service: beacon
```

Example output in a block with no blobs (this will show up pre-deneb):
```
426734:Oct 20 21:15:24.113 DEBG Successfully verified gossip block, commitments: [], root: 0x619db1360ba0e8d44ae2a0f2450ebca47e167191feecffcfac0e8d7b6c39623c, slot: 154577, graffiti: teku/nethermind, service: beacon, module: beacon_chain::beacon_chain:2765
```

2. Remove `strum::IntoStaticStr` from `AvailabilityCheckError`. This is because `IntoStaticStr` end up dropping information inside the enum. So kzg commitments in this error are dropped, making it more difficult to debug
```
AvailabilityCheckError::KzgCommitmentMismatch {
        blob_commitment: KzgCommitment,
        block_commitment: KzgCommitment,
    },
```
which is output as just `AvailabilityCheckError`

3. Some additional misc sync logs I found useful in debugging https://github.com/sigp/lighthouse/pull/4869

4. This downgrades ”Block returned for single block lookup not present” to debug because I don’t think we can fix the scenario that causes this unless we can cancel inflight rpc requests

Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-25 15:30:17 +00:00
Jimmy Chen
d4f26ee123 Add block roots heal logic in v18 schema migration. (#4875)
## Issue Addressed

Fixes #4697. 

This also unblocks the state pruning PR (#4835). Because self healing breaks if state pruning is applied to a database with missing block roots.

## Proposed Changes

- Fill in the missing block roots between last restore point slot and split slot when upgrading to latest database version.
2023-10-25 03:42:24 +00:00
Pawan Dhananjay
6315a81260 Upgrade to v1.4.0-beta.3 (#4862)
## Issue Addressed

Makes lighthouse compliant with new kzg changes in https://github.com/ethereum/consensus-specs/releases/tag/v1.4.0-beta.3

## Proposed Changes

1. Adds new official trusted setup
2. Refactors kzg to match upstream changes in https://github.com/ethereum/c-kzg-4844/pull/377
3. Updates pre-generated `BlobBundle` to work with official trusted setup. ~~Using json here instead of ssz to account for different value of `MaxBlobCommitmentsPerBlock` in minimal and mainnet. By using json, we can just use one pre generated bundle for both minimal and mainnet. Size of 2 separate ssz bundles is approximately equal to one json bundle cc @jimmygchen~~ 
Dunno what I was doing, ssz works without any issues  
4. Stores trusted_setup as just bytes in eth2_network_config so that we don't have kzg dependency in that lib and in lcli. 


Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-10-21 13:49:27 +00:00
Pawan Dhananjay
074c4951fc Reduce calls to network channel (#4863)
## Issue Addressed

N/A

## Proposed Changes

Sends blocks and blobs from http_api to the network channel for publishing in a single network channel send. This is to avoid overhead of multiple calls.
Also adds a metric for rpc blob retrieval duration.
2023-10-20 19:42:47 +00:00
Jimmy Chen
e8fba8d3a7 Enable BLS portable feature on all CI tests (#4868)
## Issue Addressed

Addresses the recent CI failures caused by caching `blst` for the wrong CPU type. 

## Proposed Changes

- Use `FEATURES: jemalloc,portable` when building Lighthouse & `lcli` in tests
- Add a new `TEST_FEATURES` and set to `portable` for all CI test jobs.
- Updated Makefiles to read the `TEST_FEATURES` environment variable, and default to none.
2023-10-20 07:30:27 +00:00
Zackary Scott
b11988223f #4512 inactivity calculation for Altair (#4807)
## Issue Addressed
#4512 
Which issue # does this PR address?

## Proposed Changes
Add inactivity calculation for Altair

Please list or describe the changes introduced by this PR.
Add inactivity calculation for Altair

## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.


Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-10-20 06:23:28 +00:00
GoodDaisy
90f78d141f fix typos (#4838) 2023-10-19 22:05:15 +00:00
João Oliveira
c6583bb5fa update libp2p (#4864)
## Issue Addressed

updates libp2p to the latest version and uses the new `SwarmBuilder`. Superseeds https://github.com/sigp/lighthouse/pull/4695/
CC @mxinden I don't think we can use both `bandwidth_loggers` with the new syntax right?
2023-10-19 21:22:55 +00:00
João Oliveira
f06391717c collect bandwidth metrics per transport (#4805)
## Issue Addressed

Following the conversation on https://github.com/libp2p/rust-libp2p/pull/3666 the changes introduced in this PR will allow us to give more insights if the bandwidth limitations happen at the transport level, namely if quic helps vs yamux and it's [window size limitation](https://github.com/libp2p/rust-yamux/issues/162) or if the bottleneck is at the gossipsub level.
## Proposed Changes

introduce new quic and tcp bandwidth metric gauges.

cc @mxinden (turned out to be easier, Thomas gave me a hint)
2023-10-19 05:19:22 +00:00
Michael Sproul
192d442718 Fix Rayon deadlock in test utils (#4837)
## Issue Addressed

Fix a deadlock in the tests that was causing tests on tree-states to run for hours without finishing: https://github.com/sigp/lighthouse/actions/runs/6491194654/job/17628138360.

## Proposed Changes

Avoid using a Mutex under the Rayon `par_iter`. Instead, use an `AtomicUsize`. I've run the new version several times in a loop and it hasn't deadlocked (it was deadlocking consistently on tree-states).

## Additional Info

The same bug exists in unstable and tree-states, but I'm not sure why it was triggering so consistently on the tree-states branch.
2023-10-18 13:36:42 +00:00
Jimmy Chen
1b4545cd9d Remove blob clones in KZG verification (#4852)
## Issue Addressed

This PR removes two instances of blob clones during blob verification that may not be necessary.
2023-10-18 06:52:54 +00:00
João Oliveira
f10d3d07c3 remove crit! logging from ListenerClosed event on Ok() (#4821)
## Issue Addressed

Since adding Quic support on https://github.com/sigp/lighthouse/pull/4577, and due to `quinn`s api nature LH now triggers the [`ListenerClosed`](https://docs.rs/libp2p/0.52.3/libp2p/swarm/struct.ListenerClosed.html) event.. @michaelsproul noticed we are logging this event as `crit!` independently of the reason. This PR matches the reason, logging with `debug!` and `error!` (instead of `crit!`) according to its `Result`  
## Additional Info
LH will still log `crit!` until https://github.com/libp2p/rust-libp2p/pull/4621 has been merged
2023-10-18 06:52:52 +00:00
Michael Sproul
5cc0f1097b Fix metric for total block production time (#4794)
## Proposed Changes

Fix the misplacement of the total block production time metric, which occurred during a previous refactor.

Total block production times are no longer skewed low (data from Holesky + blockdreamer):

```
# HELP beacon_block_production_seconds Full runtime of block production
# TYPE beacon_block_production_seconds histogram
beacon_block_production_seconds_bucket{le="0.005"} 0
beacon_block_production_seconds_bucket{le="0.01"} 0
beacon_block_production_seconds_bucket{le="0.025"} 0
beacon_block_production_seconds_bucket{le="0.05"} 0
beacon_block_production_seconds_bucket{le="0.1"} 0
beacon_block_production_seconds_bucket{le="0.25"} 0
beacon_block_production_seconds_bucket{le="0.5"} 37
beacon_block_production_seconds_bucket{le="1"} 65
beacon_block_production_seconds_bucket{le="2.5"} 66
beacon_block_production_seconds_bucket{le="5"} 66
beacon_block_production_seconds_bucket{le="10"} 66
beacon_block_production_seconds_bucket{le="+Inf"} 66
beacon_block_production_seconds_sum 34.225780452
beacon_block_production_seconds_count 66
```

## Additional Info

Cheers to @jimmygchen for helping spot this.
2023-10-18 06:52:50 +00:00
Jimmy Chen
64c156c0c1 Pre-generate test blobs bundle to improve test time. (#4829)
## Issue Addressed

Addresses #4778, and potentially fixes the flaky deneb builder test `builder_works_post_deneb`.

The [deneb builder test](c5c84f1213/beacon_node/http_api/tests/tests.rs (L5371)) has been quite flaky on our CI (`release-tests`) since it was introduced. I'm guessing that it might be timing out on the builder `get_header` call (1 second), and therefore the local payload is used, while the test expects builder payload to be used. 

On my machine the [`get_header` ](c5c84f1213/beacon_node/execution_layer/src/test_utils/mock_builder.rs (L367)) call takes about 550ms, which could easily go over 1s on slower environments (our windows CI runner is much slower than the ubuntu one).

I did a profile on the test and it showed that `blob_to_kzg_commiment` and `compute_kzg_proof` was taking a large chunk of time, so perhaps pre-generating the blobs could help stablise this test.

## Proposed Changes

Pre-generate blobs bundle for Mainnet and Minimal presets.

Before the change `get_header` took about **550ms**, and it's now reduced to **50-55ms** after the change. If timeout was indeed the cause of the flaky test, this fix should stablise it. This also brings the flaky `builder_works_post_deneb` test time from 50s to 10s. (8s if we only use a single blob)
2023-10-18 04:40:29 +00:00
realbigsean
283ec8cf24
Deneb pr updates 2 (#4851)
* use workspace deps in kzg crate

* delete unused blobs dp path field

* full match on fork name in engine api get payload v3

* only accept v3 payloads on get payload v3 endpoint in mock el

* remove FIXMEs related to merge transition tests

* move static tx to test utils

* default max_per_epoch_activation_churn_limit to mainnet value

* remove unnecessary async

* remove comment

* use task executor in `blob_sidecars` endpoint
2023-10-17 09:53:46 +11:00
Age Manning
cf544b3996
Very minor own nitpicks (#4845) 2023-10-16 16:30:14 +11:00
Michael Sproul
2d662f78ae
Use Deneb fork in generate_genesis_header
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-10-16 16:24:27 +11:00
Jimmy Chen
38e7172508
Add blob_sidecar event to SSE (#4790)
* Add `blob_sidecar` event to SSE.

* Return 202 if a block is published but failed blob validation when validation level is `Gossip`.

* Move `BlobSidecar` event to `process_gossip_blob` and add test.

* Emit `BlobSidecar` event when blobs are received over rpc.

* Improve test assertions on `SseBlobSidecar`s.

* Add quotes to blob index serialization in `SseBlobSidecar`

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-10-12 10:13:08 -04:00
realbigsean
4555e33048
Remove serde derive references (#4830)
* remove remaining uses of serde_derive

* fix lockfile

---------

Co-authored-by: João Oliveira <hello@jxs.pt>
2023-10-11 13:01:30 -04:00
ethDreamer
8660043024
Prevent Overflow LRU Cache from Exploding (#4801)
* Initial Commit of State LRU Cache

* Build State Caches After Reconstruction

* Cleanup Duplicated Code in OverflowLRUCache Tests

* Added Test for State LRU Cache

* Prune Cache of Old States During Maintenance

* Address Michael's Comments

* Few More Comments

* Removed Unused impl

* Last touch up

* Fix Clippy
2023-10-10 23:51:00 -05:00
Jimmy Chen
4ad7e15732
Address Clippy 1.73 lints on Deneb branch (#4810)
* Address Clippy 1.73 lints (#4809)

## Proposed Changes

Fix Clippy lints enabled by default in Rust 1.73.0, released today.

* Address Clippy 1.73 lints.

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-10-06 12:23:57 +05:30
Jimmy Chen
c5c84f1213
Merge branch 'unstable' into merge-unstable-to-deneb-20231005
# Conflicts:
#	.github/workflows/test-suite.yml
#	Cargo.lock
#	beacon_node/execution_layer/Cargo.toml
#	beacon_node/execution_layer/src/test_utils/mock_builder.rs
#	beacon_node/execution_layer/src/test_utils/mod.rs
#	beacon_node/network/src/service/tests.rs
#	consensus/types/src/builder_bid.rs
2023-10-05 15:54:44 +11:00
Akihito Nakano
ba8bcf4bd3 Remove deficit gossipsub scoring during topic transition (#4486)
## Issue Addressed

This PR closes https://github.com/sigp/lighthouse/issues/3237

## Proposed Changes

Remove topic weight of old topics when the fork happens.

## Additional Info

- Divided `NetworkService::start()` into `NetworkService::build()` and `NetworkService::start()` for ease of testing.
2023-10-04 00:43:28 +00:00
Michael Sproul
6ec649a4e2 Optimise head block root API (#4799)
## Issue Addressed

We've had a report of sync committee performance suffering with the beacon processor HTTP API prioritisations.

## Proposed Changes

Increase the priority of `/eth/v1/beacon/blocks/head/root` requests, which are used by the validator client to form sync committee messages, here:

441fc1691b/validator_client/src/sync_committee_service.rs (L181-L188)

Additionally, avoid loading the blinded block in all but the `block_id=block_root` case. I'm not sure why we were doing this previously, I suspect it was just an oversight during the implementation of the `finalized` status on API requests.

## Additional Info

I think this change should have minimal negative impact as:

- The block root endpoint is quick to compute (a few ms max).
- Only the priority of `head` requests is increased. Analytical processes that are making lots of block root requests for past slots are unable to DoS the beacon processor, as their requests will still be processed after attestations.
2023-10-03 23:59:37 +00:00
Pawan Dhananjay
5bab9b866e Don't downscore peers on duplicate blocks (#4791)
## Issue Addressed

N/A

## Proposed Changes

We were currently downscoring a peer for sending us a block that we already have in fork choice. This is unnecessary as we get duplicates in lighthouse only when
1. We published the block, so the block is already in fork choice
2. We imported the same block over rpc

In both scenarios, the peer who sent us the block over gossip is not at fault.

This isn't exploitable as valid duplicates will get dropped by the gossipsub duplicate filter
2023-10-03 23:59:35 +00:00
Divma
f11884ccdb enforce non zero enr ports (#4776)
## Issue Addressed

Right now lighthouse accepts zero as enr ports. Since enr ports should be reachable, zero ports should be rejected here

## Proposed Changes

- update the config to use `NonZerou16` as an ENR port for all enr-related fields.
- the enr builder from config now sets the enr to the listening port only if the enr port is not already set (prev behaviour) and the listening port is not zero (new behaviour)
- reject zero listening ports when used with `enr-match`. 
- boot node now rejects listening port as zero, since those are advertised.
- generate-bootnode-enr also rejected zero listening ports for the same reason.
- update local network scripts

## Additional Info

Unrelated, but why do we overwrite `enr-x-port` values with listening ports if `enr-match` is present? we prob should only do this for enr values that are not already set.
2023-10-03 23:59:34 +00:00
João Oliveira
0dc95a1d37 PeerManager: move the check for banned peers from connection_established (#4569)
## Issue Addressed
https://github.com/sigp/lighthouse/issues/4543
## Proposed Changes
- Removes `NotBanned` from `BanResult`, implements `Display` and `std::error::Error` for `BanResult` and changes `ban_result` return type to `Option<BanResult>` which helps returning `BanResult` on `handle_established_inbound_connection`  
- moves the check from for banned peers from `on_connection_established` to `handle_established_inbound_connection` to start addressing #4543.
- Removes `allow_block_list` as it's now redundant? Not sure about this one but if `PeerManager` keeps track of the banned peers, no need to send a `Swarm` event for `alow_block_list` to also keep that list right? 
 
## Questions

-  #4543 refers:
>  More specifically, implement the connection limit behaviour inside the peer manager.

@AgeManning do you mean copying `libp2p::connection_limits::Behaviour`'s code into `PeerManager`/ having it as an inner `NetworkBehaviour` of `PeerManager`/other? If it's the first two, I think it probably makes more sense to have it as it is as it's less code to maintain.

> Also implement the banning of peers inside the behaviour, rather than passing messages back up to the swarm.

I tried to achieve this, but we still need to pass the `PeerManagerEvent::Banned` swarm event as `DiscV5` handles it's node and ip management internally and I did not find a method to query if a peer is banned. Is there anything else we can do from here?

3397612160/beacon_node/lighthouse_network/src/discovery/mod.rs (L931-L940)

Same as the question above, I did not find a way to check if `DiscV5` has the peer banned, so that we could check here and avoid sending `Swarm` events

3397612160/beacon_node/lighthouse_network/src/peer_manager/network_behaviour.rs (L168-L178)

Is there a chance we try to dial a peer that has been banned previously? 

Thanks!
2023-10-03 23:59:32 +00:00
realbigsean
7605494791 Use only lighthouse types in the mock builder (#4793)
## Proposed Changes

- only use LH types to avoid build issues
- use warp instead of axum for the server to avoid importing the dep

## Additional Info

- wondering if we can move the `execution_layer/test_utils` to its own crate and import it as a dev dependency
- this would be made easier by separating out our engine API types into their own crate so we can use them in the test crate
- or maybe we can look into using reth types for the engine api if they are in their own crate


Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-03 17:59:28 +00:00
realbigsean
c7ddf1f0b1
add processing and processed caching to the DA checker (#4732)
* add processing and processed caching to the DA checker

* move processing cache out of critical cache

* get it compiling

* fix lints

* add docs to `AvailabilityView`

* some self review

* fix lints

* fix beacon chain tests

* cargo fmt

* make availability view easier to implement, start on testing

* move child component cache and finish test

* cargo fix

* cargo fix

* cargo fix

* fmt and lint

* make blob commitments not optional, rename some caches, add missing blobs struct

* Update beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* marks review feedback and other general cleanup

* cargo fix

* improve availability view docs

* some renames

* some renames and docs

* fix should delay lookup logic

* get rid of some wrapper methods

* fix up single lookup changes

* add a couple docs

* add single blob merge method and improve process_... docs

* update some names

* lints

* fix merge

* remove blob indices from lookup creation log

* remove blob indices from lookup creation log

* delayed lookup logging improvement

* check fork choice before doing any blob processing

* remove unused dep

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/network/src/sync/block_lookups/delayed_lookup.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* remove duplicate deps

* use gen range in random blobs geneartor

* rename processing cache fields

* require block root in rpc block construction and check block root consistency

* send peers as vec in single message

* spawn delayed lookup service from network beacon processor

* fix tests

---------

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-10-03 09:59:33 -04:00
Jack McPherson
1c98806b6f Allow libp2p to determine listening addresses (#4700)
## Issue Addressed

#4675 

## Proposed Changes

 - Update local ENR (**only port numbers**) with local addresses received from libp2p (via `SwarmEvent::NewListenAddr`)
 - Only use the zero port for CLI tests

## Additional Info

### See Also ###

 - #4705 
 - #4402 
 - #4745
2023-10-03 04:57:20 +00:00
realbigsean
67aeb6bf6b
insert cached child at the front of a chain of parent lookups (#4780)
* insert cached child at the front of a chain of parent lookups

* use vecdeque in parent lookup chain of blocks
2023-09-29 12:55:12 -04:00
realbigsean
9f37d6df77
reduce blob prune logging in forward sync (#4779) 2023-09-26 08:22:26 -04:00
Jimmy Chen
8f07a96b88
Fix failing tests. 2023-09-26 12:39:58 +10:00
Jimmy Chen
1458394cd9
Fix compilation issues after merging unstable. 2023-09-26 11:46:20 +10:00
Jimmy Chen
7a3cb135d4
Fix tests and add BlockContents decoding. Remove unused builder_threshold field in ApiTesterConfig. 2023-09-26 10:57:21 +10:00
Jimmy Chen
c0b6b92f27
Merge unstable 20230925 into deneb-free-blobs. 2023-09-26 10:32:18 +10:00
Michael Sproul
9244f7f7bc
Improvements to Deneb store upon review (#4693)
* Start testing blob pruning

* Get rid of unnecessary orphaned blob column

* Make random blob tests deterministic

* Test for pruning being blocked by finality

* Fix bugs and test fork boundary

* A few more tweaks to pruning conditions

* Tweak oldest_blob_slot semantics

* Test margin pruning

* Clean up some terminology and lints

* Schema migrations for v18

* Remove FIXME

* Prune blobs on finalization not every slot

* Fix more bugs + tests

* Address review comments
2023-09-25 14:21:54 -04:00
Paul Hauner
441fc1691b Release v4.5.0 (#4768)
## Issue Addressed

NA

## Proposed Changes

Bump versions from v4.4.1 to v4.5.0.

## Additional Info

NA
2023-09-25 05:14:01 +00:00
Lion - dapplion
5c5afafc0d
Update Deneb to 1.4.0-beta.2 (devnet-9) (#4735)
* Add MAX_PER_EPOCH_ACTIVATION_CHURN_LIMIT

* Update tests to 1.4.0-beta.2

* Implement equivocation check for proposer boost

* Use hotfix tests and fix minimal config

* Start updating fork choice tests for Deneb

* Finish implementing fork choice blob handling

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-09-25 15:05:31 +10:00
João Oliveira
0f05499e30 Fix cli options (#4772)
## Issue Addressed

Fixes breaking change introduced on https://github.com/sigp/lighthouse/pull/4674/  that doesn't allow multiple `http_enabled` `ArgGroup` flags
2023-09-22 12:00:51 +00:00
João Oliveira
dcd69dfc62 Move dependencies to workspace (#4650)
## Issue Addressed

Synchronize dependencies and edition on the workspace `Cargo.toml`

## Proposed Changes

with https://github.com/rust-lang/cargo/issues/8415 merged it's now possible to synchronize details on the workspace `Cargo.toml` like the metadata and dependencies.
By only having dependencies that are shared between multiple crates aligned on the workspace `Cargo.toml` it's easier to not miss duplicate versions of the same dependency and therefore ease on the compile times.

## Additional Info
this PR also removes the no longer required direct dependency of the `serde_derive` crate.

should be reviewed after https://github.com/sigp/lighthouse/pull/4639 get's merged.
closes https://github.com/sigp/lighthouse/issues/4651


Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-09-22 04:30:56 +00:00
João Oliveira
c5588eb66e require http and metrics for respective flags (#4674)
## Issue Addressed

following discussion on https://github.com/sigp/lighthouse/pull/4639#discussion_r1305183750 this PR makes the `http` and `metrics` sub-flags to require those main flags enabled
2023-09-22 02:33:10 +00:00
Michael Sproul
0074a3b5f5 Fix block & state queries prior to genesis (#4761)
## Issue Addressed

Closes #4751

## Proposed Changes

Prevent `state_root_at_slot` and `block_root_at_slot` from erroring out due to a call to `self.slot()?` that fails before genesis. This fixes pre-genesis queries for:

- block at slot 0
- block by genesis block root
- state at slot 0
- state by genesis state root
- state at `finalized` tag
- state at `justified` tag
2023-09-21 06:38:33 +00:00
Eitan Seri-Levi
992b476eac Add SSZ support to validator block production endpoints (#4534)
## Issue Addressed

#4531 

## Proposed Changes

add SSZ support to the following block production endpoints:

GET /eth/v2/validator/blocks/{slot}
GET /eth/v1/validator/blinded_blocks/{slot}

## Additional Info

i updated a few existing tests to use ssz instead of writing completely new tests
2023-09-21 06:38:31 +00:00
Jimmy Chen
a0478da990 Fix genesis state download panic when running in debug mode (#4753)
## Issue Addressed

#4738 

## Proposed Changes

See the above issue for details. Went with option #2 to use the async reqwest client in `Eth2NetworkConfig` and propagate the async-ness.
2023-09-21 04:17:25 +00:00
Michael Sproul
5a35278aea Add more checks and logging before genesis (#4730)
## Proposed Changes

This PR adds more logging prior to genesis, particularly on networks that start with execution enabled.

There are new checks using `eth_getBlockByHash/Number` to verify that the genesis state's `latest_execution_payload_header` matches the execution node's genesis block.

The first commit also runs the merge-readiness/Capella-readiness checks prior to genesis. This has two effects:

- Give more information on the execution node's status and its readiness for genesis.
- Prevent the `el_offline` status from being set on `/eth/v1/node/syncing`, which previously caused the VC to complain loudly.

I would like to include this for the Holesky reboot. It would have caught the misconfig that doomed the first Holesky.

## Additional Info

- Geth doesn't serve payload bodies prior to genesis, which is why we use the legacy methods. I haven't checked with other ELs yet.
- Currently this is logging errors with _Capella_ genesis states generated by `ethereum-genesis-generator` because the `withdrawals_root` is not set correctly (it is 0x0). This is not a blocker for Holesky, as it starts from Bellatrix (Pari is investigating).
2023-09-21 00:26:53 +00:00
Jimmy Chen
1e9925435e Reuse fork choice read lock instead of re-acquiring it immediately (#4688)
## Issue Addressed

I went through the code base and look for places where we acquire fork choice locks (after the deadlock bug was found and fixed in #4687), and discovered an instance where we re-acquire a lock immediately after dropping it. This shouldn't cause deadlock like the other issue, but is slightly less efficient.
2023-09-21 00:26:52 +00:00
Michael Sproul
4b6cb3db2c Prevent port re-use in HTTP API tests (#4745)
## Issue Addressed

CI is plagued by `AddrAlreadyInUse` failures, which are caused by race conditions in allocating free ports.

This PR removes all usages of the `unused_port` crate for Lighthouse's HTTP API, in favour of passing `:0` as the listen address. As a result, the listen address isn't known ahead of time and must be read from the listening socket after it binds. This requires tying some self-referential knots, which is a little disruptive, but hopefully doesn't clash too much with Deneb 🤞

There are still a few usages of `unused_tcp4_port` left in cases where we start external processes, like the `watch` Postgres DB, Anvil, Geth, Nethermind, etc. Removing these usages is non-trivial because it's hard to read the port back from an external process after starting it with `--port 0`. We might be able to do something on Linux where we read from `/proc/`, but I'll leave that for future work.
2023-09-20 01:19:03 +00:00
realbigsean
5f98a7b8ad
Deneb PR feedback updates (#4716)
* move length update outside of if let in LRU cache

* add comment and use hex for G1_POINT_AT_INFINITY

* remove some misleading comments from `ssz_snappy`

* make sure we can't overflow on blobs by range requests with large counts

* downgrade gossip verification internal availability check error

* change blob rpc responses from BlockingFnWithManualSendOnIdle to BlockingFn

* remove unnecessary collect in blobs by range response

* add a comment to blobs by range response start slot logic

* typo persist_data_availabilty_checker -> persist_data_availability_checker

* unify cheap_state_advance_to_obtain_committees
2023-09-15 15:05:48 +10:00