Commit Graph

426 Commits

Author SHA1 Message Date
Eitan Seri-Levi
01ec42e75a
Fix Rust beta compiler errors 1.78.0-beta.1 (#5439)
* remove redundant imports

* fix test

* contains key

* fmt

* Merge branch 'unstable' into fix-beta-compiler
2024-03-20 05:17:02 +00:00
Pawan Dhananjay
84a902a589
Reduce load on validator subscription channels (#5311)
* Fix tests

* Merge branch 'unstable' into unclog-channels

* Avoid reallocations

* Reduce subscription load on beacon node
2024-03-07 12:32:27 +00:00
Michael Sproul
bf118a17d4
Fix block v3 header decoding (#5366)
* Fix block v3 header decoding
2024-03-07 03:31:06 +00:00
João Oliveira
a89ff100af
improve libp2p connected peer metrics (#5314)
* patch rust-yamux dep

* improve libp2p connected peer metrics
2024-02-28 03:52:55 +00:00
Jimmy Chen
50c423ad88
Revert libp2p metrics (#4870) (#5265)
* Revert "improve libp2p connected peer metrics (#4870)"

This reverts commit 0c3fef59b3.
2024-02-20 04:19:17 +00:00
Michael Sproul
f17fb291b7
Handle unknown head during attestation publishing (#5010)
* Handle unknown head during attestation publishing

* Merge remote-tracking branch 'origin/unstable' into queue-http-attestations

* Simplify task spawner

* Improve logging

* Add a test

* Improve error logging

* Merge remote-tracking branch 'origin/unstable' into queue-http-attestations

* Fix beta compiler warnings
2024-02-15 12:24:47 +00:00
Eitan Seri-Levi
e7ef2a3a54
validator liveness endpoint should accept string encoded indices (#5184)
* deserialize string indices as u64

* client should send quoted indices
2024-02-09 04:59:39 +00:00
João Oliveira
0c3fef59b3
improve libp2p connected peer metrics (#4870)
* improve libp2p connected peer metrics

* separate discv5 port from libp2p for NAT open

* use metric family for DISCOVERY_BYTES

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into improve-metrics
2024-02-08 02:40:54 +00:00
Jimmy Chen
39e9f7dc6b
Fix Rust beta compiler errors (1.77) (#5180)
* Lint fixes

* More fixes for beta compiler.

* Format fixes

* Move `#[allow(dead_code)]` to field level.

* Remove old comment.

* Update beacon_node/execution_layer/src/test_utils/mod.rs

Co-authored-by: João Oliveira <hello@jxs.pt>

* remove duplicate line
2024-02-05 17:54:11 +00:00
Lion - dapplion
b035638f9b
Compute recent lightclient updates (#4969)
* Compute recent lightclient updates

* Review PR

* Merge remote-tracking branch 'upstream/unstable' into lc-prod-recent-updates

* Review PR

* consistent naming

* add metrics

* revert dropping reprocessing queue

* Update light client optimistic update re-processing logic. (#7)

* Add light client server simulator tests. Co-authored by @dapplion.

* Merge branch 'unstable' into fork/dapplion/lc-prod-recent-updates

* Fix lint

* Enable light client server in simulator test.

* Fix test for light client optimistic updates and finality updates.
2024-01-31 05:25:51 +00:00
Eitan Seri-Levi
1d87edb03d
Assume Content-Type is json for endpoints that require json (#4575)
* added default content type filter

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into unstable

* create custom warp json filter that ignores content type header

* cargo fmt and linting

* updated test

* updated test

* merge unstable

* merge conflicts

* workspace=true

* use Bytes instead of Buf

* resolve merge conflict

* resolve merge conflicts

* add extra error message context

* merge conflicts

* lint
2024-01-31 01:11:54 +00:00
Jack McPherson
abeb358f0b
Remove custom SSZ beacon states route (#5065)
* Remove SSZ state root route

* Remove SSZ states route from client impl

* Patch tests

* Merge branch 'unstable' into 5063-delete-ssz-state-route

* Further remove dead code
2024-01-30 00:33:05 +00:00
Pawan Dhananjay
b55b58b3c6
Fix indices filter in blobs_sidecar http endpoint (#5118)
* add get blobs unit test

* Use a multi_key_query for blob_sidecar indices

* Fix test

* Remove env_logger

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2024-01-24 09:35:24 +11:00
Michael Sproul
a403138ed0
Reduce size of futures in HTTP API to prevent stack overflows (#5104)
* Box::pin a few big futures

* Arc the blocks early in publication

* Fix more tests
2024-01-23 15:32:07 +11:00
Michael Sproul
185646acb2
Fix PublishBlockRequest SSZ decoding (#5078)
* Fix PublishBlockRequest SSZ decoding

* Send header for block v1 ssz client

* Delete unused function
2024-01-19 12:56:30 +11:00
Eitan Seri-Levi
72bcf47dd0
add content-type octet stream helper fn (#5062) 2024-01-15 16:23:39 +11:00
João Oliveira
38df87c3c5
Switch libp2p sigp gossipsub fork (#4999)
* switch libp2p source to sigp fork

* Shift the connection closing inside RPC behaviour

* Tag specific commits

* Add slow peer scoring

* Fix test

* Use default yamux config

* Pin discv5 to our libp2p fork and cargo update

* Upgrade libp2p to enable yamux gains

* Add a comment specifying the branch being used

* cleanup build output from within container
(prevents CI warnings related to fs permissions)

* Remove revision tags add branches for testing, will revert back once we're happy

* Update to latest rust-libp2p version

* Pin forks

* Update cargo.lock

* Re-pin to panic-free rust

---------

Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: antondlr <anton@delaruelle.net>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2024-01-10 16:26:52 +11:00
Michael Sproul
7e948eec9d
Fix block v3 reward encodings (#5049)
* Fix block v3 reward encodings

* Use crates.io version
2024-01-10 10:44:07 +11:00
zhiqiangxu
4a65d28b3b
remove needless Into (#5026) 2024-01-10 08:38:16 +11:00
realbigsean
f70c32ec70
create unified slashing cache (#5033)
* create unified slashing cache

* add observed slashable file

* fix broadcast validation tests

* revert block seen cache changes

* clean up slashable cache test

* check header signatures for RPC blobs

* don't throw error on RPC signature invalie
2024-01-08 10:30:57 -05:00
Eitan Seri-Levi
9c1505d082
Block v3 builder boost factor (#5035)
* builder boost factor

* default boost factor

* revert

* deprecate always_prefer_builder_payload, builder-profit-threshold, ignore_builder_override_suggestion_threshold and builder_comparison_factor flags

* revert

* set deprecated flags to no op, revert should_override_builder

* fix test, calc boosted relay value correctly, dont calculate if none

* Add deprecation warnings and restore CLI docs
2024-01-08 11:10:32 +11:00
Michael Sproul
af11e78ae1
Clean up blockv3 metadata and client (#5015)
* Improve block production v3 client

* Delete wayward line

* Overhaul JSON endpoint as well

* Rename timeout param

* Update tests

* I broke everything

* Ah this is an insane fix

* Remove unnecessary optionals

* Doc fix
2023-12-22 09:39:17 -05:00
realbigsean
c55608be10
suppress error on duplicate blobs (#4995) 2023-12-18 12:15:12 -05:00
Jimmy Chen
b0c374c1ca
Update dependencies to get rid of the yanked deps (#4994)
* Initial attempt to upgrade hashbrown to latest to get rid of the crate warnings.

* Replace `.expect()` usage with use of `const`.

* Update ahash 0.7 as well

* Remove unsafe code.

* Update `lru` to 0.12 and fix release test errors.

* Set non-blocking socket

* Bump testcontainers to 0.15.

* Fix lint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-12-15 18:31:59 +11:00
Michael Sproul
b882519d2f
Implement POST validators/validator_balances APIs (#4872)
* Add POST for fetching validators from state

* Implement POST for balances

* Tests
2023-12-08 12:09:36 +11:00
Pawan Dhananjay
31044402ee
Sidecar inclusion proof (#4900)
* Refactor BlobSidecar to new type

* Fix some compile errors

* Gossip verification compiles

* Fix http api types take 1

* Fix another round of compile errors

* Beacon node crate compiles

* EF tests compile

* Remove all blob signing from VC

* fmt

* Tests compile

* Fix some tests

* Fix more http tests

* get compiling

* Fix gossip conditions and tests

* Add basic proof generation and verification

* remove unnecessary ssz decode

* add back build_sidecar

* remove default at fork for blobs

* fix beacon chain tests

* get relase tests compiling

* fix lints

* fix existing spec tests

* add new ef tests

* fix gossip duplicate rule

* lints

* add back sidecar signature check in gossip

* add finalized descendant check to blob sidecar gossip

* fix error conversion

* fix release tests

* sidecar inclusion self review cleanup

* Add proof verification and computation metrics

* Remove accidentally committed file

* Unify some block and blob errors; add slashing conditions for sidecars

* Address review comment

* Clean up re-org tests (#4957)

* Address more review comments

* Add Comments & Eliminate Unnecessary Clones

* update names

* Update beacon_node/beacon_chain/src/metrics.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update beacon_node/network/src/network_beacon_processor/tests.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* pr feedback

* fix test compile

* Sidecar Inclusion proof small refactor and updates (#4967)

* Update some comments, variables and small cosmetic fixes.

* Couple blobs and proofs into a tuple in `PayloadAndBlobs` for simplicity and safety.

* Update function comment.

* Update testing/ef_tests/src/cases/merkle_proof_validity.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Rename the block and blob wrapper types used in the beacon API interfaces.

* make sure gossip invalid blobs are passed to the slasher (#4970)

* Add blob headers to slasher before adding to DA checker

* Replace Vec with HashSet in BlockQueue

* fmt

* Rename gindex -> index

* Simplify gossip condition

---------

Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <sean@sigmaprime.io>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-12-05 11:19:59 -05:00
Gua00va
44aaf13ff0
Standard Liveness Endpoint (#4853)
* Changes to use required Endpoint

* Format

* fixed doppleganger service

* minor fix

* efficiency changes

* fixed tests

* remove commented line

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-30 17:41:22 +11:00
GeemoCandama
8a599ec7dc
API for LightClientBootstrap, LightClientFinalityUpdate, LightClientOptimisticUpdate and light client events (#3954)
* rebase and add comment

* conditional test

* test

* optimistic chould be working now

* finality should be working now

* try again

* try again

* clippy fix

* add lc bootstrap beacon api

* add lc optimistic/finality update to events

* fmt

* That error isn't occuring on my computer but I think this should fix it

* Add missing test file

* Update light client types to comply with Altair light client spec.

* Fix test compilation

* Support deserializing light client structures for the Bellatrix fork

* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.

* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses

* Add light client bootstrap API test and fix existing ones.

* Fix test for `light-client-server` http api config.

* Appease clippy

* Efficiency improvement when retrieving beacon state.

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-28 17:14:29 +11:00
Eitan Seri-Levi
98f159cc18 fmt 2023-11-20 21:37:58 -08:00
Eitan Seri-Levi
24a0a7ffd0 remove cache check 2023-11-20 21:37:04 -08:00
Eitan Seri-Levi
228180bb35 add some more block v3 tests 2023-11-20 20:46:13 -08:00
Jimmy Chen
6b63d18420
Fix Rust beta compiler warnings (rustc 1.75.0-beta.1 (782883f60 2023-11-12)) (#4932) 2023-11-18 03:55:11 +11:00
Eitan Seri-Levi
a380f6ef1f
Add block-v3 SSZ tests (#4902)
* create block-v3 ssz tests

* fmt
2023-11-09 15:05:35 +11:00
Joel Rousseau
ac8811afac
Add missed blocks to monitored validators (#4731)
* add missed_block metric

* init missed_block in constructor

* declare beaconproposercache in ValidatorMonitor

* refacto proposer_shuffling_decision_root to use epoch instead of current.epoch

* imple new proposer_shuffling_decision_root in callers

* push missed_blocks

* prune missed_blocks

* only add to hashmap if it's a monitored validator

* remove current_epoch dup + typos

* extract in func

* add prom metrics

* checkpoint is not only epoch but slot as well

* add safeguard if we start a new chain at slot 0

* clean

* remove unnecessary negative value for a slot

* typo in comment

* remove unused current_epoch

* share beacon_proposer_cache between validator_monitor and beacon_chain

* pass Hash256::zero()

* debug objects

* fix loop: lag is at the head

* sed s/get_slot/get_epoch

* fewer calls to cache.get_epoch

* fix typos

* remove cache first call

* export TYPICAL_SLOTS_PER_EPOCH and use it in validator_monitor

* switch to gauge & loop over missed_blocks hashset

* fix subnet_service tests

* remove unused var

* clean + fix nits

* add beacon_proposer_cache + validator_monitor in builder

* fix store_tests

* fix builder tests

* add tests

* add validator monitor set of tests

* clean tests

* nits

* optimise imports

* lint

* typo

* added self.aggregatable

* duplicate proposer_shuffling_decision_root

* remove duplication in passing beacon_proposer_cache

* remove duplication in passing beacon_proposer_cache

* using indices

* fmt

* implement missed blocks total

* nits

* avoid heap allocation

* remove recursion limit

* fix lint

* Fix valdiator monitor builder pattern

Unify validator monitor config struct

* renaming metrics

* renaming metrics in validator monitor

* add log if there's a missing validator index

* consistent log

* fix loop

* better loop

* move gauge to counter

* fmt

* add error message

* lint

* fix prom metrics

* set gauge to 0 when non-finalized epochs

* better wording

* remove hash256::zero in favour of block_root

* fix gauge total label

* fix last missed block validator

* Add `MissedBlock` struct

* Fix comment

* Refactor non-finalized block loop

* Fix off-by-one

* Avoid string allocation

* Fix compile error

* Remove non-finalized blocks metric

* fix func clojure

* remove unused variable

* remove unused DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD

* remove unused DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD in builder

* add validator index depending on the fork name

* typos

---------

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-11-09 15:05:14 +11:00
Eitan Seri-Levi
07f53b18fc Block v3 endpoint (#4629)
## Issue Addressed

#4582

## Proposed Changes

Add a new v3 block fetching flow that can decide to return a Full OR Blinded payload

## Additional Info



Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-11-03 00:12:18 +00:00
Eitan Seri-Levi
4ce01ddd11 Activate clippy::manual_let_else lint (#4889)
## Issue Addressed

#4888

## Proposed Changes

Enabled `clippy::manual_let_else` lint and resolved the warning messages.
2023-10-31 10:31:02 +00:00
Pawan Dhananjay
074c4951fc Reduce calls to network channel (#4863)
## Issue Addressed

N/A

## Proposed Changes

Sends blocks and blobs from http_api to the network channel for publishing in a single network channel send. This is to avoid overhead of multiple calls.
Also adds a metric for rpc blob retrieval duration.
2023-10-20 19:42:47 +00:00
realbigsean
283ec8cf24
Deneb pr updates 2 (#4851)
* use workspace deps in kzg crate

* delete unused blobs dp path field

* full match on fork name in engine api get payload v3

* only accept v3 payloads on get payload v3 endpoint in mock el

* remove FIXMEs related to merge transition tests

* move static tx to test utils

* default max_per_epoch_activation_churn_limit to mainnet value

* remove unnecessary async

* remove comment

* use task executor in `blob_sidecars` endpoint
2023-10-17 09:53:46 +11:00
Jimmy Chen
38e7172508
Add blob_sidecar event to SSE (#4790)
* Add `blob_sidecar` event to SSE.

* Return 202 if a block is published but failed blob validation when validation level is `Gossip`.

* Move `BlobSidecar` event to `process_gossip_blob` and add test.

* Emit `BlobSidecar` event when blobs are received over rpc.

* Improve test assertions on `SseBlobSidecar`s.

* Add quotes to blob index serialization in `SseBlobSidecar`

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-10-12 10:13:08 -04:00
Jimmy Chen
4ad7e15732
Address Clippy 1.73 lints on Deneb branch (#4810)
* Address Clippy 1.73 lints (#4809)

## Proposed Changes

Fix Clippy lints enabled by default in Rust 1.73.0, released today.

* Address Clippy 1.73 lints.

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-10-06 12:23:57 +05:30
Jimmy Chen
c5c84f1213
Merge branch 'unstable' into merge-unstable-to-deneb-20231005
# Conflicts:
#	.github/workflows/test-suite.yml
#	Cargo.lock
#	beacon_node/execution_layer/Cargo.toml
#	beacon_node/execution_layer/src/test_utils/mock_builder.rs
#	beacon_node/execution_layer/src/test_utils/mod.rs
#	beacon_node/network/src/service/tests.rs
#	consensus/types/src/builder_bid.rs
2023-10-05 15:54:44 +11:00
Michael Sproul
6ec649a4e2 Optimise head block root API (#4799)
## Issue Addressed

We've had a report of sync committee performance suffering with the beacon processor HTTP API prioritisations.

## Proposed Changes

Increase the priority of `/eth/v1/beacon/blocks/head/root` requests, which are used by the validator client to form sync committee messages, here:

441fc1691b/validator_client/src/sync_committee_service.rs (L181-L188)

Additionally, avoid loading the blinded block in all but the `block_id=block_root` case. I'm not sure why we were doing this previously, I suspect it was just an oversight during the implementation of the `finalized` status on API requests.

## Additional Info

I think this change should have minimal negative impact as:

- The block root endpoint is quick to compute (a few ms max).
- Only the priority of `head` requests is increased. Analytical processes that are making lots of block root requests for past slots are unable to DoS the beacon processor, as their requests will still be processed after attestations.
2023-10-03 23:59:37 +00:00
realbigsean
7605494791 Use only lighthouse types in the mock builder (#4793)
## Proposed Changes

- only use LH types to avoid build issues
- use warp instead of axum for the server to avoid importing the dep

## Additional Info

- wondering if we can move the `execution_layer/test_utils` to its own crate and import it as a dev dependency
- this would be made easier by separating out our engine API types into their own crate so we can use them in the test crate
- or maybe we can look into using reth types for the engine api if they are in their own crate


Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-03 17:59:28 +00:00
realbigsean
c7ddf1f0b1
add processing and processed caching to the DA checker (#4732)
* add processing and processed caching to the DA checker

* move processing cache out of critical cache

* get it compiling

* fix lints

* add docs to `AvailabilityView`

* some self review

* fix lints

* fix beacon chain tests

* cargo fmt

* make availability view easier to implement, start on testing

* move child component cache and finish test

* cargo fix

* cargo fix

* cargo fix

* fmt and lint

* make blob commitments not optional, rename some caches, add missing blobs struct

* Update beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* marks review feedback and other general cleanup

* cargo fix

* improve availability view docs

* some renames

* some renames and docs

* fix should delay lookup logic

* get rid of some wrapper methods

* fix up single lookup changes

* add a couple docs

* add single blob merge method and improve process_... docs

* update some names

* lints

* fix merge

* remove blob indices from lookup creation log

* remove blob indices from lookup creation log

* delayed lookup logging improvement

* check fork choice before doing any blob processing

* remove unused dep

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/network/src/sync/block_lookups/delayed_lookup.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* remove duplicate deps

* use gen range in random blobs geneartor

* rename processing cache fields

* require block root in rpc block construction and check block root consistency

* send peers as vec in single message

* spawn delayed lookup service from network beacon processor

* fix tests

---------

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-10-03 09:59:33 -04:00
Jimmy Chen
8f07a96b88
Fix failing tests. 2023-09-26 12:39:58 +10:00
Jimmy Chen
7a3cb135d4
Fix tests and add BlockContents decoding. Remove unused builder_threshold field in ApiTesterConfig. 2023-09-26 10:57:21 +10:00
Jimmy Chen
c0b6b92f27
Merge unstable 20230925 into deneb-free-blobs. 2023-09-26 10:32:18 +10:00
Michael Sproul
9244f7f7bc
Improvements to Deneb store upon review (#4693)
* Start testing blob pruning

* Get rid of unnecessary orphaned blob column

* Make random blob tests deterministic

* Test for pruning being blocked by finality

* Fix bugs and test fork boundary

* A few more tweaks to pruning conditions

* Tweak oldest_blob_slot semantics

* Test margin pruning

* Clean up some terminology and lints

* Schema migrations for v18

* Remove FIXME

* Prune blobs on finalization not every slot

* Fix more bugs + tests

* Address review comments
2023-09-25 14:21:54 -04:00
João Oliveira
dcd69dfc62 Move dependencies to workspace (#4650)
## Issue Addressed

Synchronize dependencies and edition on the workspace `Cargo.toml`

## Proposed Changes

with https://github.com/rust-lang/cargo/issues/8415 merged it's now possible to synchronize details on the workspace `Cargo.toml` like the metadata and dependencies.
By only having dependencies that are shared between multiple crates aligned on the workspace `Cargo.toml` it's easier to not miss duplicate versions of the same dependency and therefore ease on the compile times.

## Additional Info
this PR also removes the no longer required direct dependency of the `serde_derive` crate.

should be reviewed after https://github.com/sigp/lighthouse/pull/4639 get's merged.
closes https://github.com/sigp/lighthouse/issues/4651


Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-09-22 04:30:56 +00:00
Eitan Seri-Levi
992b476eac Add SSZ support to validator block production endpoints (#4534)
## Issue Addressed

#4531 

## Proposed Changes

add SSZ support to the following block production endpoints:

GET /eth/v2/validator/blocks/{slot}
GET /eth/v1/validator/blinded_blocks/{slot}

## Additional Info

i updated a few existing tests to use ssz instead of writing completely new tests
2023-09-21 06:38:31 +00:00