Commit Graph

306 Commits

Author SHA1 Message Date
Akihito Nakano
8cec8a6793
Fix double counted metrics (#5476)
* Fix double counted metrics
2024-03-26 03:10:26 +00:00
Eitan Seri-Levi
e4d4e439cb
Add Capella & Deneb light client support (#4946)
* rebase and add comment

* conditional test

* test

* optimistic chould be working now

* finality should be working now

* try again

* try again

* clippy fix

* add lc bootstrap beacon api

* add lc optimistic/finality update to events

* fmt

* That error isn't occuring on my computer but I think this should fix it

* Merge branch 'unstable' into light_client_beacon_api_1

# Conflicts:
#	beacon_node/beacon_chain/src/events.rs
#	beacon_node/http_api/src/lib.rs
#	beacon_node/http_api/src/test_utils.rs
#	beacon_node/http_api/tests/main.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/network/src/beacon_processor/worker/rpc_methods.rs
#	beacon_node/tests/test.rs
#	common/eth2/src/types.rs
#	lighthouse/src/main.rs

* Add missing test file

* Update light client types to comply with Altair light client spec.

* Fix test compilation

* Merge branch 'unstable' into light_client_beacon_api_1

* Support deserializing light client structures for the Bellatrix fork

* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.

* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses

* Add light client bootstrap API test and fix existing ones.

* Merge branch 'unstable' into light_client_beacon_api_1

* Fix test for `light-client-server` http api config.

* Appease clippy

* Add Altair light client SSZ tests

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into light_client_beacon_api_1

* updates to light client header

* light client header from signed beacon block

* using options

* implement helper functions

* placeholder conversion from vec hash256 to exec branch

* add deneb

* using fixed vector

* remove unwraps

* by epoch

* compute merkle proof

* merkle proof

* update comments

* resolve merge conflicts

* linting

* Merge branch 'unstable' into light-client-ssz-tests

# Conflicts:
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	consensus/types/src/light_client_bootstrap.rs
#	consensus/types/src/light_client_header.rs

* superstruct attempt

* superstruct changes

* lint

* altair

* update

* update

* changes to light_client_optimistic_ and finality

* merge unstable

* refactor

* resolved merge conflicts

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* block_to_light_client_header fork aware

* fmt

* comment fix

* comment fix

* include merge fork, update deserialize_by_fork, refactor

* fmt

* pass by ref to prevent clone

* rename merkle proof fn

* add FIXME

* LightClientHeader TestRandom

* fix comments

* fork version deserialize

* merge unstable

* move fn arguments, fork name calc

* use task executor

* remove unneeded fns

* remove dead code

* add manual ssz decoding/encoding and add ssz_tests_by_fork macro

* merge deneb types with tests

* merge ssz tests, revert code deletion, cleanup

* move chainspec

* update ssz tests

* fmt

* light client ssz tests

* change to superstruct

* changes from feedback

* linting

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* test fix

* cleanup

* Remove unused `derive`.

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* beta compiler fix

* merge
2024-03-25 11:01:56 +00:00
João Oliveira
6edf031490
disable libp2p upnp (#5449)
* disable libp2p upnp

when passing --disable-upnp cli flag
2024-03-21 01:13:03 +00:00
Akihito Nakano
7117772fb3
Fix peer count metrics (#5404)
* Set the peers_per_client metrics directly, rather than using increment/decrement

* Move PEERS_CONNECTED related update to the same place

* Move PEERS_CONNECTED_MULTI related update to the same place

* Rename

* Remove unused variables
2024-03-20 06:45:16 +00:00
Eitan Seri-Levi
01ec42e75a
Fix Rust beta compiler errors 1.78.0-beta.1 (#5439)
* remove redundant imports

* fix test

* contains key

* fmt

* Merge branch 'unstable' into fix-beta-compiler
2024-03-20 05:17:02 +00:00
Pawan Dhananjay
54b1c229e1
Downgrade rate limited log (#5381)
* Address review comments

* Downgrade log for by_root requests
2024-03-10 23:05:24 +00:00
Jonas Bostoen
641f6be3f0
Explicit peers (#5333)
* Merge branch 'unstable' into feature/explicit-peers

* Merge latest unstable

* refactor: remove explicit-peers flag, mark trusted peers as explicit instead

* feat(beacon_node): add explicit peers to GossipSub, mark as trusted

* feat(beacon_node): add explicit peers cli + config
2024-03-07 22:22:39 +00:00
Age Manning
de91c77cb2
Improve peer performance for NAT'd nodes (#5345)
* Merge latest unstable

* Reduce diff

* Reduce logic, discover up to max peers

* Peer discovery for Natd peers
2024-03-07 12:32:30 +00:00
Age Manning
85c3204d70
Correct the metrics for topic subscriptions (#5344)
* Handle fork boundaries

* Merge latest unstable

* Topic subscription fix
2024-03-07 12:32:21 +00:00
Age Manning
fc8f1a4ca7
Attempt to publish to at least mesh_n peers (#5357)
* Code improvements

* Fix gossipsub tests

* Merge latest unstable

* Differentiate errors and better scoring

* Attempt to publish to mesh_n peers
2024-03-07 09:48:51 +00:00
Age Manning
f919f82e4f
Improve logging around peer scoring (#5325)
* Improve logging around score states

* Improve logs also
2024-03-04 19:39:23 +00:00
Age Manning
64e563f5e9
Recognize the Caplin consensus client (#5304)
* Caplin has joined the party

* Fix typo
2024-02-28 05:38:24 +00:00
João Oliveira
a89ff100af
improve libp2p connected peer metrics (#5314)
* patch rust-yamux dep

* improve libp2p connected peer metrics
2024-02-28 03:52:55 +00:00
João Oliveira
65c4ff0775
remove exit-future (#5183)
* remove exit-future usage,

as it is non maintained, and replace with async-channel which is already in the repo.

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future
2024-02-27 22:12:44 +00:00
João Oliveira
abd99652b4
remove nat module and use libp2p upnp (#4840)
* remove nat module and use libp2p upnp

* update Cargo.lock

* remove no longer used dependencies

* restore nat module refactored

* log successful mapping

* only activate upnp if config enabled

reduce logs to debug!

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* address review

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* address review
2024-02-27 07:29:18 +00:00
Age Manning
d36241b4a1
Track multiaddr in connection status (#5308)
* Record the multiaddr for connected peers
2024-02-27 06:32:48 +00:00
Jimmy Chen
50c423ad88
Revert libp2p metrics (#4870) (#5265)
* Revert "improve libp2p connected peer metrics (#4870)"

This reverts commit 0c3fef59b3.
2024-02-20 04:19:17 +00:00
João Oliveira
a229b52723
Deactivate RPC Connection Handler after goodbye message is sent (#5250)
* Deactivate RPC Connection Handler

after goodbye message is sent

* nit: use to_string instead of format

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rpc-shutdown-improvement

* clippy

* Fix cargo.lock

* Merge latest unstable
2024-02-19 07:16:01 +00:00
Michael Sproul
7c23625193
Quieten gossip republish logs (#5235)
* Quieten gossip republish logs
2024-02-15 04:18:23 +00:00
João Oliveira
256d9042d3
Drop gossipsub stale messages when polling ConnectionHandler. (#5175)
* drop gossipsub stale messages

* convert async-channel::Receiver to Peekable,

to be able to Peek next message without dropping it
2024-02-15 00:41:52 +00:00
João Oliveira
0c3fef59b3
improve libp2p connected peer metrics (#4870)
* improve libp2p connected peer metrics

* separate discv5 port from libp2p for NAT open

* use metric family for DISCOVERY_BYTES

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into improve-metrics
2024-02-08 02:40:54 +00:00
Age Manning
4db84de563
Improve network parameters (#5177)
* Modify network parameters for current mainnet conditions
2024-02-08 02:40:47 +00:00
João Oliveira
675a231b45
increment peer per topic count on graft messages (#5212)
* increment peer per topic count on graft messages
2024-02-08 02:00:49 +00:00
Age Manning
853042746b
Downgrade gossipsub duplicate logs (#5163)
* Downgrade duplicate publish logs

* Maintain backwards compatiblity, deprecate flag

* The tests had to go, because there's no config to test against

* Update help_bn.md
2024-02-06 07:24:01 +00:00
Jimmy Chen
39e9f7dc6b
Fix Rust beta compiler errors (1.77) (#5180)
* Lint fixes

* More fixes for beta compiler.

* Format fixes

* Move `#[allow(dead_code)]` to field level.

* Remove old comment.

* Update beacon_node/execution_layer/src/test_utils/mod.rs

Co-authored-by: João Oliveira <hello@jxs.pt>

* remove duplicate line
2024-02-05 17:54:11 +00:00
Age Manning
f02189c86a
Prevent QUIC logs when quic is disabled (#5071)
* Prevent logs and dialing quic multiaddrs when not supported

* Merge latest unstable
2024-01-31 18:11:49 +00:00
Age Manning
4273004bd9
Add gossipsub as a Lighthouse behaviour (#5066)
* Move gossipsub as a lighthouse behaviour

* Update dependencies, pin to corrected libp2p version

* Merge latest unstable

* Fix test

* Remove unused dep

* Fix cargo.lock

* Re-order behaviour, pin upstream libp2p

* Pin discv5 to latest version
2024-01-31 17:32:31 +00:00
Lion - dapplion
b035638f9b
Compute recent lightclient updates (#4969)
* Compute recent lightclient updates

* Review PR

* Merge remote-tracking branch 'upstream/unstable' into lc-prod-recent-updates

* Review PR

* consistent naming

* add metrics

* revert dropping reprocessing queue

* Update light client optimistic update re-processing logic. (#7)

* Add light client server simulator tests. Co-authored by @dapplion.

* Merge branch 'unstable' into fork/dapplion/lc-prod-recent-updates

* Fix lint

* Enable light client server in simulator test.

* Fix test for light client optimistic updates and finality updates.
2024-01-31 05:25:51 +00:00
Age Manning
a36a12a8d2
Correct multiple dial bug (#5113)
* Dialing the same peer-id error fix

* Improve dialing logging

* Update beacon_node/lighthouse_network/src/peer_manager/mod.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
2024-01-24 10:10:05 +11:00
Age Manning
5851027bfd
Correct discovery logic (#5111) 2024-01-23 16:58:24 +11:00
Age Manning
b7c9b27872
Gossipsub fanout correction (#5110)
* Correct fanout in gossipsub

* Upgrade discv5 to pin new libp2p version

* Update cargo.lock
2024-01-23 16:38:00 +11:00
João Oliveira
38df87c3c5
Switch libp2p sigp gossipsub fork (#4999)
* switch libp2p source to sigp fork

* Shift the connection closing inside RPC behaviour

* Tag specific commits

* Add slow peer scoring

* Fix test

* Use default yamux config

* Pin discv5 to our libp2p fork and cargo update

* Upgrade libp2p to enable yamux gains

* Add a comment specifying the branch being used

* cleanup build output from within container
(prevents CI warnings related to fs permissions)

* Remove revision tags add branches for testing, will revert back once we're happy

* Update to latest rust-libp2p version

* Pin forks

* Update cargo.lock

* Re-pin to panic-free rust

---------

Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: antondlr <anton@delaruelle.net>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2024-01-10 16:26:52 +11:00
realbigsean
b47e3f252e
Runtime rpc request sizes (#4841)
* add runtime variable list type

* add configs to ChainSpec

* git rid of max request blocks type

* fix tests and lints

* remove todos

* git rid of old const usage

* fix decode impl

* add new config to `Config` api struct

* add docs fix compilt

* move methods for per-fork-spec to chainspec

* get values off chain spec

* fix compile

* remove min by root size

* add tests for runtime var list

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2024-01-09 10:23:47 +11:00
Jimmy Chen
b0c374c1ca
Update dependencies to get rid of the yanked deps (#4994)
* Initial attempt to upgrade hashbrown to latest to get rid of the crate warnings.

* Replace `.expect()` usage with use of `const`.

* Update ahash 0.7 as well

* Remove unsafe code.

* Update `lru` to 0.12 and fix release test errors.

* Set non-blocking socket

* Bump testcontainers to 0.15.

* Fix lint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-12-15 18:31:59 +11:00
Jimmy Chen
366f0d7ac2
Add flag to disable warning logs for duplicate gossip messages (#5009)
* Add flag to disable warning logs for duplicate gossip messages.

* Update Lighthouse book.
2023-12-15 09:26:51 +11:00
Age Manning
69f1b7afec
Disable flood publishing (#4383)
* Disable flood publish

* Change default configuration
2023-12-12 09:45:54 +11:00
Divma
6c0c41c7ac
upgrade libp2p to v0.53.* (#4935)
* update libp2p and address compiler errors

* remove bandwidth logging from transport

* use libp2p registry

* make clippy happy

* use rust 1.73

* correct rpc keep alive

* remove comments and obsolte code

* remove libp2p prefix

* make clippy happy

* use quic under facade

* remove fast msg id

* bubble up close statements

* fix wrong comment
2023-12-07 20:39:59 +11:00
Pawan Dhananjay
31044402ee
Sidecar inclusion proof (#4900)
* Refactor BlobSidecar to new type

* Fix some compile errors

* Gossip verification compiles

* Fix http api types take 1

* Fix another round of compile errors

* Beacon node crate compiles

* EF tests compile

* Remove all blob signing from VC

* fmt

* Tests compile

* Fix some tests

* Fix more http tests

* get compiling

* Fix gossip conditions and tests

* Add basic proof generation and verification

* remove unnecessary ssz decode

* add back build_sidecar

* remove default at fork for blobs

* fix beacon chain tests

* get relase tests compiling

* fix lints

* fix existing spec tests

* add new ef tests

* fix gossip duplicate rule

* lints

* add back sidecar signature check in gossip

* add finalized descendant check to blob sidecar gossip

* fix error conversion

* fix release tests

* sidecar inclusion self review cleanup

* Add proof verification and computation metrics

* Remove accidentally committed file

* Unify some block and blob errors; add slashing conditions for sidecars

* Address review comment

* Clean up re-org tests (#4957)

* Address more review comments

* Add Comments & Eliminate Unnecessary Clones

* update names

* Update beacon_node/beacon_chain/src/metrics.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update beacon_node/network/src/network_beacon_processor/tests.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* pr feedback

* fix test compile

* Sidecar Inclusion proof small refactor and updates (#4967)

* Update some comments, variables and small cosmetic fixes.

* Couple blobs and proofs into a tuple in `PayloadAndBlobs` for simplicity and safety.

* Update function comment.

* Update testing/ef_tests/src/cases/merkle_proof_validity.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Rename the block and blob wrapper types used in the beacon API interfaces.

* make sure gossip invalid blobs are passed to the slasher (#4970)

* Add blob headers to slasher before adding to DA checker

* Replace Vec with HashSet in BlockQueue

* fmt

* Rename gindex -> index

* Simplify gossip condition

---------

Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <sean@sigmaprime.io>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-12-05 11:19:59 -05:00
GeemoCandama
8a599ec7dc
API for LightClientBootstrap, LightClientFinalityUpdate, LightClientOptimisticUpdate and light client events (#3954)
* rebase and add comment

* conditional test

* test

* optimistic chould be working now

* finality should be working now

* try again

* try again

* clippy fix

* add lc bootstrap beacon api

* add lc optimistic/finality update to events

* fmt

* That error isn't occuring on my computer but I think this should fix it

* Add missing test file

* Update light client types to comply with Altair light client spec.

* Fix test compilation

* Support deserializing light client structures for the Bellatrix fork

* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.

* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses

* Add light client bootstrap API test and fix existing ones.

* Fix test for `light-client-server` http api config.

* Appease clippy

* Efficiency improvement when retrieving beacon state.

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-28 17:14:29 +11:00
Jimmy Chen
6b63d18420
Fix Rust beta compiler warnings (rustc 1.75.0-beta.1 (782883f60 2023-11-12)) (#4932) 2023-11-18 03:55:11 +11:00
Eitan Seri-Levi
4ce01ddd11 Activate clippy::manual_let_else lint (#4889)
## Issue Addressed

#4888

## Proposed Changes

Enabled `clippy::manual_let_else` lint and resolved the warning messages.
2023-10-31 10:31:02 +00:00
GoodDaisy
90f78d141f fix typos (#4838) 2023-10-19 22:05:15 +00:00
João Oliveira
c6583bb5fa update libp2p (#4864)
## Issue Addressed

updates libp2p to the latest version and uses the new `SwarmBuilder`. Superseeds https://github.com/sigp/lighthouse/pull/4695/
CC @mxinden I don't think we can use both `bandwidth_loggers` with the new syntax right?
2023-10-19 21:22:55 +00:00
João Oliveira
f06391717c collect bandwidth metrics per transport (#4805)
## Issue Addressed

Following the conversation on https://github.com/libp2p/rust-libp2p/pull/3666 the changes introduced in this PR will allow us to give more insights if the bandwidth limitations happen at the transport level, namely if quic helps vs yamux and it's [window size limitation](https://github.com/libp2p/rust-yamux/issues/162) or if the bottleneck is at the gossipsub level.
## Proposed Changes

introduce new quic and tcp bandwidth metric gauges.

cc @mxinden (turned out to be easier, Thomas gave me a hint)
2023-10-19 05:19:22 +00:00
João Oliveira
f10d3d07c3 remove crit! logging from ListenerClosed event on Ok() (#4821)
## Issue Addressed

Since adding Quic support on https://github.com/sigp/lighthouse/pull/4577, and due to `quinn`s api nature LH now triggers the [`ListenerClosed`](https://docs.rs/libp2p/0.52.3/libp2p/swarm/struct.ListenerClosed.html) event.. @michaelsproul noticed we are logging this event as `crit!` independently of the reason. This PR matches the reason, logging with `debug!` and `error!` (instead of `crit!`) according to its `Result`  
## Additional Info
LH will still log `crit!` until https://github.com/libp2p/rust-libp2p/pull/4621 has been merged
2023-10-18 06:52:52 +00:00
Age Manning
cf544b3996
Very minor own nitpicks (#4845) 2023-10-16 16:30:14 +11:00
realbigsean
4555e33048
Remove serde derive references (#4830)
* remove remaining uses of serde_derive

* fix lockfile

---------

Co-authored-by: João Oliveira <hello@jxs.pt>
2023-10-11 13:01:30 -04:00
Jimmy Chen
4ad7e15732
Address Clippy 1.73 lints on Deneb branch (#4810)
* Address Clippy 1.73 lints (#4809)

## Proposed Changes

Fix Clippy lints enabled by default in Rust 1.73.0, released today.

* Address Clippy 1.73 lints.

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-10-06 12:23:57 +05:30
Jimmy Chen
c5c84f1213
Merge branch 'unstable' into merge-unstable-to-deneb-20231005
# Conflicts:
#	.github/workflows/test-suite.yml
#	Cargo.lock
#	beacon_node/execution_layer/Cargo.toml
#	beacon_node/execution_layer/src/test_utils/mock_builder.rs
#	beacon_node/execution_layer/src/test_utils/mod.rs
#	beacon_node/network/src/service/tests.rs
#	consensus/types/src/builder_bid.rs
2023-10-05 15:54:44 +11:00
Akihito Nakano
ba8bcf4bd3 Remove deficit gossipsub scoring during topic transition (#4486)
## Issue Addressed

This PR closes https://github.com/sigp/lighthouse/issues/3237

## Proposed Changes

Remove topic weight of old topics when the fork happens.

## Additional Info

- Divided `NetworkService::start()` into `NetworkService::build()` and `NetworkService::start()` for ease of testing.
2023-10-04 00:43:28 +00:00