Commit Graph

5967 Commits

Author SHA1 Message Date
Gua00va
44aaf13ff0
Standard Liveness Endpoint (#4853)
* Changes to use required Endpoint

* Format

* fixed doppleganger service

* minor fix

* efficiency changes

* fixed tests

* remove commented line

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-30 17:41:22 +11:00
Michael Sproul
547ed1de63
Clone state ahead of block production (#4925)
* Clone state ahead of block production

* Add pruning and fix logging

* Don't hold 2 states in mem
2023-11-30 13:49:35 +11:00
ethDreamer
43d98153d6
Refactor & Fix Bugs in Payload Selection Logic (#4950)
* Refactor & Fix Bugs in Payload Selection Logic

* Fix lint

* Update beacon_node/execution_layer/src/lib.rs

* Finish renaming function
2023-11-29 15:20:12 +11:00
Paul Hauner
86163d94f2
Add lcli mock-el (#4587)
* Track time to early attester cache

* Add debug log for early attester cache

* Add mock-el command to lcli

* Revert beacon chain changes

* Tidy, fix compilation errors

* Add required flag

* Default to SYNCING response

* Hide flag
2023-11-29 10:04:29 +11:00
GeemoCandama
8a599ec7dc
API for LightClientBootstrap, LightClientFinalityUpdate, LightClientOptimisticUpdate and light client events (#3954)
* rebase and add comment

* conditional test

* test

* optimistic chould be working now

* finality should be working now

* try again

* try again

* clippy fix

* add lc bootstrap beacon api

* add lc optimistic/finality update to events

* fmt

* That error isn't occuring on my computer but I think this should fix it

* Add missing test file

* Update light client types to comply with Altair light client spec.

* Fix test compilation

* Support deserializing light client structures for the Bellatrix fork

* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.

* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses

* Add light client bootstrap API test and fix existing ones.

* Fix test for `light-client-server` http api config.

* Appease clippy

* Efficiency improvement when retrieving beacon state.

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-28 17:14:29 +11:00
Michael Sproul
44c1817c2b
Remove block-delay-ms (#4956) 2023-11-28 16:00:28 +11:00
Jimmy Chen
c88cb371a0
Remove MAX_BLOBS_PER_BLOCK from Holesky config. (#4955) 2023-11-27 23:07:40 +11:00
Alexander Uvizhev
b4556a3d62
Broadcast various requests as per #4684 (#4920)
* multiple broadcast flags

* rewrite with single --broadcast option

* satisfy cargo fmt

* shorten sync-committee-messages

* fix a doc comment and a test

* use strum

* Add broadcast test to simulator

* bring --disable-run-on-all flag back with deprecation notice
2023-11-27 15:39:37 +11:00
realbigsean
e856a904ef
Merge pull request #4939 from eserilev/block-v3-general-tests
Add more block v3 tests
2023-11-21 14:52:52 -05:00
Eitan Seri-Levi
98f159cc18 fmt 2023-11-20 21:37:58 -08:00
Eitan Seri-Levi
24a0a7ffd0 remove cache check 2023-11-20 21:37:04 -08:00
Eitan Seri-Levi
228180bb35 add some more block v3 tests 2023-11-20 20:46:13 -08:00
Jimmy Chen
6b63d18420
Fix Rust beta compiler warnings (rustc 1.75.0-beta.1 (782883f60 2023-11-12)) (#4932) 2023-11-18 03:55:11 +11:00
Mac L
68e076d60a
Remove legacy database migrations (#4919)
* Remove legacy db migrations

* Fix tests

* Remove migrations to/from v15

* Remove v16 migrations
2023-11-16 22:09:05 +11:00
Michael Sproul
d04e361129
Fix missed block logs (#4922) 2023-11-16 22:07:58 +11:00
Zackary Scott
e181741d38
Fix for issue 4860 - Added in process_justification_and_finalization (#4877)
* Added in process_justification_and_finalization

Added in process_justification_and_finalization to compute_attestation_rewards_altair to take into account justified attestations when coming out of inactivity leak. Also added in test to check for this edge case.

* Added in justification and finalization for compute_attestation_rewards_base

* Added in test for altair rewards without inactivity leak
2023-11-16 22:07:48 +11:00
Michael Sproul
051c3e842f
Always use a separate database for blobs (#4892)
* Always use a separate blobs DB

* Add + update tests
2023-11-09 16:51:36 +11:00
chonghe
1b8c0ed987
Update local testnet doc and parameters (#4749)
* update

* revise link

* update parameters

* update doc

* Update doc

* add quic port

* remove debug log

* Fix el_bootnode not being killed

* Fix time

* Fix doc in manual creation of testnet

* Update file

* update api doc

* Revert "update api doc"

This reverts commit ed695743deca4818ad074e5285e04764965b1be5.

* add git clone

* Fix path

* Fix path

* Update scripts/local_testnet/setup_time.sh

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update scripts/local_testnet/README.md

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Fix SLOT_PER_EPOCH that changes with mainnet or minimal

* Embedded setup_time.sh in start_local_testnet.sh

* fix slot per epoch constant

* Add comment

* Add CANCUN_TIME

* Fix CANCUN_TIME constant 32 slots

* Correct typo

* chmod +x ./setup_time.sh

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-09 15:06:10 +11:00
chonghe
7fd9389a8c
Delete deprecated cli flags (#4906)
* Delete BN spec flag and VC beacon-node flag

* Remove warn

* slog

* add warn

* delete eth1-endpoint

* delete server from vc cli.rs

* delete server flag in config.rs

* delete delete-lockfiles in vc

* delete allow-unsynced flag in VC

* delete strict-fee-recipient in VC and warn log

* delete merge flag in bn (hidden)

* delete count-unrealized and count-unrealized-full in bn (hidden)

* delete http-disable-legacy-spec in bn (hidden)

* delete eth1-endpoint in lcli

* delete warn message lcli

* delete eth1-endpoints

* delete minify in slashing protection

* delete minify related

* Remove mut

* add back warn! log

* Indentation

* Delete count-unrealized

* Delete eth1-endpoints

* Delete eth1-endpoint test

* delete eth1-endpints test

* delete allow-unsynced test

* Add back lcli eth1-endpoint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-11-09 15:05:55 +11:00
ethDreamer
7818100777
Verify KZG in Bulk During Block Sync (#4903) 2023-11-09 15:05:44 +11:00
Eitan Seri-Levi
a380f6ef1f
Add block-v3 SSZ tests (#4902)
* create block-v3 ssz tests

* fmt
2023-11-09 15:05:35 +11:00
Joel Rousseau
ac8811afac
Add missed blocks to monitored validators (#4731)
* add missed_block metric

* init missed_block in constructor

* declare beaconproposercache in ValidatorMonitor

* refacto proposer_shuffling_decision_root to use epoch instead of current.epoch

* imple new proposer_shuffling_decision_root in callers

* push missed_blocks

* prune missed_blocks

* only add to hashmap if it's a monitored validator

* remove current_epoch dup + typos

* extract in func

* add prom metrics

* checkpoint is not only epoch but slot as well

* add safeguard if we start a new chain at slot 0

* clean

* remove unnecessary negative value for a slot

* typo in comment

* remove unused current_epoch

* share beacon_proposer_cache between validator_monitor and beacon_chain

* pass Hash256::zero()

* debug objects

* fix loop: lag is at the head

* sed s/get_slot/get_epoch

* fewer calls to cache.get_epoch

* fix typos

* remove cache first call

* export TYPICAL_SLOTS_PER_EPOCH and use it in validator_monitor

* switch to gauge & loop over missed_blocks hashset

* fix subnet_service tests

* remove unused var

* clean + fix nits

* add beacon_proposer_cache + validator_monitor in builder

* fix store_tests

* fix builder tests

* add tests

* add validator monitor set of tests

* clean tests

* nits

* optimise imports

* lint

* typo

* added self.aggregatable

* duplicate proposer_shuffling_decision_root

* remove duplication in passing beacon_proposer_cache

* remove duplication in passing beacon_proposer_cache

* using indices

* fmt

* implement missed blocks total

* nits

* avoid heap allocation

* remove recursion limit

* fix lint

* Fix valdiator monitor builder pattern

Unify validator monitor config struct

* renaming metrics

* renaming metrics in validator monitor

* add log if there's a missing validator index

* consistent log

* fix loop

* better loop

* move gauge to counter

* fmt

* add error message

* lint

* fix prom metrics

* set gauge to 0 when non-finalized epochs

* better wording

* remove hash256::zero in favour of block_root

* fix gauge total label

* fix last missed block validator

* Add `MissedBlock` struct

* Fix comment

* Refactor non-finalized block loop

* Fix off-by-one

* Avoid string allocation

* Fix compile error

* Remove non-finalized blocks metric

* fix func clojure

* remove unused variable

* remove unused DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD

* remove unused DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD in builder

* add validator index depending on the fork name

* typos

---------

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-11-09 15:05:14 +11:00
realbigsean
bcca88a150 more logging improvements (#4885)
## Issue Addressed

-downgrades `Missing components over rpc` to debug because this isn't unusual and just results in a re-try
-removes the result from  `Block component processed for lookup` because this prints the full block on an unknown parent error

Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-11-03 13:31:27 +00:00
Justin Traglia
f2aabe915b Fix comment for blob sidecar observation pruning (#4893)
## Issue Addressed

The comment implies that observations for the given slot would be retained but they are not.

## Proposed Changes

I'm pretty sure the functionality is correct and the comment is slightly incorrect, so just update the comment. The comment needs to say something along the lines of "less than or equal to" rather than just "less than."

## Additional Info

It doesn't make sense to keep finalized observations since those are no longer accepted.
2023-11-03 12:38:13 +00:00
Jimmy Chen
36d8849813 Add commmand for pruning states (#4835)
## Issue Addressed

Closes #4481. 

(Continuation of #4648)

## Proposed Changes

- [x] Add `lighthouse db prune-states`
- [x] Make it work
- [x] Ensure block roots are handled correctly (to be addressed in 4735)
- [x] Check perf on mainnet/Goerli/Gnosis (takes a few seconds max)
- [x] Run block root healing logic (#4875 ) at the beginning
- [x] Add some tests
- [x] Update docs
- [x] Add `--freezer` flag and other improvements to `lighthouse db inspect`

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Jimmy Chen <jimmy@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-11-03 00:12:19 +00:00
Eitan Seri-Levi
07f53b18fc Block v3 endpoint (#4629)
## Issue Addressed

#4582

## Proposed Changes

Add a new v3 block fetching flow that can decide to return a Full OR Blinded payload

## Additional Info



Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-11-03 00:12:18 +00:00
realbigsean
42da392edc fix deneb sync bug (#4869)
## Issue Addressed

I observed our forward sync on devnet 9 would stall when we would hit this log:
```
250425:Oct 19 00:54:17.133 WARN Blocks and blobs request for range received invalid data, error: KzgCommitmentMismatch, batch_id: 4338, peer_id: 16Uiu2HAmHbmkEQFDrJfNuy1aYyAfHkNUwSD9FN7EVAqGJ8YTF9Mh, service: sync, module: network::sync::manager:1036
```

## Proposed Changes

`range_sync_block_and_blob_response` [here](1cb02a13a5/beacon_node/network/src/sync/manager.rs (L1013)) removes the request from the sync manager. later, however if there's an error, `inject_error` [here](1cb02a13a5/beacon_node/network/src/sync/manager.rs (L1055)) expects the request to exist so we can handle retry logic. So this PR just re-inserts the request (withthout any accumulated blobs or blocks) when we hit an error here.

The issue is unique to block+blob sync because the error here is only possible from mismatches between blocks + blobs after we've downloaded both, there's no equivalent error in block sync



Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-31 21:04:18 +00:00
Eitan Seri-Levi
4ce01ddd11 Activate clippy::manual_let_else lint (#4889)
## Issue Addressed

#4888

## Proposed Changes

Enabled `clippy::manual_let_else` lint and resolved the warning messages.
2023-10-31 10:31:02 +00:00
realbigsean
a9f9dc241d restore cargo vendor in test suite (#4886)
## Issue Addressed

resolves https://github.com/sigp/lighthouse/issues/4440

## Proposed Changes

restore our `cargo vendor` test in CI

changes to `c-kzg` here mean we no longer have to compile it twice and get duplicate source errors: https://github.com/sigp/lighthouse/pull/4862

Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-27 10:21:47 +00:00
Michael Sproul
c574f8136e Fix block backfill with genesis skip slots (#4820)
## Issue Addressed

Closes #4817.

## Proposed Changes

- Fill in the linear block roots array between 0 and the slot of the first block (e.g. slots 0 and 1 on Holesky).
- Backport the `--freezer`, `--skip` and `--limit` options for `lighthouse db inspect` from tree-states. This allows us to easily view the database corruption of 4817 using `lighthouse db inspect --network holesky --freezer --column bbr --output values --limit 2`.
- Backport the `iter_column_from` change and `MemoryStore` overhaul from tree-states. These are required to enable `lighthouse db inspect`.
- Rework `freezer_upper_limit` to allow state lookups for slots below the `state_lower_limit`. Currently state lookups will fail until state reconstruction completes entirely.

There is a new regression test for the main bug, but no test for the `freezer_upper_limit` fix because we don't currently support running state reconstruction partially (see #3026). This will be fixed once we merge `tree-states`! In lieu of an automated test, I've tested manually on a Holesky node while it was reconstructing.

## Additional Info

Users who backfilled Holesky to slot 0 (e.g. using `--reconstruct-historic-states`) need to either:

- Re-sync from genesis.
- Re-sync using checkpoint sync and the changes from this PR.

Due to the recency of the Holesky genesis, writing a custom pass to fix up broken databases (which would require its own thorough testing) was deemed unnecessary. This is the primary reason for this PR being marked `backwards-incompat`.

This will create few conflicts with Deneb, which I've already resolved on `tree-states-deneb` and will be happy to backport to Deneb once this PR is merged to unstable.
2023-10-27 05:08:49 +00:00
zhiqiangxu
b82d1a993c fix docs about --builder (#4754)
Align the document with the [implementation](2841f60686/beacon_node/execution_layer/src/lib.rs (L740)) regarding the `--builder` flag.


The `tokio::join!` macro takes a list of async expressions and evaluates them concurrently on the same task.
2023-10-26 05:23:50 +00:00
realbigsean
ba891e1fed deneb related logging improvements (#4859)
1. Add commitments to logs and update the `Display` implementation of `KzgCommitment` to become truncated similarly to block root.

I've been finding it difficult to debug scenarios involving multiple blobs for the same `(index, block_root)`. Logging the commitment will help with this, we can match it to what exists in the block. 

Example output:

```
Oct 20 21:13:36.700 DEBG Successfully verified gossip blob       commitment: 0xa3c1…1cd8, index: 0, root: 0xf31e…f9de, slot: 154568
Oct 20 21:13:36.785 DEBG Successfully verified gossip block      commitments: [0xa3c1…1cd8, 0x8655…02ff, 0x8d6a…955a, 0x84ac…3a1b, 0x9752…629b, 0xb9fc…20fb], root: 0xf31eeb732702e429e89057b15e1c0c631e8452e09e03cb1924353f536ef4f9de, slot: 154568, graffiti: teku/besu, service: beacon
```

Example output in a block with no blobs (this will show up pre-deneb):
```
426734:Oct 20 21:15:24.113 DEBG Successfully verified gossip block, commitments: [], root: 0x619db1360ba0e8d44ae2a0f2450ebca47e167191feecffcfac0e8d7b6c39623c, slot: 154577, graffiti: teku/nethermind, service: beacon, module: beacon_chain::beacon_chain:2765
```

2. Remove `strum::IntoStaticStr` from `AvailabilityCheckError`. This is because `IntoStaticStr` end up dropping information inside the enum. So kzg commitments in this error are dropped, making it more difficult to debug
```
AvailabilityCheckError::KzgCommitmentMismatch {
        blob_commitment: KzgCommitment,
        block_commitment: KzgCommitment,
    },
```
which is output as just `AvailabilityCheckError`

3. Some additional misc sync logs I found useful in debugging https://github.com/sigp/lighthouse/pull/4869

4. This downgrades ”Block returned for single block lookup not present” to debug because I don’t think we can fix the scenario that causes this unless we can cancel inflight rpc requests

Co-authored-by: realbigsean <seananderson33@gmail.com>
2023-10-25 15:30:17 +00:00
Jimmy Chen
d4f26ee123 Add block roots heal logic in v18 schema migration. (#4875)
## Issue Addressed

Fixes #4697. 

This also unblocks the state pruning PR (#4835). Because self healing breaks if state pruning is applied to a database with missing block roots.

## Proposed Changes

- Fill in the missing block roots between last restore point slot and split slot when upgrading to latest database version.
2023-10-25 03:42:24 +00:00
antondlr
a228e61773 don't make lcli on self-hosted runners (#4874)
## Issue Addressed

Our self-hosted runners now have a modern (Deneb-ready) version of `lcli` preinstalled so we no longer need to compile it.
2023-10-25 03:42:23 +00:00
Pawan Dhananjay
6315a81260 Upgrade to v1.4.0-beta.3 (#4862)
## Issue Addressed

Makes lighthouse compliant with new kzg changes in https://github.com/ethereum/consensus-specs/releases/tag/v1.4.0-beta.3

## Proposed Changes

1. Adds new official trusted setup
2. Refactors kzg to match upstream changes in https://github.com/ethereum/c-kzg-4844/pull/377
3. Updates pre-generated `BlobBundle` to work with official trusted setup. ~~Using json here instead of ssz to account for different value of `MaxBlobCommitmentsPerBlock` in minimal and mainnet. By using json, we can just use one pre generated bundle for both minimal and mainnet. Size of 2 separate ssz bundles is approximately equal to one json bundle cc @jimmygchen~~ 
Dunno what I was doing, ssz works without any issues  
4. Stores trusted_setup as just bytes in eth2_network_config so that we don't have kzg dependency in that lib and in lcli. 


Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-10-21 13:49:27 +00:00
Pawan Dhananjay
074c4951fc Reduce calls to network channel (#4863)
## Issue Addressed

N/A

## Proposed Changes

Sends blocks and blobs from http_api to the network channel for publishing in a single network channel send. This is to avoid overhead of multiple calls.
Also adds a metric for rpc blob retrieval duration.
2023-10-20 19:42:47 +00:00
Jimmy Chen
e8fba8d3a7 Enable BLS portable feature on all CI tests (#4868)
## Issue Addressed

Addresses the recent CI failures caused by caching `blst` for the wrong CPU type. 

## Proposed Changes

- Use `FEATURES: jemalloc,portable` when building Lighthouse & `lcli` in tests
- Add a new `TEST_FEATURES` and set to `portable` for all CI test jobs.
- Updated Makefiles to read the `TEST_FEATURES` environment variable, and default to none.
2023-10-20 07:30:27 +00:00
Jimmy Chen
8880675eda Add make lint to development environment section in Book (#4866)
## Issue Addressed

We run `clippy` as part of our CI, so it would help new devs if we add the `make lint` command to the dev setup section in the Lighthouse book.
2023-10-20 06:23:29 +00:00
Zackary Scott
b11988223f #4512 inactivity calculation for Altair (#4807)
## Issue Addressed
#4512 
Which issue # does this PR address?

## Proposed Changes
Add inactivity calculation for Altair

Please list or describe the changes introduced by this PR.
Add inactivity calculation for Altair

## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.


Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-10-20 06:23:28 +00:00
GoodDaisy
90f78d141f fix typos (#4838) 2023-10-19 22:05:15 +00:00
João Oliveira
c6583bb5fa update libp2p (#4864)
## Issue Addressed

updates libp2p to the latest version and uses the new `SwarmBuilder`. Superseeds https://github.com/sigp/lighthouse/pull/4695/
CC @mxinden I don't think we can use both `bandwidth_loggers` with the new syntax right?
2023-10-19 21:22:55 +00:00
Michael Sproul
8c28d175b8 Fix: write post state in lcli skip-slots (#4843)
## Issue Addressed

Fix a bug in `lcli skip-slots` that resulted in it always writing the pre-state to the output file.

## Proposed Changes

Correctly keep track of the post-state, and write it.
2023-10-19 05:19:25 +00:00
Dustin Brickwood
1c6356f8f3 chore: replace deprecated hub with gh for releases (#4839)
## Issue Addressed

- The tool hub that is used to create draft releases has been removed as of October 2. See: https://github.com/actions/runner-images/issues/8362
- This change replaces `hub` usage in favour of [`gh`](https://cli.github.com/manual/gh_release_create) 

## Proposed Changes

- This change replaces `hub` usage in favour of [`gh`](https://cli.github.com/manual/gh_release_create) 

## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.


Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-10-19 05:19:24 +00:00
Joe Clapis
1de02f731b Add CARGO_USE_GIT_CLI to the Dockerfile to work around an OOM bug during cross-compiling (#4828)
## Issue Addressed
#4827 

## Proposed Changes

This PR introduces a new build-arg to the Lighthouse Dockerfile: `CARGO_USE_GIT_CLI`. This arg will be passed into the `CARGO_NET_GIT_FETCH_WITH_CLI` [environment variable](https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli), which instructs `cargo` to use the git CLI during `fetch` operations instead of the git library. Doing so works around [a bug](https://github.com/rust-lang/cargo/issues/10583) with the git library that causes it to go OOM during `fetch` operations on `arm64` platforms.

The default value is `false` so this doesn't affect Lighthouse builds or the CI pipeline. Running a build with `--build-arg CARGO_USE_GIT_CLI=true` will activate it, which is necessary to cross-compile the `arm64` binary when not using `cross` (i.e., when building via the Dockerfile instead of natively if you don't have a rust environment ready to go).

Special thanks to @michaelsproul for helping me repro the initial problem.

Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-10-19 05:19:23 +00:00
João Oliveira
f06391717c collect bandwidth metrics per transport (#4805)
## Issue Addressed

Following the conversation on https://github.com/libp2p/rust-libp2p/pull/3666 the changes introduced in this PR will allow us to give more insights if the bandwidth limitations happen at the transport level, namely if quic helps vs yamux and it's [window size limitation](https://github.com/libp2p/rust-yamux/issues/162) or if the bottleneck is at the gossipsub level.
## Proposed Changes

introduce new quic and tcp bandwidth metric gauges.

cc @mxinden (turned out to be easier, Thomas gave me a hint)
2023-10-19 05:19:22 +00:00
Jimmy Chen
98cac2bc6b Deneb review .github (CI cleanup) (#4696)
## Issue Addressed

Related to https://github.com/sigp/lighthouse/issues/4676.

Deneb-specifc CI code to be removed before merging to `unstable`. Dot not merge until we're ready to merge into `unstable`, as we may need to release deneb docker images before merging.

Keep in mind that most of the changes in the below PR (to `unstable`) have already 
been merged to `deneb-free-blobs`, so merging `deneb-free-blobs` into `unstable` would include those changes - it would be ok if the release runners are ready, otherwise we may want to exclude them before merging.
- https://github.com/sigp/lighthouse/pull/4592
2023-10-18 15:23:31 +00:00
Michael Sproul
5bbeedb5b7 Reduce nextest threads to 8 (#4846)
## Issue Addressed

Fix OOMs caused by too many concurrent tests. The runner machine is currently liable to run `32 * 5 = 160` tests in parallel. If each test uses say 300MB max, this is 48GB of RAM!

## Proposed Changes

Reduce the number of threads per runner job to 8. This should cap the memory at 4x lower than the current limit, i.e. around 12GB. If we continue to run out of RAM, we should consider more sophisticated limits.
2023-10-18 14:19:41 +00:00
Michael Sproul
192d442718 Fix Rayon deadlock in test utils (#4837)
## Issue Addressed

Fix a deadlock in the tests that was causing tests on tree-states to run for hours without finishing: https://github.com/sigp/lighthouse/actions/runs/6491194654/job/17628138360.

## Proposed Changes

Avoid using a Mutex under the Rayon `par_iter`. Instead, use an `AtomicUsize`. I've run the new version several times in a loop and it hasn't deadlocked (it was deadlocking consistently on tree-states).

## Additional Info

The same bug exists in unstable and tree-states, but I'm not sure why it was triggering so consistently on the tree-states branch.
2023-10-18 13:36:42 +00:00
Michael Sproul
463e62e833 Generalise compare_fields to work with iterators (#4823)
## Proposed Changes

Add `compare_fields(as_iter)` as a field attribute to `compare_fields_derive`. This allows any iterable type to be compared in the same as a slice (by index). 

This is forwards-compatible with tree-states types like `List` and `Vector` which can not be cast to slices.
2023-10-18 12:59:53 +00:00
Jimmy Chen
1b4545cd9d Remove blob clones in KZG verification (#4852)
## Issue Addressed

This PR removes two instances of blob clones during blob verification that may not be necessary.
2023-10-18 06:52:54 +00:00