Commit Graph

3679 Commits

Author SHA1 Message Date
realbigsean
2627463366 add a github action for build multi-arch docker images (#1574)
## Issue Addressed

#1512

## Proposed Changes

Use Github Actions to automate the Docker image build, so that we can make a multi-arch image.  

## Additional Info

This change will require adding the DOCKER_USERNAME and DOCKER_PASSWORD secrets in Github. It will also require disabling the Docker Hub automated build.
2020-09-04 02:43:32 +00:00
Antoine Detante
9c9176c1d1 Allow to use the same password when importing multiple keystores (#1479) (#1510)
## Issue Addressed

#1479 

## Proposed Changes

* Add an optional flag `reuse-password` in the `import` command of account_manager, allowing to use the same password for all imported keystores.
2020-09-04 01:49:21 +00:00
Pawan Dhananjay
87181204d0 Minor documentation fixes (#1297)
## Issue Addressed

N/A

## Proposed Changes

- Fix a wrong command in the validator generation example.
- Replace occurrences of 'passphrase' with 'password'. This is mostly because I felt that there was a lot of mixing of the two phrases in the documentation and the actual commands which is a bit confusing. Picked 'password' everywhere because it felt more appropriate but I don't mind changing it to 'passphrase' as long it's consistent everywhere.
2020-09-02 04:59:22 +00:00
Age Manning
fb9d828e5e Extended Gossipsub metrics (#1577)
## Issue Addressed

N/A

## Proposed Changes

Adds extended metrics to get a better idea of what is happening at the gossipsub layer of lighthouse. This provides information about mesh statistics per topics, subscriptions and peer scores. 

## Additional Info
2020-09-01 06:59:14 +00:00
Age Manning
8301a984eb Revert 1502 - Switching docker user to lighthouse (#1578)
## Issue Addressed

The lighthouse user has recently changed to `lighthouse` from root. 

This requires uses to change ownership of their current docker mounted volumes and the upgrade path is non-trivial. 
This reverts #1502 and we will include it in a major release in the future.

## Proposed Changes

N/A

## Additional Info

N/A
2020-09-01 01:32:02 +00:00
Maximilian Ehlers
7d71d98dc1 Creates a new lighthouse user and makes it the default user to be use… (#1502)
…d in the Docker image

## Issue Addressed
https://github.com/sigp/lighthouse/issues/1459

## Proposed Changes

- Create new `lighthouse` user and group in Docker container
- Set user as the default user
2020-08-31 07:52:26 +00:00
realbigsean
c34e8efb12 Increase logging channel capacity (#1570)
## Issue Addressed

#1464

## Proposed Changes

Increase the slog-async log channel size from the default of 128 to 2048 to reduce the number of dropped logs. 

## Additional Info
2020-08-31 02:36:19 +00:00
Pawan Dhananjay
adea7992f8 Eth1 network exit on wrong network id (#1563)
## Issue Addressed

Fixes #1509 

## Proposed Changes

Exit the beacon node if the eth1 endpoint points to an invalid eth1 network. Check the network id before every eth1 cache update and display an error log if the network id has changed to an invalid one.
2020-08-31 02:36:17 +00:00
blacktemplar
c18d37c202 Use Gossipsub 1.1 (#1516)
## Issue Addressed

#1172

## Proposed Changes

* updates the libp2p dependency
* small adaptions based on changes in libp2p
* report not just valid messages but also invalid and distinguish between `IGNORE`d messages and `REJECT`ed messages


Co-authored-by: Age Manning <Age@AgeManning.com>
2020-08-30 13:06:50 +00:00
tobisako
b6340ec495 fix change flag name end_after_checks to continue_after_checks (#1573)
## Issue Addressed

Resolve #1387 

## Proposed Changes

Replace flag name **end_after_checks** to ** continue_after_checks**
Change condition to simple (remove **!**, It's no change logic.)

## Additional Info

Operation check
- [x] subcommand `eth1-sim` with ganach-cli
  - [x] `./simulator eth1-sim` -> test is completes
  - [x] `./simulator eth1-sim --continue_after_checks` -> test is never completes
  - [x] `./simulator eth1-sim -c` -> test is never completes
  - [x] `./simulator eth1-sim -c true` -> error: Found (clap)
  - [x] `./simulator eth1-sim -c false` -> error: Found (clap)
- [x] subcommand `no-eth1-sim`
  - [x] `./simulator no-eth1-sim` -> test is completes
  - [x] `./simulator no-eth1-sim --continue_after_checks` -> test is never completes
  - [x] `./simulator no-eth1-sim -c` -> test is never completes
  - [x] `./simulator no-eth1-sim -c true` -> error: Found (clap)
  - [x] `./simulator no-eth1-sim -c false` -> error: Found (clap)
2020-08-27 23:21:21 +00:00
Paul Hauner
967700c1ff Bump version to v0.2.8 (#1572)
## Issue Addressed

NA

## Proposed Changes

- Bump versions
- Run `cargo update`

## Additional Info

NA
2020-08-27 07:04:12 +00:00
Adam Szkoda
d9f4819fe0 Alternative (to BeaconChainHarness) BeaconChain testing API (#1380)
The PR:

* Adds the ability to generate a crucial test scenario that isn't possible with `BeaconChainHarness` (i.e. two blocks occupying the same slot; previously forks necessitated skipping slots):

![image](https://user-images.githubusercontent.com/165678/88195404-4bce3580-cc40-11ea-8c08-b48d2e1d5959.png)

* New testing API: Instead of repeatedly calling add_block(), you generate a sorted `Vec<Slot>` and leave it up to the framework to generate blocks at those slots.
* Jumping backwards to an earlier epoch is a hard error, so that tests necessarily generate blocks in a epoch-by-epoch manner.
* Configures the test logger so that output is printed on the console in case a test fails.  The logger also plays well with `--nocapture`, contrary to the existing testing framework
* Rewrites existing fork pruning tests to use the new API
* Adds a tests that triggers finalization at a non epoch boundary slot
* Renamed `BeaconChainYoke` to `BeaconChainTestingRig` because the former has been too confusing
* Fixed multiple tests (e.g. `block_production_different_shuffling_long`, `delete_blocks_and_states`, `shuffling_compatible_simple_fork`) that relied on a weird (and accidental) feature of the old `BeaconChainHarness` that attestations aren't produced for epochs earlier than the current one, thus masking potential bugs in test cases.

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-08-26 09:24:55 +00:00
Michael Sproul
30bb7aecfb Check Cargo.lock freshness on CI (#1565)
Check that `Cargo.lock` is up-to-date on CI so we're not having to push messy lockfile fix ups after releases.
2020-08-26 00:01:08 +00:00
Michael Sproul
4763f03dcc Fix bug in database pruning (#1564)
## Issue Addressed

Closes #1488

## Proposed Changes

* Prevent the pruning algorithm from over-eagerly deleting states at skipped slots when they are shared with the canonical chain.
* Add `debug` logging to the pruning algorithm so we have so better chance of debugging future issues from logs.
* Modify the handling of the "finalized state" in the beacon chain, so that it's always the state at the first slot of the finalized epoch (previously it was the state at the finalized block). This gives database pruning a clearer and cleaner view of things, and will marginally impact the pruning of the op pool, observed proposers, etc (in ways that are safe as far as I can tell).
* Remove duplicated `RevertedFinalizedEpoch` check from `after_finalization`
* Delete useless and unused `max_finality_distance`
* Add tests that exercise pruning with shared states at skip slots
* Delete unnecessary `block_strategy` argument from `add_blocks` and friends in the test harness (will likely conflict with #1380 slightly, sorry @adaszko -- but we can fix that)
* Bonus: add a `BeaconChain::with_head` method. I didn't end up needing it, but it turned out quite nice, so I figured we could keep it?

## Additional Info

Any users who have experienced pruning errors on Medalla will need to resync after upgrading to a release including this change. This should end unbounded `chain_db` growth! 🎉
2020-08-26 00:01:06 +00:00
Pawan Dhananjay
175471a64b Fix order of testnet config load (#1558)
## Issue Addressed

Fixes #1552 

## Proposed Changes

Earlier, we were always loading the hardcoded default testnet config which is a mainnet spec. So running lighthouse with `--spec` option anything other than mainnet gave errors because we tried loading a mainnet genesis spec with `minimal`/`interop` flags.

This PR fixes the order of loading such that we load the hardcoded default spec only if neither `--testnet` and `--testnet-dir` flags are present.
2020-08-25 06:01:42 +00:00
Paul Hauner
dfd02d6179 Bump to v0.2.7 (#1561)
## Issue Addressed

NA

## Proposed Changes

- Update to v0.2.7
- Add script to make update easy.

## Additional Info

NA
2020-08-24 08:25:34 +00:00
Paul Hauner
3569506acd Remove rayon from rest_api (#1562)
## Issue Addressed

NA

## Proposed Changes

Addresses a deadlock condition described here: https://hackmd.io/ijQlqOdqSGaWmIo6zMVV-A?view

## Additional Info

NA
2020-08-24 07:28:54 +00:00
Paul Hauner
c895dc8971 Shift HTTP server heavy-lifting to blocking executor (#1518)
## Issue Addressed

NA

## Proposed Changes

Shift practically all HTTP endpoint handlers to the blocking executor (some very light tasks are left on the core executor).

## Additional Info

This PR covers the `rest_api` which will soon be refactored to suit the standard API. As such, I've cut a few corners and left some existing issues open in this patch. What I have done here should leave the API in state that is not necessary *exactly* the same, but good enough for us to run validators with. Specifically, the number of blocking workers that can be spawned is unbounded and I have not implemented a queue; this will need to be fixed when we implement the standard API.
2020-08-24 03:06:10 +00:00
blacktemplar
2bc9115a94 reuse beacon_node methods for initializing network configs in boot_node (#1520)
## Issue Addressed

#1378

## Proposed Changes

Boot node reuses code from beacon_node to initialize network config. This also enables using the network directory to store/load the enr and the private key.

## Additional Info

Note that before this PR the port cli arguments were off (the argument was named `enr-port` but used as `boot-node-enr-port`).
Therefore as port always the cli port argument was used (for both enr and listening). Now the enr-port argument can be used to overwrite the listening port as the public port others should connect to.

Last but not least note, that this restructuring reuses `ethlibp2p::NetworkConfig` that has many more options than the ones used in the boot node. For example the network config has an own `discv5_config` field that gets never used in the boot node and instead another `Discv5Config` gets created later in the boot node process.

Co-authored-by: Age Manning <Age@AgeManning.com>
2020-08-21 12:00:01 +00:00
Nat
3cfd70d7fd Docs: Fix reference to incorrect password file. (#1556)
Leftover "mywallet.pass" -> "wally.pass"

Thanks @pecurliarly (from Discord)!
2020-08-21 03:50:37 +00:00
blacktemplar
3f0a113c7f ban IP addresses if too many banned peers for this IP address (#1543)
## Issue Addressed

#1283 

## Proposed Changes

All peers with the same IP will be considered banned as long as there are more than 5 (constant) peers with this IP that have a score below the ban threshold. As soon as some of those 5 peers get unbanned (through decay) and if there are then less than 5 peers with a score below the threshold the IP will be considered not banned anymore.
2020-08-21 01:41:12 +00:00
Paul Hauner
ebb25b5569 Bump version to v0.2.6 (#1549)
## Issue Addressed

NA

## Proposed Changes

See title.

## Additional Info

NA
2020-08-19 09:31:01 +00:00
Pawan Dhananjay
bbed42f30c Refactor attestation service (#1415)
## Issue Addressed

N/A

## Proposed Changes

Refactor attestation service to send out requests to find peers for subnets as soon as we get attestation duties. 
Earlier, we had much more involved logic to send the discovery requests to the discovery service only 6 slots before the attestation slot. Now that discovery is much smarter with grouped queries, the complexity in attestation service can be reduced considerably.



Co-authored-by: Age Manning <Age@AgeManning.com>
2020-08-19 08:46:25 +00:00
divma
fdc6e2aa8e Shutdown like a Sir (#1545)
## Issue Addressed
#1494 

## Proposed Changes
- Give the TaskExecutor the sender side of a channel that a task can clone to request shutting down
- The receiver side of this channel is in environment and now we block until ctrl+c or an internal shutdown signal is received
- The swarm now informs when it has reached 0 listeners
- The network receives this message and requests the shutdown
2020-08-19 05:51:14 +00:00
Paul Hauner
8e7dd7b2b1 Add remaining network ops to queuing system (#1546)
## Issue Addressed

NA

## Proposed Changes

- Refactors the `BeaconProcessor` to remove some excessive nesting and file bloat
  - Sorry about the noise from this, it's all contained in 4d3f8c5 though.
- Adds exits, proposer slashings, attester slashings to the `BeaconProcessor` so we don't get overwhelmed with large amounts of slashings (which happened a few hours ago).

## Additional Info

NA
2020-08-19 05:09:53 +00:00
Age Manning
33b2a3d0e0 Version bump to v0.2.5 (#1540)
## Description

Version bumps lighthouse to v0.2.5
2020-08-18 11:23:08 +00:00
Paul Hauner
93b7c3b7ff Set default max skips to 700 (#1542)
## Issue Addressed

NA

## Proposed Changes

Sets the default max skips to 700 so that it can cover the 693 slot skip from `80894 - 80201`.

## Additional Info

NA
2020-08-18 09:27:04 +00:00
Age Manning
2d0b214b57 Clean up logs (#1541)
## Description

This PR improves some logging for the end-user. 

It downgrades some warning logs and removes the slots per second sync speed if we are syncing and the speed is 0. This is likely because we are syncing from a finalised checkpoint and the head doesn't change.
2020-08-18 08:11:39 +00:00
Paul Hauner
d4f763bbae Fix mistake with attestation skip slots (#1539)
## Issue Addressed

NA

## Proposed Changes

- Fixes a mistake I made in #1530 which resulted us in *not* rejecting attestations that we intended to reject.
- Adds skip-slot checks for blocks earlier in import process, so it rejects gossip and RPC blocks.

## Additional Info

NA
2020-08-18 06:28:26 +00:00
Age Manning
e1e5002d3c Fingerprint Lodestar (#1536)
Fingerprints the Lodestar client
2020-08-18 06:28:24 +00:00
Paul Hauner
46dd530476 Allow import of Prysm keystores (#1535)
## Issue Addressed

- Resolves #1361

## Proposed Changes

Loosens the constraints imposed by EIP-2335 so we can import keys from Prysm.

## Additional Info

NA
2020-08-18 06:28:20 +00:00
Age Manning
8311074d68 Purge out-dated head chains on chain completion (#1538)
## Description

There can be many head chains queued up to complete. Currently we try and process all of these to completion before we consider the node synced. 

In a chaotic network, there can be many of these and processing them to completion can be very expensive and slow. This PR removes any non-syncing head chains from the queue, and re-status's the peers. If, after we have synced to head on one chain, there is still a valid head chain to download, it will be re-established once the status has been returned. 

This should assist with getting nodes to sync on medalla faster.
2020-08-18 05:22:34 +00:00
Age Manning
3bb30754d9 Keep track of failed head chains and prevent re-lookups (#1534)
## Overview

There are forked chains which get referenced by blocks and attestations on a network. Typically if these chains are very long, we stop looking up the chain and downvote the peer. In extreme circumstances, many peers are on many chains, the chains can be very deep and become time consuming performing lookups. 

This PR adds a cache to known failed chain lookups. This prevents us from starting a parent-lookup (or stopping one half way through) if we have attempted the chain lookup in the past.
2020-08-18 03:54:09 +00:00
Age Manning
cc44a64d15 Limit parallelism of head chain sync (#1527)
## Description

Currently lighthouse load-balances across peers a single finalized chain. The chain is selected via the most peers. Once synced to the latest finalized epoch Lighthouse creates chains amongst its peers and syncs them all in parallel amongst each peer (grouped by their current head block). 

This is typically fast and relatively efficient under normal operations. However if the chain has not finalized in a long time, the head chains can grow quite long. Peer's head chains will update every slot as new blocks are added to the head. Syncing all head chains in parallel is a bottleneck and highly inefficient in block duplication leads to RPC timeouts when attempting to handle all new heads chains at once. 

This PR limits the parallelism of head syncing chains to 2. We now sync at most two head chains at a time. This allows for the possiblity of sync progressing alongside a peer being slow and holding up one chain via RPC timeouts.
2020-08-18 02:49:24 +00:00
divma
46dbf027af Do not reset batch ids & redownload out of range batches (#1528)
The changes are somewhat simple but should solve two issues:
- When quickly changing between chains once and a second time back again, batchIds would collide and cause havoc. 
- If we got an out of range response from a peer, sync would remain in syncing but without advancing

Changes:
- remove the batch id. Identify each batch (inside a chain) by its starting epoch. Target epochs for downloading and processing now advance by EPOCHS_PER_BATCH
- for the same reason, move the "to_be_downloaded_id" to be an epoch
- remove a sneaky line that dropped an out of range batch without downloading it
- bonus: put the chain_id in the log given to the chain. This is why explicitly logging the chain_id is removed
2020-08-18 01:29:51 +00:00
Paul Hauner
9a97a0b14f Prepare for v0.2.4 (#1533)
## Issue Addressed

NA

## Proposed Changes

NA

## Additional Info

NA
2020-08-17 12:13:42 +00:00
Michael Sproul
719a69aee0 Ignore blocks that skip a large distance from their parent (#1530)
## Proposed Changes

To mitigate the impact of minority forks on RAM and disk usage, this change rejects blocks whose parent lies more than 320 slots (10 epochs, ~1 hour) in the past. The behaviour is configurable via `lighthouse bn --max-skip-slots N`, and can be turned off entirely using `--max-skip-slots none`.

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-08-17 10:54:58 +00:00
Paul Hauner
a58aa6ee55 Revert back to discv5 alpha 8 to maintain ARM support (#1531)
## Issue Addressed

NA

## Proposed Changes

See title.

## Additional Info

NA
2020-08-17 10:06:08 +00:00
Paul Hauner
73cbfbdfd0 Ensure RUSTFLAGS is passed through on cross compile (#1529)
## Issue Addressed

NA

## Proposed Changes

Tells `cross` (used for cross-compiling) to read the `RUSTFLAGS`env and pass it through during build. This allows us to use `-g` and get debug info. 

## Additional Info

NA
2020-08-17 10:06:06 +00:00
Paul Hauner
f85485884f Process gossip blocks on the GossipProcessor (#1523)
## Issue Addressed

NA

## Proposed Changes

Moves beacon block processing over to the newly-added `GossipProcessor`. This moves the task off the core executor onto the blocking one.

## Additional Info

- With this PR, gossip blocks are being ignored during sync.
2020-08-17 09:20:27 +00:00
Paul Hauner
61d5b592cb Memory usage reduction (#1522)
## Issue Addressed

NA

## Proposed Changes

- Adds a new function to allow getting a state with a bad state root history for attestation verification. This reduces unnecessary tree hashing during attestation processing, which accounted for 23% of memory allocations (by bytes) in a recent `heaptrack` observation.
- Don't clone caches on intermediate epoch-boundary states during block processing.
- Reject blocks that are known to fork choice earlier during gossip processing, instead of waiting until after state has been loaded (this only happens in edge-case).
- Avoid multiple re-allocations by creating a "forced" exact size iterator.

## Additional Info

NA
2020-08-17 08:05:13 +00:00
Age Manning
3c689a6837 Remove yamux support (#1526)
## Issue Addressed

There is currently an issue with yamux when connecting to prysm peers. The source of the issue is currently unknown. 

This PR removes yamux support to force mplex negotation. We can add back yamux support once we have isolated and corrected the issue.
2020-08-17 05:05:06 +00:00
Age Manning
afdc4fea1d Correct logic for peer sync identification (#1525)
Fix a small sync bug which can mis-classify newly connected peers.
2020-08-17 03:00:10 +00:00
Pawan Dhananjay
850a2d5985 Persist metadata and enr across restarts (#1513)
## Issue Addressed

Resolves #1489 

## Proposed Changes

- Change starting metadata seq num to 0 according to the [spec](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/phase0/p2p-interface.md#metadata).
- Remove metadata field from `NetworkGlobals`
- Persist metadata to disk on every update
- Load metadata seq number from disk on restart
- Persist enr to disk on update to ensure enr sequence number increments are persisted as well.

## Additional info

Since we modified starting metadata seq num to 0 from 1, we might still see `Invalid Sequence number provided` like in #1489  from prysm nodes if they have our metadata cached.
2020-08-17 02:13:28 +00:00
divma
113b40f321 Add multiaddr support in bootnodes (#1481)
## Issue Addressed
#1384 

Only catch, as currently implemented, when dialing the multiaddr nodes, there is no way to ask the peer manager if they are already connected or dialing
2020-08-17 02:13:26 +00:00
Age Manning
99acfb50f2 Update gossipsub duplicate cache (#1524)
This potentially handles memory leak issues by preventing adding references to already seen gossipsub messages.
2020-08-17 01:27:33 +00:00
Age Manning
c75c06cf16 Update discv5 to alpha.9 (#1517)
## Discovery v5 update

In this update we remove the openssl dependency in favour of rust-crypto. 

The update also removes a series of unnecessary async functions which may improve some of the issues we have been experiencing.
2020-08-15 04:02:14 +00:00
Paul Hauner
6aeb896480 Commit Cargo.lock changes, add build scripts (#1521)
## Issue Addressed

NA

## Proposed Changes

This PR commits the `Cargo.lock` file so it does not indicate a dirty git tree in the version tag. This code should be used for the `v0.2.3` release.

Also, adds a `Makefile` command to produce tarballs for upload on release.

## Additional Info

NA
2020-08-14 22:24:27 +00:00
Paul Hauner
f4a7311008 Update to v0.2.3 (#1519)
## Issue Addressed

NA

## Proposed Changes

Bump versions to v0.2.3.

## Additional Info

NA
2020-08-14 08:32:31 +00:00
Paul Hauner
619ad106cf Restrict fork choice getters to finalized blocks (#1475)
## Issue Addressed

- Resolves #1451

## Proposed Changes

- Restricts the `contains_block` and `contains_block` so they only indicate a block is present if it descends from the finalized root. This helps to ensure that fork choice never points to a block that has been pruned from the database.
- Resolves #1451
- Before importing a block, double-check that its parent is known and a descendant of the finalized root.
- Split a big, monolithic block verification test into smaller tests. 

## Additional Notes

I suspect there would be a craftier way to do the `is_descendant_of_finalized` check, but we're a bit tight on time now and we can optimize later if it starts showing in benches.

## TODO

- [x] Tests
2020-08-14 06:36:38 +00:00