Commit Graph

3533 Commits

Author SHA1 Message Date
Age Manning
c75c06cf16 Update discv5 to alpha.9 (#1517)
## Discovery v5 update

In this update we remove the openssl dependency in favour of rust-crypto. 

The update also removes a series of unnecessary async functions which may improve some of the issues we have been experiencing.
2020-08-15 04:02:14 +00:00
Paul Hauner
6aeb896480 Commit Cargo.lock changes, add build scripts (#1521)
## Issue Addressed

NA

## Proposed Changes

This PR commits the `Cargo.lock` file so it does not indicate a dirty git tree in the version tag. This code should be used for the `v0.2.3` release.

Also, adds a `Makefile` command to produce tarballs for upload on release.

## Additional Info

NA
2020-08-14 22:24:27 +00:00
Paul Hauner
f4a7311008 Update to v0.2.3 (#1519)
## Issue Addressed

NA

## Proposed Changes

Bump versions to v0.2.3.

## Additional Info

NA
2020-08-14 08:32:31 +00:00
Paul Hauner
619ad106cf Restrict fork choice getters to finalized blocks (#1475)
## Issue Addressed

- Resolves #1451

## Proposed Changes

- Restricts the `contains_block` and `contains_block` so they only indicate a block is present if it descends from the finalized root. This helps to ensure that fork choice never points to a block that has been pruned from the database.
- Resolves #1451
- Before importing a block, double-check that its parent is known and a descendant of the finalized root.
- Split a big, monolithic block verification test into smaller tests. 

## Additional Notes

I suspect there would be a craftier way to do the `is_descendant_of_finalized` check, but we're a bit tight on time now and we can optimize later if it starts showing in benches.

## TODO

- [x] Tests
2020-08-14 06:36:38 +00:00
Paul Hauner
b0a3731fff Introduce a queue for attestations from the network (#1511)
## Issue Addressed

N/A

## Proposed Changes

Introduces the `GossipProcessor`, a multi-threaded (multi-tasked?), non-blocking processor for some messages from the network which require verification and import into the `BeaconChain`.

Initial testing indicates that this massively improves system stability by (a) moving block tasks from the normal executor (b) spreading out attestation load.

## Additional Info

TBC
2020-08-14 04:38:45 +00:00
Pawan Dhananjay
e3d45eda1e Log to file without json format (#1485)
## Issue Addressed

N/A

## Proposed Changes

Earlier, to log to a file, the only options were to redirect stdout/stderr to a file or use json logging. 
Redirecting to stdout/stderr works well but causes issues with mistakenly overwriting the file instead of appending which has resulted in loss of precious logs on multiple occasions for me.

Json logging creates a timestamped backup of the file if it already exists, but the json format itself is hugely annoying.

This PR modifies the `--logfile` option to log as it does in the terminal to a logfile.
2020-08-13 07:00:37 +00:00
Adam Szkoda
05a8399769 Wind down the SSE thread when the client disconnects (#1514)
These started to appear when I `^C` `curl -N http://localhost:5052/beacon/fork/stream`: `Aug 12 13:00:01.539 ERRO Couldn't stream piece hyper::Error(ChannelClosed), service: http`

Something must have changed in hyper since SSE has been implemented because I'm sure I haven't seen those errors before.

This PR properly detects a closed SSE stream and cleans up.
2020-08-13 06:12:18 +00:00
ladidan
e6f45524f9 Update key-management.md (#1515)
## Issue Addressed

consequent use of "wally"

## Proposed Changes

Please list or describe the changes introduced by this PR.

## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.
2020-08-13 05:25:51 +00:00
Adam Szkoda
8a1a4051cf Fix a bug in fork pruning (#1507)
Extracted from https://github.com/sigp/lighthouse/pull/1380 because merging #1380 proves to be contentious.

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-08-12 07:00:00 +00:00
ladidan
61367efa64 Update key-management.md (#1508)
## Issue Addressed

minor documentation changes in order to have identical command prompts and description below

## Proposed Changes

adjust description "wally" to align with command prompt

## Additional Info

devs might give it a thought whether command line should be "mywallet"
I personally prefer "wally" for minimization reasons =)
2020-08-12 01:24:36 +00:00
Age Manning
70089f5231 Prints the version when starting lighthouse (#1506)
Prints the version to assist with debugging logs.
2020-08-11 08:55:19 +00:00
Paul Hauner
b063df5bf9 Cross-compile to vendored x86_84, aarch64 (Raspberry Pi 4) (#1497)
## Issue Addressed

NA

## Proposed Changes

Adds support for using the [`cross`](https://github.com/rust-embedded/cross) project to produce cross-compiled binaries using Docker images.

Provides quite clean and simple cross-compiles cause all the complexity is hidden in Dockerfiles. It does require you to be in the `docker` group though.

## Details

- Adds shortcut commands to `Makefile`
- Ensures `reqwest` and `discv5` use vendored openssl libs (i.e., static not shared).
- Switches to a [commit](284f705964) of blst that has a renamed C function to avoid a collision with openssl (upstream issue: https://github.com/supranational/blst/issues/21).
- Updates `ring` to the latest satisfiable version, since an earlier version was causing issues with `cross`.
- Off-topic, but adds extra message about Windows support as suggested by Discord user.

## Additional Info

- ~~Blocked on #1495~~
- There are no tests in CI for this yet for a few reasons:
  - I'm hesitant to add more long-running tasks.
  - Short-term bitrot should be avoided since we'll use it each release.
  - In the long term I think it would be good to automate binary creation on a release.
- I observed the binaries increase in size from 50mb to 52mb after these changes.
2020-08-11 05:16:30 +00:00
Adam Szkoda
b83fcd5e5c Local testnet fixes (#1499)
Fixes some outdated instructions and improves scripts portability so that they work e.g. on NixOS.
2020-08-11 02:16:33 +00:00
divma
1a67d15701 Mitigate too many outgoing connections (#1469)
limit simultaneous outgoing connections attempts to a reasonable top as an extra layer of protection
also shift the keep alive logic of the rpc handler to avoid needing to update it by hand. I think In rare cases this could make shutting down a connection a bit faster.
2020-08-11 02:16:31 +00:00
realbigsean
ec84183e05 Add graffiti cli flag to the validator client. (#1425)
## Issue Addressed

#1419

## Proposed Changes

Creates a `--graffiti` cli flag in the validator client. If the flag is set, it overrides graffiti in the beacon node. 

## Additional Info
2020-08-11 02:16:29 +00:00
divma
95b55d7170 Block error display (#1503)
## Issue Addressed

#1486
2020-08-11 01:30:26 +00:00
Age Manning
134676fd6f Version bump to v0.2.2 (#1496)
Version bump to v0.2.2
2020-08-10 06:49:03 +00:00
Age Manning
cbfae87aa6 Upgrade logs (#1495)
## Issue Addressed

#1483 

## Proposed Changes

Upgrades the log to a critical if a listener fails. We are able to listen on many interfaces so a single instance is not critical. We should however gracefully shutdown the client if we have no listeners, although the client can still function solely on outgoing connections.

For now a critical is raised and I leave #1494 for more sophisticated handling of this. 

This also updates discv5 to handle errors of binding to a UDP socket such that lighthouse is now able to handle them.
2020-08-10 05:19:51 +00:00
Age Manning
04e4389efe Patch gossipsub (#1490)
## Issue Addressed

Some nodes not following head, high CPU usage and HTTP API delays

## Proposed Changes

Patches gossipsub. Gossipsub was using an `lru_time_cache` to check for duplicates. This contained an `O(N)` lookup for every gossipsub message to update the time cache. This was causing high cpu usage and blocking network threads. 

This PR introduces a custom cache without `O(N)` inserts. 

This also adds built in safety mechanisms to prevent gossipsub from excessively retrying connections upon failure. A maximum limit is set after which we disconnect from the node from too many failed substream connections.
2020-08-08 08:09:04 +00:00
Age Manning
08a31c5a1a Disconnect peers (#1484)
## Issue Addressed

Peers that connected after the peer limit may remain connected in some circumstances. 

This ensures peers not in the peer manager's list get disconnected. Further logging is also added to track this behaviour.
2020-08-08 06:08:44 +00:00
Age Manning
a1f9769040 Libp2p update (#1482)
Updates to latest libp2p master. 

This now has native noise support. 

This PR
- Removes secio support
- Prioritises mplex over yamux
2020-08-08 02:17:32 +00:00
Naoya Okanami
1d5d3e3ea7 Fix typo (#1476)
just a typo fix :)
managment -> management
2020-08-07 00:09:39 +00:00
Paul Hauner
b354a83faa Upgrade version to v0.2.1 (#1478)
## Issue Addressed

NA

## Proposed Changes

Updates version strings to `v0.2.1`.

## Additional Info

N/A
2020-08-06 14:35:08 +00:00
Paul Hauner
0b287f6ece Push naive attestations into op pool (#1466)
## Issue Addressed

NA

## Proposed Changes

- When producing a block, go and ensure every attestation in the naive aggregation pool is included in the operation pool. This should help us increase the number of useful attestations in a block.
- Lift the `RwLock`s inside `NaiveAggregationPool` up into a single high-level lock. There were race conditions in the existing setup and it was hard to reason about.

## Additional Info

NA
2020-08-06 07:26:46 +00:00
Age Manning
ee036cba7e Correct version string (#1474)
Corrects the version string to expected result

i.e 
`Lighthouse/v0.2.0-0feb3cf1/aarch64-linux`
2020-08-06 05:34:54 +00:00
Naoya Okanami
f4fe2ac533 Fix double lighthouse (#1473)
## Issue Addressed

Resolve https://github.com/sigp/lighthouse/issues/1465

## Proposed Changes

Replace `"Lighthouse/"` with an empty string.

## Additional Info

NA
2020-08-06 05:34:52 +00:00
divma
7d87e11e0f Fix rpc coded response display (#1470)
Prevent errors to be printed in debug mode
2020-08-06 04:29:23 +00:00
Jan K
cfae5fbbc4 Update installation.md (#1467)
Install cmake on macOS

## Issue Addressed

Installation error on macOS

## Proposed Changes

Add instructions for installing `cmake` on macOS via homebrew.
2020-08-06 04:29:21 +00:00
Pawan Dhananjay
983f768034 Remove ssz encoding support from rpc (#1457)
## Issue Addressed

Partially resolves #1422 

## Proposed Changes

Remove ssz encoding from req/resp in rpc.
2020-08-06 04:29:19 +00:00
divma
138c0cf7f0 Remove block clone (#1448)
## Issue Addressed

#1028 

A bit late, but I think if `BlockError` had a kind (the current `BlockError` minus everything on the variants that comes directly from the block) and the original block, more clones could be removed
2020-08-06 04:29:17 +00:00
Pawan Dhananjay
82a0973935 Fix EnrForkId computation (#1441)
## Issue Addressed

Fixes #1433 

## Proposed Changes

Fix the computation for EnrForkId for generating pre-genesis enr.
2020-08-06 04:29:15 +00:00
Age Manning
09a615b2c0 Lighthouse crate v0.2.0 bump (#1450)
## Description

This PR marks Lighthouse v0.2.0. 

This release marks the stable version of Lighthouse, ready for the approaching Medalla testnet.
2020-08-06 03:43:05 +00:00
divma
924ba66218 Update v0.12.2 gossip params (#1449)
## Issue Addressed
#1422
2020-08-06 00:04:33 +00:00
Paul Hauner
6206d8e79b Ensure .git is copied into docker (#1462)
## Issue Addressed

- Resolves #1461 

## Proposed Changes

Copy the `.git` directory across when building docker so we can get commit information.

Unfortunately this means duplicating you `.git` directory which might be quite large (mine is >100mb). Notably this directory isn't contained in the final image, just the intermediate builder image.

## Additional Info

NA
2020-08-05 03:05:36 +00:00
Paul Hauner
5629126f45 Add reason to invalid attestation log (#1460)
## Issue Addressed

NA

## Proposed Changes

Adds an extra field to a debug log so we can see *why* an attestation was invalid.

## Additional Info

NA
2020-08-05 01:49:52 +00:00
Taneli Hukkinen
20ee893969 Fix a typo in validator client CLI (#1456) 2020-08-04 12:51:06 +00:00
Paul Hauner
0feb3cf19a Update README.md (#1455)
## Issue Addressed

NA

## Proposed Changes

Fixes and out-of-date statement.

## Additional Info

NA
2020-08-04 09:19:14 +00:00
Paul Hauner
f26adc0a36 Lighthouse v0.2.0 (Medalla) (#1452)
## Issue Addressed

NA

## Proposed Changes

- Moves the git-based versioning we were doing into the `lighthouse_version` crate in `common`.
- Removes the `beacon_node/version` crate, replacing it with `lighthouse_version`.
- Bumps the version to `v0.2.0`.

## Additional Info

There are now two types of version string:

1. `const VERSION: &str = Lighthouse/v0.2.0-1419501f2+`
1. `version_with_platform() = Lighthouse/v0.2.0-1419501f2+/x86_64-linux`

(1) is handy cause it's a `const` and shorter. (2) has platform info so it's more useful. Note that the plus-sign (`+`) indicates the the git commit is dirty (it used to be `(modified)` but I had to shorten it to fit into graffiti).

These version strings are now included on:

- `lighthouse --version`
- `lcli --version`
- `curl localhost:5052/node/version`
- p2p messages when we communicate our version

You can update the version by changing this constant (version is not related to a `Cargo.toml`):

b9ad7102d5/common/lighthouse_version/src/lib.rs (L4-L15)
2020-08-04 07:44:53 +00:00
Paul Hauner
d4dd25883f Update sigp/blst commit (#1454)
## Issue Addressed

NA

## Proposed Changes

Merges `blst/master` into our `sigp/portable` branch.

## Additional Info

NA
2020-08-04 06:20:09 +00:00
Michael Sproul
c5722093d3 Clarify lack of Windows support (#1453)
## Issue Addressed

Replaces and closes #1446

## Proposed Changes

Clarify that Windows is only supported via WSL in the docs.
2020-08-04 06:20:07 +00:00
divma
1bbecbcf26 Track gossip subscriptions as a metric (#1445)
## Issue Addressed
#1399 

## Proposed Changes
Set an Int gauge per topic and inc/dec when peers subscribe/unsubscribe
2020-08-04 04:18:10 +00:00
Age Manning
31707ccf45 Shift author to sigma prime on some crates (#1440)
Shifts the author to sigma prime on some crates
2020-08-04 02:31:41 +00:00
Age Manning
1419501f2e Update peerdb constants (#1444)
Increases the cache for disconnected and banned peers.
2020-08-03 12:48:22 +00:00
Age Manning
6c17b4696f Update score decay behaviour (#1442)
## Issue Addressed

Banned peer's scores would decay rapidly once the banning timeout expired. 

## Proposed Changes

Update the decay logic for banned peers to start once the banned peer timeout expires.
2020-08-03 11:12:19 +00:00
Age Manning
37679b8898
Update score decay behaviour (#1442) 2020-08-03 20:46:08 +10:00
Age Manning
f634f073a8 Correct issue with network message passing (#1439)
## Issue Addressed

Sync was breaking occasionally. The root cause appears to be identify crashing as events we being sent to the protocol after nodes were banned. Have not been able to reproduce sync issues since this update. 

## Proposed Changes

Only send messages to sub-behaviour protocols if the peer manager thinks the peer is connected. All other messages are dropped.
2020-08-03 09:35:53 +00:00
Paul Hauner
142e033c34 Add Medalla genesis state, more boot enr (#1438)
## Issue Addressed

NA

## Proposed Changes

- Include the Medalla `genesis.ssz` file.
- Adds the boot nodes from here: 27e7e7cc87/medalla/bootnodes.txt

## Additional Info

NA
2020-08-03 07:49:24 +00:00
Age Manning
3b5da8f35f Gossipsub update (#1432)
## Issue Addressed

The most recent gossipsub update had an issue where some privacy settings lead to not sending a sequence number with the message. Although Lighthouse treats these as valid (based on current configuration) other clients may not. 

This corrects gossipsub to send sequence numbers where expected and based on the configuration settings.
2020-08-02 13:19:56 +00:00
Michael Sproul
3ea01ac26b Add top-level feature to enable Milagro (#1428)
## Proposed Changes

In the continuing war against unportable binaries I figured we should have an option to enable building the Lighthouse binary itself with Milagro. This PR adds a `milagro` feature that can be used with `cargo install --path lighthouse --features milagro --force --locked`. The BLS library in-use will also show up under `lighthouse --version` like this:

```
Lighthouse 0.1.2-7d8acc20a(modified)
BLS Library: milagro
```

Future work: add other cool stuff like the compiler version and CPU target to `--version`.
2020-08-01 05:52:55 +00:00
Michael Sproul
d0f1a3e59f Check deposit signatures when submitting via CLI (#1414)
## Proposed Changes

Having been bitten by submitting deposits with invalid signatures (for the wrong testnet), I'm proposing we check the signatures of deposits when submitting them via `lighthouse account validator deposit`.

The impact on performance is likely to be minimal because waiting for an Eth1 transaction confirmation takes a lot longer than verifying a single signature.

It would be great to have the same protection in the deposit web UI in the Lighthouse Book, but I imagine that will be obsoleted by the validator web UI work.
2020-07-31 05:47:11 +00:00