Commit Graph

3525 Commits

Author SHA1 Message Date
Adam Szkoda
8a1a4051cf Fix a bug in fork pruning (#1507)
Extracted from https://github.com/sigp/lighthouse/pull/1380 because merging #1380 proves to be contentious.

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-08-12 07:00:00 +00:00
ladidan
61367efa64 Update key-management.md (#1508)
## Issue Addressed

minor documentation changes in order to have identical command prompts and description below

## Proposed Changes

adjust description "wally" to align with command prompt

## Additional Info

devs might give it a thought whether command line should be "mywallet"
I personally prefer "wally" for minimization reasons =)
2020-08-12 01:24:36 +00:00
Age Manning
70089f5231 Prints the version when starting lighthouse (#1506)
Prints the version to assist with debugging logs.
2020-08-11 08:55:19 +00:00
Paul Hauner
b063df5bf9 Cross-compile to vendored x86_84, aarch64 (Raspberry Pi 4) (#1497)
## Issue Addressed

NA

## Proposed Changes

Adds support for using the [`cross`](https://github.com/rust-embedded/cross) project to produce cross-compiled binaries using Docker images.

Provides quite clean and simple cross-compiles cause all the complexity is hidden in Dockerfiles. It does require you to be in the `docker` group though.

## Details

- Adds shortcut commands to `Makefile`
- Ensures `reqwest` and `discv5` use vendored openssl libs (i.e., static not shared).
- Switches to a [commit](284f705964) of blst that has a renamed C function to avoid a collision with openssl (upstream issue: https://github.com/supranational/blst/issues/21).
- Updates `ring` to the latest satisfiable version, since an earlier version was causing issues with `cross`.
- Off-topic, but adds extra message about Windows support as suggested by Discord user.

## Additional Info

- ~~Blocked on #1495~~
- There are no tests in CI for this yet for a few reasons:
  - I'm hesitant to add more long-running tasks.
  - Short-term bitrot should be avoided since we'll use it each release.
  - In the long term I think it would be good to automate binary creation on a release.
- I observed the binaries increase in size from 50mb to 52mb after these changes.
2020-08-11 05:16:30 +00:00
Adam Szkoda
b83fcd5e5c Local testnet fixes (#1499)
Fixes some outdated instructions and improves scripts portability so that they work e.g. on NixOS.
2020-08-11 02:16:33 +00:00
divma
1a67d15701 Mitigate too many outgoing connections (#1469)
limit simultaneous outgoing connections attempts to a reasonable top as an extra layer of protection
also shift the keep alive logic of the rpc handler to avoid needing to update it by hand. I think In rare cases this could make shutting down a connection a bit faster.
2020-08-11 02:16:31 +00:00
realbigsean
ec84183e05 Add graffiti cli flag to the validator client. (#1425)
## Issue Addressed

#1419

## Proposed Changes

Creates a `--graffiti` cli flag in the validator client. If the flag is set, it overrides graffiti in the beacon node. 

## Additional Info
2020-08-11 02:16:29 +00:00
divma
95b55d7170 Block error display (#1503)
## Issue Addressed

#1486
2020-08-11 01:30:26 +00:00
Age Manning
134676fd6f Version bump to v0.2.2 (#1496)
Version bump to v0.2.2
2020-08-10 06:49:03 +00:00
Age Manning
cbfae87aa6 Upgrade logs (#1495)
## Issue Addressed

#1483 

## Proposed Changes

Upgrades the log to a critical if a listener fails. We are able to listen on many interfaces so a single instance is not critical. We should however gracefully shutdown the client if we have no listeners, although the client can still function solely on outgoing connections.

For now a critical is raised and I leave #1494 for more sophisticated handling of this. 

This also updates discv5 to handle errors of binding to a UDP socket such that lighthouse is now able to handle them.
2020-08-10 05:19:51 +00:00
Age Manning
04e4389efe Patch gossipsub (#1490)
## Issue Addressed

Some nodes not following head, high CPU usage and HTTP API delays

## Proposed Changes

Patches gossipsub. Gossipsub was using an `lru_time_cache` to check for duplicates. This contained an `O(N)` lookup for every gossipsub message to update the time cache. This was causing high cpu usage and blocking network threads. 

This PR introduces a custom cache without `O(N)` inserts. 

This also adds built in safety mechanisms to prevent gossipsub from excessively retrying connections upon failure. A maximum limit is set after which we disconnect from the node from too many failed substream connections.
2020-08-08 08:09:04 +00:00
Age Manning
08a31c5a1a Disconnect peers (#1484)
## Issue Addressed

Peers that connected after the peer limit may remain connected in some circumstances. 

This ensures peers not in the peer manager's list get disconnected. Further logging is also added to track this behaviour.
2020-08-08 06:08:44 +00:00
Age Manning
a1f9769040 Libp2p update (#1482)
Updates to latest libp2p master. 

This now has native noise support. 

This PR
- Removes secio support
- Prioritises mplex over yamux
2020-08-08 02:17:32 +00:00
Naoya Okanami
1d5d3e3ea7 Fix typo (#1476)
just a typo fix :)
managment -> management
2020-08-07 00:09:39 +00:00
Paul Hauner
b354a83faa Upgrade version to v0.2.1 (#1478)
## Issue Addressed

NA

## Proposed Changes

Updates version strings to `v0.2.1`.

## Additional Info

N/A
2020-08-06 14:35:08 +00:00
Paul Hauner
0b287f6ece Push naive attestations into op pool (#1466)
## Issue Addressed

NA

## Proposed Changes

- When producing a block, go and ensure every attestation in the naive aggregation pool is included in the operation pool. This should help us increase the number of useful attestations in a block.
- Lift the `RwLock`s inside `NaiveAggregationPool` up into a single high-level lock. There were race conditions in the existing setup and it was hard to reason about.

## Additional Info

NA
2020-08-06 07:26:46 +00:00
Age Manning
ee036cba7e Correct version string (#1474)
Corrects the version string to expected result

i.e 
`Lighthouse/v0.2.0-0feb3cf1/aarch64-linux`
2020-08-06 05:34:54 +00:00
Naoya Okanami
f4fe2ac533 Fix double lighthouse (#1473)
## Issue Addressed

Resolve https://github.com/sigp/lighthouse/issues/1465

## Proposed Changes

Replace `"Lighthouse/"` with an empty string.

## Additional Info

NA
2020-08-06 05:34:52 +00:00
divma
7d87e11e0f Fix rpc coded response display (#1470)
Prevent errors to be printed in debug mode
2020-08-06 04:29:23 +00:00
Jan K
cfae5fbbc4 Update installation.md (#1467)
Install cmake on macOS

## Issue Addressed

Installation error on macOS

## Proposed Changes

Add instructions for installing `cmake` on macOS via homebrew.
2020-08-06 04:29:21 +00:00
Pawan Dhananjay
983f768034 Remove ssz encoding support from rpc (#1457)
## Issue Addressed

Partially resolves #1422 

## Proposed Changes

Remove ssz encoding from req/resp in rpc.
2020-08-06 04:29:19 +00:00
divma
138c0cf7f0 Remove block clone (#1448)
## Issue Addressed

#1028 

A bit late, but I think if `BlockError` had a kind (the current `BlockError` minus everything on the variants that comes directly from the block) and the original block, more clones could be removed
2020-08-06 04:29:17 +00:00
Pawan Dhananjay
82a0973935 Fix EnrForkId computation (#1441)
## Issue Addressed

Fixes #1433 

## Proposed Changes

Fix the computation for EnrForkId for generating pre-genesis enr.
2020-08-06 04:29:15 +00:00
Age Manning
09a615b2c0 Lighthouse crate v0.2.0 bump (#1450)
## Description

This PR marks Lighthouse v0.2.0. 

This release marks the stable version of Lighthouse, ready for the approaching Medalla testnet.
2020-08-06 03:43:05 +00:00
divma
924ba66218 Update v0.12.2 gossip params (#1449)
## Issue Addressed
#1422
2020-08-06 00:04:33 +00:00
Paul Hauner
6206d8e79b Ensure .git is copied into docker (#1462)
## Issue Addressed

- Resolves #1461 

## Proposed Changes

Copy the `.git` directory across when building docker so we can get commit information.

Unfortunately this means duplicating you `.git` directory which might be quite large (mine is >100mb). Notably this directory isn't contained in the final image, just the intermediate builder image.

## Additional Info

NA
2020-08-05 03:05:36 +00:00
Paul Hauner
5629126f45 Add reason to invalid attestation log (#1460)
## Issue Addressed

NA

## Proposed Changes

Adds an extra field to a debug log so we can see *why* an attestation was invalid.

## Additional Info

NA
2020-08-05 01:49:52 +00:00
Taneli Hukkinen
20ee893969 Fix a typo in validator client CLI (#1456) 2020-08-04 12:51:06 +00:00
Paul Hauner
0feb3cf19a Update README.md (#1455)
## Issue Addressed

NA

## Proposed Changes

Fixes and out-of-date statement.

## Additional Info

NA
2020-08-04 09:19:14 +00:00
Paul Hauner
f26adc0a36 Lighthouse v0.2.0 (Medalla) (#1452)
## Issue Addressed

NA

## Proposed Changes

- Moves the git-based versioning we were doing into the `lighthouse_version` crate in `common`.
- Removes the `beacon_node/version` crate, replacing it with `lighthouse_version`.
- Bumps the version to `v0.2.0`.

## Additional Info

There are now two types of version string:

1. `const VERSION: &str = Lighthouse/v0.2.0-1419501f2+`
1. `version_with_platform() = Lighthouse/v0.2.0-1419501f2+/x86_64-linux`

(1) is handy cause it's a `const` and shorter. (2) has platform info so it's more useful. Note that the plus-sign (`+`) indicates the the git commit is dirty (it used to be `(modified)` but I had to shorten it to fit into graffiti).

These version strings are now included on:

- `lighthouse --version`
- `lcli --version`
- `curl localhost:5052/node/version`
- p2p messages when we communicate our version

You can update the version by changing this constant (version is not related to a `Cargo.toml`):

b9ad7102d5/common/lighthouse_version/src/lib.rs (L4-L15)
2020-08-04 07:44:53 +00:00
Paul Hauner
d4dd25883f Update sigp/blst commit (#1454)
## Issue Addressed

NA

## Proposed Changes

Merges `blst/master` into our `sigp/portable` branch.

## Additional Info

NA
2020-08-04 06:20:09 +00:00
Michael Sproul
c5722093d3 Clarify lack of Windows support (#1453)
## Issue Addressed

Replaces and closes #1446

## Proposed Changes

Clarify that Windows is only supported via WSL in the docs.
2020-08-04 06:20:07 +00:00
divma
1bbecbcf26 Track gossip subscriptions as a metric (#1445)
## Issue Addressed
#1399 

## Proposed Changes
Set an Int gauge per topic and inc/dec when peers subscribe/unsubscribe
2020-08-04 04:18:10 +00:00
Age Manning
31707ccf45 Shift author to sigma prime on some crates (#1440)
Shifts the author to sigma prime on some crates
2020-08-04 02:31:41 +00:00
Age Manning
1419501f2e Update peerdb constants (#1444)
Increases the cache for disconnected and banned peers.
2020-08-03 12:48:22 +00:00
Age Manning
6c17b4696f Update score decay behaviour (#1442)
## Issue Addressed

Banned peer's scores would decay rapidly once the banning timeout expired. 

## Proposed Changes

Update the decay logic for banned peers to start once the banned peer timeout expires.
2020-08-03 11:12:19 +00:00
Age Manning
37679b8898
Update score decay behaviour (#1442) 2020-08-03 20:46:08 +10:00
Age Manning
f634f073a8 Correct issue with network message passing (#1439)
## Issue Addressed

Sync was breaking occasionally. The root cause appears to be identify crashing as events we being sent to the protocol after nodes were banned. Have not been able to reproduce sync issues since this update. 

## Proposed Changes

Only send messages to sub-behaviour protocols if the peer manager thinks the peer is connected. All other messages are dropped.
2020-08-03 09:35:53 +00:00
Paul Hauner
142e033c34 Add Medalla genesis state, more boot enr (#1438)
## Issue Addressed

NA

## Proposed Changes

- Include the Medalla `genesis.ssz` file.
- Adds the boot nodes from here: 27e7e7cc87/medalla/bootnodes.txt

## Additional Info

NA
2020-08-03 07:49:24 +00:00
Age Manning
3b5da8f35f Gossipsub update (#1432)
## Issue Addressed

The most recent gossipsub update had an issue where some privacy settings lead to not sending a sequence number with the message. Although Lighthouse treats these as valid (based on current configuration) other clients may not. 

This corrects gossipsub to send sequence numbers where expected and based on the configuration settings.
2020-08-02 13:19:56 +00:00
Michael Sproul
3ea01ac26b Add top-level feature to enable Milagro (#1428)
## Proposed Changes

In the continuing war against unportable binaries I figured we should have an option to enable building the Lighthouse binary itself with Milagro. This PR adds a `milagro` feature that can be used with `cargo install --path lighthouse --features milagro --force --locked`. The BLS library in-use will also show up under `lighthouse --version` like this:

```
Lighthouse 0.1.2-7d8acc20a(modified)
BLS Library: milagro
```

Future work: add other cool stuff like the compiler version and CPU target to `--version`.
2020-08-01 05:52:55 +00:00
Michael Sproul
d0f1a3e59f Check deposit signatures when submitting via CLI (#1414)
## Proposed Changes

Having been bitten by submitting deposits with invalid signatures (for the wrong testnet), I'm proposing we check the signatures of deposits when submitting them via `lighthouse account validator deposit`.

The impact on performance is likely to be minimal because waiting for an Eth1 transaction confirmation takes a lot longer than verifying a single signature.

It would be great to have the same protection in the deposit web UI in the Lighthouse Book, but I imagine that will be obsoleted by the validator web UI work.
2020-07-31 05:47:11 +00:00
divma
4d77784bb8 Rate limit RPC requests (#1402)
## Issue Addressed
#1056 

## Proposed Changes
- Add a rate limiter to the RPC behaviour. This also means the rate limiting occurs just before the door to the application level, so the number of connections a peer opens does not affect this (this would happen in the future if put on the handler)
- The algorithm used is the leaky bucket as a meter / token bucket implemented the GCRA way
- Each protocol has its own limit. Due to the way the algorithm works, the "small" protocols have a hard limit, while bbrange and bbroot allow [burstiness](https://www.wikiwand.com/en/Burstiness). This is so that a peer can't request hundreds of individual requests expecting only one block in a short period of time, it also allows a peer to send two half size requests instead of one with max if they want to without getting limited, and.. it also allows a peer to request a batch of the maximum size and then send _appropriately spaced_ requests of really small sizes. From what I've seen in sync this is plausible when reaching the target slot.

## Additional Info
Needs to be heavily tested
2020-07-31 05:47:09 +00:00
Michael Sproul
7d8acc20a0 Add a flag to make lighthouse portable across machines (#1423)
## Issue Addressed

Closes #1395

## Proposed Changes

* Add a feature to `lighthouse` and `lcli` called `portable` which enables the `portable` feature on our fork of BLST. This feature turns off the `-march=native` C compiler flag that produces binaries highly targeted to the host CPU's instruction set.
* Tweak the `Makefile` so that when the `PORTABLE` environment variable is set to `true`, it compiles with this feature.
* Temporarily enable `PORTABLE=true` in the Docker build so that the image on Docker Hub is portable. Eventually I think we should enable `PORTABLE=true` _only on Docker Hub_, so that users building locally can take advantage of the tasty compiler magic. This seems to be possible by setting a Docker Hub environment variable: https://docs.docker.com/docker-hub/builds/#environment-variables-for-builds

## Additional Info

Tested by compiling on a very new CPU (Intel Core i7-8550U) and copying the binary to a very old CPU (Intel Core i3 530). Before the portability fix, this produced the SIGILL crash described in #1395, and after the fix, it worked smoothly.

I'm in the process of testing the Docker build and running some benches to confirm that the performance penalty isn't too severe.
2020-07-31 05:00:39 +00:00
Age Manning
2ede9caaa6 Book modifications to include testnet param (#1418)
## Issue Addressed

N/A

## Proposed Changes

Update the lighthouse book to accommodate the new `--testnet` CLI param. 

## Additional Info
2020-07-31 02:14:50 +00:00
Age Manning
a37e75f44b
Downgrade sync and rpc warn logs (#1417)
* Downgrade sycn and rpc warn logs

* Correct warning
2020-07-30 13:52:44 +10:00
Age Manning
febb300a2d Limit incoming connection requests (#1413)
## Issue Addressed

Limits the number of incoming connections and adjusts the buffer sizes in libp2p
2020-07-29 06:39:30 +00:00
Paul Hauner
36d3d37cb4 Add support for multiple testnet flags (#1396)
## Issue Addressed

NA

## Proposed Changes

Allows for multiple "hardcoded" testnets.

## Additional Info

This PR is incomplete.

## TODO

- [x] Add flag to CLI, integrate with rest of Lighthouse.


Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-07-29 06:39:29 +00:00
Age Manning
395d99ce03 Sync update (#1412)
## Issue Addressed

Recurring sync loop and invalid batch downloading

## Proposed Changes

Shifts the batches to include the first slot of each epoch. This ensures the finalized is always downloaded once a chain has completed syncing. 

Also add in logic to prevent re-dialing disconnected peers. Non-performant peers get disconnected during sync, this prevents re-connection to these during sync. 

## Additional Info

N/A
2020-07-29 05:25:10 +00:00
Michael Sproul
f53dedb27d Improve account manager CLI (#1404)
## Proposed Changes

Fixes some sharp edges on the new `lighthouse account validator list` command, and the account manager CLI.

* Validator names/keys are always printed in the same order due to the use of a sorted `BTreeMap`
* The `validator list` subcommand now respects the `--validator-dir` flag, instead of always looking in `~/.lighthouse/validators`
* The `--help` now shows a description for the `wallet` subcommand instead of just `TODO`
2020-07-29 04:32:52 +00:00