Commit Graph

1241 Commits

Author SHA1 Message Date
divma
4d77784bb8 Rate limit RPC requests (#1402)
## Issue Addressed
#1056 

## Proposed Changes
- Add a rate limiter to the RPC behaviour. This also means the rate limiting occurs just before the door to the application level, so the number of connections a peer opens does not affect this (this would happen in the future if put on the handler)
- The algorithm used is the leaky bucket as a meter / token bucket implemented the GCRA way
- Each protocol has its own limit. Due to the way the algorithm works, the "small" protocols have a hard limit, while bbrange and bbroot allow [burstiness](https://www.wikiwand.com/en/Burstiness). This is so that a peer can't request hundreds of individual requests expecting only one block in a short period of time, it also allows a peer to send two half size requests instead of one with max if they want to without getting limited, and.. it also allows a peer to request a batch of the maximum size and then send _appropriately spaced_ requests of really small sizes. From what I've seen in sync this is plausible when reaching the target slot.

## Additional Info
Needs to be heavily tested
2020-07-31 05:47:09 +00:00
Age Manning
a37e75f44b
Downgrade sync and rpc warn logs (#1417)
* Downgrade sycn and rpc warn logs

* Correct warning
2020-07-30 13:52:44 +10:00
Age Manning
febb300a2d Limit incoming connection requests (#1413)
## Issue Addressed

Limits the number of incoming connections and adjusts the buffer sizes in libp2p
2020-07-29 06:39:30 +00:00
Paul Hauner
36d3d37cb4 Add support for multiple testnet flags (#1396)
## Issue Addressed

NA

## Proposed Changes

Allows for multiple "hardcoded" testnets.

## Additional Info

This PR is incomplete.

## TODO

- [x] Add flag to CLI, integrate with rest of Lighthouse.


Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-07-29 06:39:29 +00:00
Age Manning
395d99ce03 Sync update (#1412)
## Issue Addressed

Recurring sync loop and invalid batch downloading

## Proposed Changes

Shifts the batches to include the first slot of each epoch. This ensures the finalized is always downloaded once a chain has completed syncing. 

Also add in logic to prevent re-dialing disconnected peers. Non-performant peers get disconnected during sync, this prevents re-connection to these during sync. 

## Additional Info

N/A
2020-07-29 05:25:10 +00:00
Age Manning
ba0f3daf9d Gossipsub update (#1400)
## Issue Addressed

N/A

## Proposed Changes

This provides a number of corrections and improvements to gossipsub. Specifically
- Enables options for greater privacy around the message author
- Provides greater flexibility on message validation
- Prevents unvalidated messages from being gossiped
- Shifts the duplicate cache to a time-based cache inside gossipsub
- Updates the message-id to handle bytes
- Bug fixes related to mesh maintenance and topic subscription. This should improve our attestation inclusion rate.
2020-07-29 03:40:22 +00:00
realbigsean
09b40b7a5e Discover query grouping (#1364)
## Issue Addressed

#1281

## Proposed Changes

Groups queries for specific subnets into groups of up to 3.

## Additional Info
2020-07-29 02:43:50 +00:00
divma
9ae9df806c Fix clippy lints rpc (#1401)
## Issue Addressed
#1388 partially (eth2_libp2p & network)

## Proposed Changes 
TLDR at the end
- *Complex types* are 3 on the handlers/Behaviours but the types are `Poll<ComplexType>` where `ComplexType` comes from the traits of libp2p. Those, I don't thing are worth an alias. A couple more were from using tokio combinators and were removed writing things the async way and using [`BoxFuture`](https://docs.rs/futures/0.3.5/futures/future/type.BoxFuture.html)
- The *cognitive complexity*.. I tried to address those before (they come from the poll functions too) and tbh they are cognitively simpler to understand the way they are now. Moving separate parts to functions doesn't add much since that code is not repeated and they all do early returns. If moved those returns would now need to be wrapped in an Option, probably, and checked to be returned again. I would leave them like that but that's just preference.
- *Too many arguments*: They are not easily put together in a wrapping struct since the parameters don't relate semantically (Ex: fn new with a log, a reference to the chain, a peer, etc) but some may differ.
- *Needless returns* were indeed needless

## Additional Info
TLDR: removed needless return, used BoxFuture and async, left the rest untouched since those lgtm
2020-07-28 01:39:42 +00:00
Paul Hauner
0b5be9b2c0
Add info about peer scoring to block/attestation errors (#1393)
* Add comments to `BlockError`

* Add `AttnError` comments

* Clean up
2020-07-26 13:16:49 +10:00
Paul Hauner
e5d9d6179f Add info about valid deposit count to logs (#1391)
## Issue Addressed

NA

## Proposed Changes

Adds a `valid_deposits` field to the logs whilst waiting for genesis:

```
Jul 25 11:02:25.631 INFO Waiting for more deposits               valid_deposits: 3085, total_deposits: 3188, min_genesis_active_validators: 16384, service: beacon
```

In this example we can see there are `3188` deposits, but only `3085` have valid signatures.

## Additional Info

NA
2020-07-25 04:44:10 +00:00
Paul Hauner
b73c497be2 Support multiple BLS implementations (#1335)
## Issue Addressed

NA

## Proposed Changes

- Refactor the `bls` crate to support multiple BLS "backends" (e.g., milagro, blst, etc).
- Removes some duplicate, unused code in `common/rest_types/src/validator.rs`.
- Removes the old "upgrade legacy keypairs" functionality (these were unencrypted keys that haven't been supported for a few testnets, no one should be using them anymore).

## Additional Info

Most of the files changed are just inconsequential changes to function names.

## TODO

- [x] Optimization levels
- [x] Infinity point: https://github.com/supranational/blst/issues/11
- [x] Ensure milagro *and* blst are tested via CI
- [x] What to do with unsafe code?
- [x] Test infinity point in signature sets
2020-07-25 02:03:18 +00:00
blacktemplar
23a8f31f83 Fix clippy warnings (#1385)
## Issue Addressed

NA

## Proposed Changes

Fixes most clippy warnings and ignores the rest of them, see issue #1388.
2020-07-23 14:18:00 +00:00
divma
ba10c80633 Refactor inbound substream logic with async (#1325)
## Issue Addressed
#1112 

The logic is slightly different but still valid wrt to error handling.
- Inbound state is either Busy with a future that return the subtream (and info about the processing)
- The state machine works as follows:
  - `Idle` with pending responses => `Busy`
  - `Busy` => finished ? if so and there are new pending responses then `Busy`, if not then `Idle`
               => not finished remains `Busy`
- Add an `InboundInfo` for readability
- Other stuff:
  - Close inbound substreams when all expected responses are sent
  - Remove the error variants from `RPCCodedResponse` and use the codes instead
  - Fix various spelling mistakes because I got sloppy last time

Sorry for the delay

Co-authored-by: Age Manning <Age@AgeManning.com>
2020-07-23 12:30:43 +00:00
blacktemplar
3c4daec9af
replace max_peers cli argument by target_peers and use excess peers above target_peers capped by a new constant PEER_EXCESS_FACTOR (relative to target_peers) (#1383) 2020-07-23 13:55:36 +10:00
Pawan Dhananjay
3a888d6ef3 Fix early return from DepositLog parsing (#1382)
## Issue Addressed

N/A

## Proposed Changes

When parsing deposit logs, we were returning an error in case `PublicKeyBytes` or `SignatureBytes` didn't convert to valid bls `PublicKey` or `Signature` types. This would stall our import of deposit logs. 
Fixes this by returning `signature_is_valid` as `false` in `DepositLog` if the bytes are invalid `PublicKey/Signature` types.

Tested this fix on the Onyx deposit contract where the bug was observed and it works correctly as expected.
2020-07-22 10:24:37 +00:00
Akihito Nakano
41f7547645 Remove unused event handler function (#1377)
## Issue Addressed

`websocket_event_handler` has been unused since #1107.
2020-07-22 10:24:35 +00:00
Akihito Nakano
ea0e936ac4 Small improvement: encapsulate a public field (#1362)
## Issue Addressed

This PR makes the `Eth1Chain::use_dummy_backend` field private. I believe this could be good to ensure the consistency  of a Eth1Chain instance. 💡
2020-07-22 09:34:57 +00:00
blacktemplar
f61a7113ac
Do not send regular status updates during syncing (#1375) 2020-07-22 15:39:56 +10:00
Age Manning
0620f54f2f
Update discv5 and remove discovery retries (#1373) 2020-07-21 14:19:55 +10:00
Pawan Dhananjay
1c90c816b7
Close outbound stream after sending (#1369) 2020-07-21 10:57:44 +10:00
Age Manning
e940dcea47
Updates libp2p and noise hack (#1367) 2020-07-21 10:54:47 +10:00
Pawan Dhananjay
b885d79ac3
Fix attestation propagation (#1360)
* Add `should_process` for conditional processing of Attestations

* Remove ATTESTATIONS_IGNORED metric
2020-07-20 12:55:32 +10:00
Adam Szkoda
fc5e6cbbb0 Add ability to configure CORS header (#1345)
## Issue Addressed

https://github.com/sigp/lighthouse/issues/1177

## Proposed Changes

Add a command line option (`--http-allow-origin`) and a config item for configuring the `Access-Control-Allow-Origin` response header.  This should unblock making XMLHttpRequests.
2020-07-16 07:23:14 +00:00
Age Manning
4a01f44206 Downgrade libp2p and gosispsub (#1358)
Downgrades libp2p and the gossipsub updates. 

This looks to resolve the CPU usage issue we have been seeing. 

The root cause is likely inside the latest gossipsub updates, which will be addressed in a later PR
2020-07-15 05:04:09 +00:00
Akihito Nakano
4b213032b2 Tiny improvement: lifetime parameters (#1354)
## Issue Addressed

This PR improves lifetime parameters that can be elided. :octocat:
2020-07-14 22:26:39 +00:00
pscott
e164371083 Set Graffiti via CLI (#1320)
## Issue Addressed

Closes #1319 

## Proposed Changes

This issue:
1. Allows users to edit their Graffiti via the cli option `--graffiti`. If the graffiti is too long, lighthouse will not start and throw an error message. Otherwise, it will set the Graffiti to be the one provided by the user, right-padded with 0s.
2. Create a new `Graffiti` type and unify the code around it. With this type, everything is enforced at compile-time, and the code can be (I think...) panic-free! :)

## Additional info

Currently, only `&str` are supported, as this is the returned type by `.arg("graffiti")`.
Since this is user-input, I tried being as careful as I could. This is also why I created the `Graffiti` type, to make sure I could check as much as possible at compile time.
2020-07-14 08:05:02 +00:00
Age Manning
e6a8635b38
Discovery update (#1349)
* Improve logging

* Discovery update
2020-07-11 12:35:59 +10:00
Akihito Nakano
9ae218bfac
Remove outdated comments (#1336) 2020-07-11 10:33:11 +10:00
Pawan Dhananjay
275148a152
Fix query expiry computation (#1346) 2020-07-09 11:43:48 +10:00
Age Manning
025b262e01
Peer scoring updates (#1342)
* Prevent banned peers from interacting with the behaviour

* Update smallvec

* Add logs

* Correct log update
2020-07-07 20:08:05 +10:00
Age Manning
f500b24242
Update smallvec (#1339) 2020-07-07 16:57:27 +10:00
Michael Sproul
20a48df80a
Fix race condition in VC block proposal service (#1282)
Closes #918
Closes #923
2020-07-07 14:03:21 +10:00
Age Manning
5bc8fea2e0
Activate peer scoring (#1284)
* Initial score structure

* Peer manager update

* Updates to dialing

* Correct tests

* Correct typos and remove unused function

* Integrate scoring into the network crate

* Clean warnings

* Formatting

* Shift core functionality into the behaviour

* Temp commit

* Shift disconnections into the behaviour

* Temp commit

* Update libp2p and gossipsub

* Remove gossipsub lru cache

* Correct merge conflicts

* Modify handler and correct tests

* Update enr network globals on socket update

* Apply clippy lints

* Add new prysm fingerprint

* More clippy fixes
2020-07-07 10:13:16 +10:00
Age Manning
5977c00edb
Update libp2p and gossipsub (#1324)
* Update libp2p and gossipsub

* Remove gossipsub lru cache
2020-07-06 20:34:40 +10:00
Paul Hauner
e429c3eefe
Remove old block processing shim (#1327)
* Remove old block processing shim

* Run rustfmt

* Fix log formatting

* Swap peer ids over to display
2020-07-06 16:28:00 +10:00
Paul Hauner
25cd91ce26
Update deps (#1322)
* Run cargo update

* Upgrade prometheus

* Update hex

* Upgrade parking-lot

* Upgrade num-bigint

* Upgrade sha2

* Update dockerfile Rust version

* Run cargo update
2020-07-06 11:55:56 +10:00
Adam Szkoda
c7f47af9fb
Harden the freezing procedure against failures (#1323)
* Enable logging in tests

* Migrate states to the freezer atomically
2020-07-03 09:47:31 +10:00
Adam Szkoda
536728b975
Write new blocks and states to the database atomically (#1285)
* Mostly atomic put_state()
* Reduce number of vec allocations
* Make crucial db operations atomic
* Save restore points
* Remove StateBatch
* Merge two HotColdDB impls
* Further reduce allocations
* Review feedback
* Silence clippy warning
2020-07-01 12:45:57 +10:00
Akihito Nakano
a7a79ce4b7
Fix a retry limit condition (#1306) 2020-06-29 21:16:12 +10:00
Pawan Dhananjay
d4dd9fae07
Fix BlocksByRootRequest min/max bounds calculation (#1312) 2020-06-29 21:03:14 +10:00
Age Manning
fc0b8adcd7
Temporary noise fix (#1311)
* Add workaround to noise and add back secio fallback

* Hardcoded size hack

* prioritise secio
2020-06-29 20:25:42 +10:00
ethDreamer
721323f045
get_active_validator_indices() now has bound check (#1300) 2020-06-29 18:21:51 +10:00
Mark Mackey
d90bd648d8
Fixed 1 line bug 2020-06-28 21:12:03 -04:00
Paul Hauner
9f6ee212ff
Merge pull request #1294 from gakonst/fix/self-instead-of-service
fix(eth1/service): use self instead of Service
2020-06-29 10:18:36 +10:00
Age Manning
9fc290a344
Add waker to attestation service (#1305)
* Add waker to attestation service

* Formatting
2020-06-28 22:29:27 +10:00
Paul Hauner
6e7d5c6a7c
Add metrics for validator subscriptions (#1302) 2020-06-28 10:47:03 +10:00
Michael Sproul
7688b5f1dd
Merge remote-tracking branch 'origin/master' into spec-v0.12 2020-06-26 12:57:56 +10:00
Paul Hauner
e0e41fc8e5
Cache deposit signature verification (#1298)
* Bake in Altona testnet (without genesis state)

* Add sig verification, without optimization

* Start integration with genesis service

* Update config.yml

* Fix eth2_testnet_config test

* Stop using default spec in genesis

* Fix lcli compile error

* Update min genesis time

* Fix typo
2020-06-26 11:43:06 +10:00
pscott
02174e21d8
Fix clippy's performance lints (#1286)
* Fix clippy perf lints

* Cargo fmt

* Add  and  to lint rule in Makefile

* Fix some leftover clippy lints
2020-06-26 00:04:08 +10:00
Georgios Konstantopoulos
4ddfc032e2
test(eth1/service): make tests use references 2020-06-25 08:57:26 +03:00