Commit Graph

205 Commits

Author SHA1 Message Date
platschi
2357a41868 Fix typo in advanced networking docs (#2412)
## Proposed Changes

Fixing a typo in the advanced networking docs which mentions ``--target-peer`` instead of the correct ``--target-peers`` flag
2021-06-22 00:17:06 +00:00
Michael Sproul
6583ce325b Minify slashing protection interchange data (#2380)
## Issue Addressed

Closes #2354

## Proposed Changes

Add a `minify` method to `slashing_protection::Interchange` that keeps only the maximum-epoch attestation and maximum-slot block for each validator. Specifically, `minify` constructs "synthetic" attestations (with no `signing_root`) containing the maximum source epoch _and_ the maximum target epoch from the input. This is equivalent to the `minify_synth` algorithm that I've formally verified in this repository:

https://github.com/michaelsproul/slashing-proofs

## Additional Info

Includes the JSON loading optimisation from #2347
2021-06-21 05:46:36 +00:00
Pawan Dhananjay
fdaeec631b Monitoring service api (#2251)
## Issue Addressed

N/A

## Proposed Changes

Adds a client side api for collecting system and process metrics and pushing it to a monitoring service.
2021-05-26 05:58:41 +00:00
ethDreamer
ba55e140ae Enable Compatibility with Windows (#2333)
## Issue Addressed

Windows incompatibility.

## Proposed Changes

On windows, lighthouse needs to default to STDIN as tty doesn't exist. Also Windows uses ACLs for file permissions. So to mirror chmod 600, we will remove every entry in a file's ACL and add only a single SID that is an alias for the file owner.

Beyond that, there were several changes made to different unit tests because windows has slightly different error messages as well as frustrating nuances around killing a process :/

## Additional Info

Tested on my Windows VM and it appears to work, also compiled & tested on Linux with these changes. Permissions look correct on both platforms now. Just waiting for my validator to activate on Prater so I can test running full validator client on windows.

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2021-05-19 23:05:16 +00:00
Michael Sproul
0754ba3be7 Fix typos in redundancy docs (#2320)
## Proposed Changes

Fix a long-standing typo in the redundancy docs that uses `lighthouse bn` instead of `lighthouse vc`.
2021-04-25 23:55:59 +00:00
Pawan Dhananjay
2992ca66cd Add a no-wait flag for voluntary exits (#2292)
## Issue Addressed

N/A

## Proposed Changes

Adds a `no-wait` flag to the validator exit command which exits right after publishing the voluntary exit to the beacon chain. It does not wait for confirmation that the exit has been included in the beacon chain. By default, the flag is false.

cc @stefa2k
2021-04-16 05:26:53 +00:00
stefa2k
66590d043c Correcting command example validator exit (#2291)
## Issue Addressed

None

## Proposed Changes

Using correct flag in example for exiting validator.

## Additional Info

None
2021-04-03 00:38:51 +00:00
stefa2k
39928d5c69 correcting name of flag (#2285)
## Issue Addressed

None, just a very small fix of documentation

## Proposed Changes

Fixing naming of paramter listed in documentation.

## Additional Info

No changes to code, just fixing documentation
2021-03-29 06:39:15 +00:00
Håvard Anda Estensen
d3d8c22edf Add to docs that macOS binaries are available (#2281)
## Proposed Changes

macOS binaries are available and should be listed in the Lighthouse Book
2021-03-26 04:53:58 +00:00
Pawan Dhananjay
da8791abd7 Set graffiti per validator (#2044)
## Issue Addressed

Resolves #1944 

## Proposed Changes

Adds a "graffiti" key to the `validator_definitions.yml`. Setting the key will override anything passed through the validator `--graffiti` flag. 
Returns an error if the value for the graffiti key is > 32 bytes instead of silently truncating.
2021-03-02 22:35:46 +00:00
Michael Sproul
e2ff9c66a1 Remove links to old master branch (#2190)
## Proposed Changes

In preparation for deleting the `master` branch, remove all links to it from the book/README.
2021-02-11 06:06:54 +00:00
Paul Hauner
2b2a358522 Detailed validator monitoring (#2151)
## Issue Addressed

- Resolves #2064

## Proposed Changes

Adds a `ValidatorMonitor` struct which provides additional logging and Grafana metrics for specific validators.

Use `lighthouse bn --validator-monitor` to automatically enable monitoring for any validator that hits the [subnet subscription](https://ethereum.github.io/eth2.0-APIs/#/Validator/prepareBeaconCommitteeSubnet) HTTP API endpoint.

Also, use `lighthouse bn --validator-monitor-pubkeys` to supply a list of validators which will always be monitored.

See the new docs included in this PR for more info.

## TODO

- [x] Track validator balance, `slashed` status, etc.
- [x] ~~Register slashings in current epoch, not offense epoch~~
- [ ] Publish Grafana dashboard, update TODO link in docs
- [x] ~~#2130 is merged into this branch, resolve that~~
2021-01-20 19:19:38 +00:00
Taneli Hukkinen
9cdfa94ba4 Update docs: Change --beacon-node to --beacon-nodes (#2145)
## Issue Addressed

The docs use the deprecated `--beacon-node` flag

## Proposed Changes

Reference the new `--beacon-nodes` flag in docs
2021-01-19 03:50:08 +00:00
Paul Hauner
1d535659d6 Add docs about redundancy (#2142)
## Issue Addressed

- Resolves #2140

## Proposed Changes

Adds some documentation on the topic of "redundancy".

## Additional Info

NA
2021-01-12 00:26:22 +00:00
Pawan Dhananjay
166f617b19 Add docs for /lighthouse/validators/keystore api (#2071)
## Issue Addressed

Resolves #2061 
Resolves #2066 

## Proposed Changes

Document the `/lighthouse/validators/keystore` validator api method. 
The newly generated/imported keystore is always added to the key cache from this function call
65dcdc361b/validator_client/src/validator_store.rs (L105-L109)

which eventually invokes `KeyCache::add` here if enabled
65dcdc361b/validator_client/src/initialized_validators.rs (L192)
2020-12-21 07:43:04 +00:00
Rémy Roy
0f5f3b522e Fix default values and --network flag in Voluntary exits book page (#2056)
## Issue Addressed

None yet reported.

## Proposed Changes

Fix the old flag in the Voluntary exits book page to use the new `--network` flag. Also fix the default value for that flag.
2020-12-06 22:16:05 +00:00
Conor Svensson
075eecdcb1 Fix broken custom data directories link (#2000)
## Issue Addressed

No issue - its a broken link in the docs.

## Proposed Changes

Fix the broken link.

## Additional Info

N/A in this instance.
2020-12-01 00:04:44 +00:00
Paul Hauner
77f3539654 Improve eth1 block sync (#2008)
## Issue Addressed

NA

## Proposed Changes

- Log about eth1 whilst waiting for genesis.
- For the block and deposit caches, update them after each download instead of when *all* downloads are complete.
  - This prevents the case where a single timeout error can cause us to drop *all* previously download blocks/deposits.
- Set `max_log_requests_per_update` to avoid timeouts due to very large log counts in a response.
- Set `max_blocks_per_update` to prevent a single update of the block cache to download an unreasonable number of blocks.
  - This shouldn't have any affect in normal use, it's just a safe-guard against bugs.
- Increase the timeout for eth1 calls from 15s to 60s, as per @pawanjay176's experience with Infura.

## Additional Info

NA
2020-11-30 20:29:17 +00:00
Justin
cadcc9a76b Fix possible typo in build from source instructions (#1990) 2020-11-28 06:41:34 +00:00
Paul Hauner
26741944b1 Add metrics to VC (#1954)
## Issue Addressed

NA

## Proposed Changes

- Adds a HTTP server to the VC which provides Prometheus metrics.
- Moves the health metrics into the `lighthouse_metrics` crate so it can be shared between BN/VC.
- Sprinkle some metrics around the VC.
- Update the book to indicate that we now have VC metrics.
- Shifts the "waiting for genesis" logic later in the `ProductionValidatorClient::new_from_cli`
  - This is worth attention during the review.

## Additional Info

- ~~`clippy` has some new lints that are failing. I'll deal with that in another PR.~~
2020-11-26 01:10:51 +00:00
realbigsean
7b6a97e73c FAQ/Doc updates (#1966)
## Issue Addressed

N/A

## Proposed Changes

Adding a few FAQ's, updating some formatting


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-11-25 05:51:10 +00:00
Michael Sproul
20339ade01 Refine and test slashing protection semantics (#1885)
## Issue Addressed

Closes #1873

## Proposed Changes

Fixes the bug in slashing protection import (#1873) by pruning the database upon import.

Also expands the test generator to cover this case and a few others which are under discussion here:

https://ethereum-magicians.org/t/eip-3076-validator-client-interchange-format-slashing-protection/4883

## Additional Info

Depending on the outcome of the discussion on Eth Magicians, we can either wait for consensus before merging, or merge our preferred solution and patch things later.
2020-11-24 07:21:14 +00:00
Paul Hauner
e504645767 Update validator guide for mainnet (#1951)
## Issue Addressed

NA

## Proposed Changes

Updates the validator guide to provide instructions for mainnet users.

## Additional Info

- ~~Blocked on #1751~~
2020-11-24 04:42:17 +00:00
Paul Hauner
21617aa87f Change --testnet flag to --network (#1751)
## Issue Addressed

- Resolves #1689

## Proposed Changes

TBC

## Additional Info

NA
2020-11-23 23:54:03 +00:00
Michael Sproul
7d644103c6 Tweak slasher DB schema and pruning (#1948)
## Issue Addressed

Resolves #1890

## Proposed Changes

Change the slasher database schema to key indexed attestations by `(target_epoch, indexed_attestation_root)` instead of just `indexed_attestation_root`. This allows more straight-forward pruning (linear scan), that is also "re-entrant". By re-entrant, we mean that a pruning pass that gets stuck because of a `MapFull` error can attempt to commit midway, and be resumed later without issue. The previous pruning strategy for indexed attestations did not have this property. There was also a flaw in the previous pruning that could leave "zombie" indexed attestations in the database (ones not referenced by any attester record), which could build up and contribute to bloat (although in practice I think they occur quite infrequently).

## Additional Info

During testing I noticed that a `MapFull` error can still occur during the commit of the transaction itself, which is irritating, but not unbearable. This PR should at least reduce the frequency with which users need to manually resize their DB, and if the `MapFull` on commit rears its ugly head too often we could use a dynamic strategy (temporarily increase the size of the map until the transaction commits).

The extra bytes for the epoch make the database a bit heavier, so the size estimate docs have been updated to reflect this. This is also a breaking schema change, so anyone using a v0 database from a few hours ago will need to drop it and update 😅
2020-11-23 21:33:51 +00:00
Michael Sproul
5828ff1204 Implement slasher (#1567)
This is an implementation of a slasher that lives inside the BN and can be enabled via `lighthouse bn --slasher`.

Features included in this PR:

- [x] Detection of attester slashing conditions (double votes, surrounds existing, surrounded by existing)
- [x] Integration into Lighthouse's attestation verification flow
- [x] Detection of proposer slashing conditions
- [x] Extraction of attestations from blocks as they are verified
- [x] Compression of chunks
- [x] Configurable history length
- [x] Pruning of old attestations and blocks
- [x] More tests

Future work:

* Focus on a slice of history separate from the most recent N epochs (e.g. epochs `current - K` to `current - M`)
* Run out-of-process
* Ingest attestations from the chain without a resync

Design notes are here https://hackmd.io/@sproul/HJSEklmPL
2020-11-23 03:43:22 +00:00
Paul Hauner
48f73b21e6 Expand eth1 block cache, add more logs (#1938)
## Issue Addressed

NA

## Proposed Changes

- Caches later blocks than is required by `ETH1_FOLLOW_DISTANCE`.
- Adds logging to `warn` if the eth1 cache is insufficiently primed.
- Use `max_by_key` instead of `max_by` in `BeaconChain::Eth1Chain` since it's simpler.
- Rename `voting_period_start_timestamp` to `voting_target_timestamp` for accuracy.

## Additional Info

The reason for eating into the `ETH1_FOLLOW_DISTANCE` and caching blocks that are closer to the head is due to possibility for `SECONDS_PER_ETH1_BLOCK` to be incorrect (as is the case for the Pyrmont testnet on Goerli).

If `SECONDS_PER_ETH1_BLOCK` is too short, we'll skip back too far from the head and skip over blocks that would be valid [`is_candidate_block`](https://github.com/ethereum/eth2.0-specs/blob/v1.0.0/specs/phase0/validator.md#eth1-data) blocks. This was the case on the Pyrmont testnet and resulted in Lighthouse choosing blocks that were about 30 minutes older than is ideal.
2020-11-21 00:26:15 +00:00
Michael Sproul
37369c6a56 Document system requirements (#1934)
## Proposed Changes

Document some minimal and recommended system specs for running Lighthouse on mainnet with a modest number of validators.
2020-11-19 21:23:56 +00:00
Paul Hauner
fe71f25c3a Add Pyrmont testnet (#1904)
## Issue Addressed

NA

## Proposed Changes

- Replace Zinken with Pyrmont (Zinken has been sun-setted).
- Ensure Mainnet is build in the build script.

## Additional Info

NA
2020-11-16 05:11:35 +00:00
Michael Sproul
b3fc48e887 Update slashing protection interchange to v5 (#1816)
## Proposed Changes

Update the slashing protection interchange format to v5 in preparation for finalisation as part of an EIP.

Also, add some more tests and update the commit hash for https://github.com/eth2-clients/slashing-protection-interchange-tests to include the new generated tests.
2020-11-09 05:04:01 +00:00
Geoffry Song
63fe5542e7 Remove mention of OpenSSL from documentation (#1844)
If I'm not mistaken, openssl is no longer a dependency of lighthouse, so it can no longer cause build issues.
2020-11-09 02:31:31 +00:00
Paul Hauner
157e31027a Add warnings for deposits (#1858)
## Issue Addressed

NA

## Proposed Changes

Add some warnings to discourage users to user Lighthouse for mainnet.

## Additional Info

NA
2020-11-04 19:46:42 +00:00
Michael Sproul
2ff5828310 Downgrade ADX check to a warning (#1846)
## Issue Addressed

Closes #1842

## Proposed Changes

Due to the lies told to us by VPS providers about what CPU features they support, we are forced to check for the availability of CPU features like ADX by just _running code and seeing if it crashes_. The prominent warning should hopefully help users who have truly incompatible CPUs work out what is going on, while not burdening users of cheap VPSs.
2020-11-02 22:35:37 +00:00
Paul Hauner
7afbaa807e Return eth1-related data via the API (#1797)
## Issue Addressed

- Related to #1691

## Proposed Changes

Adds the following API endpoints:

- `GET lighthouse/eth1/syncing`: status about how synced we are with Eth1.
- `GET lighthouse/eth1/block_cache`: all locally cached eth1 blocks.
- `GET lighthouse/eth1/deposit_cache`: all locally cached eth1 deposits.

Additionally:

- Moves some types from the `beacon_node/eth1` to the `common/eth2` crate, so they can be used in the API without duplication.
- Allow `update_deposit_cache` and `update_block_cache` to take an optional head block number to avoid duplicate requests.

## Additional Info

TBC
2020-11-02 00:37:30 +00:00
Pawan Dhananjay
56f9394141 Add cli option for voluntary exits (#1781)
## Issue Addressed

Resolve #1652 

## Proposed Changes

Adds a cli option for voluntary exits. The flow is similar to prysm's where after entering the password for the validator keystore (or load password from `secrets` if present) the user is given multiple warnings about the operation being irreversible, then redirected to the docs webpage(not added yet) which explains what a voluntary exit is and the consequences of exiting and then prompted to enter a phrase from the docs webpage as a final confirmation. 

Example usage
```
$ lighthouse --testnet zinken account validator exit --validator <validator-pubkey> --beacon-node http://localhost:5052

Running account manager for zinken testnet                                                                                                          
validator-dir path: "..."

Enter the keystore password:  for validator in ...

Password is correct

Publishing a voluntary exit for validator: ...              
WARNING: This is an irreversible operation                                                                                                                    
WARNING: Withdrawing staked eth will not be possible until Eth1/Eth2 merge Please visit [website] to make sure you understand the implications of a voluntary exit.            
                                                                                                                                             
Enter the phrase from the above URL to confirm the voluntary exit:
Exit my validator
Published voluntary exit for validator ...
```

## Additional info

Not sure if we should have batch exits (`--validator all`) option for exiting all the validators in the `validators` directory. I'm slightly leaning towards having only single exits but don't have a strong preference.
2020-10-29 23:25:19 +00:00
Michael Sproul
36bd4d87f0 Update to spec v1.0.0-rc.0 and BLSv4 (#1765)
## Issue Addressed

Closes #1504 
Closes #1505
Replaces #1703
Closes #1707

## Proposed Changes

* Update BLST and Milagro to versions compatible with BLSv4 spec
* Update Lighthouse to spec v1.0.0-rc.0, and update EF test vectors
* Use the v1.0.0 constants for `MainnetEthSpec`.
* Rename `InteropEthSpec` -> `V012LegacyEthSpec`
    * Change all constants to suit the mainnet `v0.12.3` specification (i.e., Medalla).
* Deprecate the `--spec` flag for the `lighthouse` binary
    * This value is now obtained from the `config_name` field of the `YamlConfig`.
        * Built in testnet YAML files have been updated.
    * Ignore the `--spec` value, if supplied, log a warning that it will be deprecated
    * `lcli` still has the spec flag, that's fine because it's dev tooling.
* Remove the `E: EthSpec` from `YamlConfig`
    * This means we need to deser the genesis `BeaconState` on-demand, but this is fine.
* Swap the old "minimal", "mainnet" strings over to the new `EthSpecId` enum.
* Always require a `CONFIG_NAME` field in `YamlConfig` (it used to have a default).

## Additional Info

Lots of breaking changes, do not merge! ~~We will likely need a Lighthouse v0.4.0 branch, and possibly a long-term v0.3.0 branch to keep Medalla alive~~.

Co-authored-by: Kirk Baird <baird.k@outlook.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-10-28 22:19:38 +00:00
Paul Hauner
b829257cca Ssz state (#1749)
## Issue Addressed

NA

## Proposed Changes

Adds a `lighthouse/beacon/states/:state_id/ssz` endpoint to allow us to pull the genesis state from the API.

## Additional Info

NA
2020-10-22 06:05:49 +00:00
Paul Hauner
e1eec7828b Fix error in VC API docs (#1800)
## Issue Addressed

NA

## Proposed Changes

- Ensure the `description` field is included with the output (as per the implementation).

## Additional Info

NA
2020-10-22 00:26:04 +00:00
Pawan Dhananjay
aadbab47cc Doc fixes (#1762)
## Issue Addressed

N/A

## Proposed Changes

Minor doc fixes. Adds a section on custom data directories.



Co-authored-by: Michael Sproul <micsproul@gmail.com>
2020-10-15 00:37:00 +00:00
Michael Sproul
467de4c8d0 Add docs for slashing protection (#1760)
## Proposed Changes

* Add documentation about slashing protection, including how to troubleshoot issues and move between clients.
* Add an error message if the validator client is started with 0 validators. Previously it would hit an error relating to the slashing protection database not existing, which wrongly pushed people towards using the unsafe `--init-slashing-protection` flag.
2020-10-13 22:10:07 +00:00
realbigsean
95c96ac567 Small doc fix (#1761)
## Issue Addressed

N/A

## Proposed Changes

Looks like there was some text left over from a merge.

## Additional Info
2020-10-12 23:56:49 +00:00
realbigsean
83ae12a1b4 Fix epoch, slot, and effective balance quoting (#1756)
## Issue Addressed

Resolves #1717

## Proposed Changes

Add quoting for epochs, slots, and `effective_balance`

## Additional Info
2020-10-11 23:58:12 +00:00
Paul Hauner
db3e0578e9
Merge branch 'v0.3.0-staging' into v3-master 2020-10-09 15:27:08 +11:00
Paul Hauner
414138f137 Update docs for v0.3.0 (#1742)
## Issue Addressed

NA

## Proposed Changes

- Remove Metamask deposits from the docs.
    - Restructure docs to be launchpad-centric.
- Remove references to sigp/lighthouse-docker.
- Add section about binaries.


## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.
2020-10-09 00:43:49 +00:00
Justin
cf74e0baed
Document need for port 9000 to be open (fix #730) (#731)
Co-authored-by: Age Manning <Age@AgeManning.com>

Edited by Paul H when cherry-picking from master to v0.3.0-staging
2020-10-05 17:34:24 +11:00
Justin
39bd511838 Document need for port 9000 to be open (fix #730) (#731)
Co-authored-by: Age Manning <Age@AgeManning.com>
2020-10-05 03:20:53 +00:00
Geoffry Song
8fde9a4016
Wallet creation: Make mnemonic length configurable, default to 24 words. (#1697)
## Issue Addressed

Fixes #1665.

## Proposed Changes

`lighthouse account_manager wallet create` now generates a 24-word
mnemonic. The user can override this by passing `--mnemonic-length 12`
(or another legal bip39 length).

## Additional Info

CLI `--help`:
```
        --mnemonic-length <MNEMONIC_LENGTH>       The number of words to use for the mnemonic phrase. [default: 24]
```

In case of an invalid argument:
```
% lighthouse account_manager wallet create --mnemonic-length 25
error: Invalid value for '--mnemonic-length <MNEMONIC_LENGTH>': Mnemonic length must be one of 12, 15, 18, 21, 24
```
2020-10-03 10:01:06 +10:00
Paul Hauner
6ea3bc5e52 Implement VC API (#1657)
## Issue Addressed

NA

## Proposed Changes

- Implements a HTTP API for the validator client.
- Creates EIP-2335 keystores with an empty `description` field, instead of a missing `description` field. Adds option to set name.
- Be more graceful with setups without any validators (yet)
    - Remove an error log when there are no validators.
    - Create the `validator` dir if it doesn't exist.
- Allow building a `ValidatorDir` without a withdrawal keystore (required for the API method where we only post a voting keystore).
- Add optional `description` field to `validator_definitions.yml`

## TODO

- [x] Signature header, as per https://github.com/sigp/lighthouse/issues/1269#issuecomment-649879855
- [x] Return validator descriptions
- [x] Return deposit data
- [x] Respect the mnemonic offset
- [x] Check that mnemonic can derive returned keys
- [x] Be strict about non-localhost
- [x] Allow graceful start without any validators (+ create validator dir)
- [x] Docs final pass
- [x] Swap to EIP-2335 description field. 
- [x] Fix Zerioze TODO in VC api types.
- [x] Zeroize secp256k1 key

## Endpoints

- [x] `GET /lighthouse/version`
- [x] `GET /lighthouse/health`
- [x] `GET /lighthouse/validators` 
- [x] `POST /lighthouse/validators/hd`
- [x] `POST /lighthouse/validators/keystore`
- [x] `PATCH /lighthouse/validators/:validator_pubkey`
- [ ] ~~`POST /lighthouse/validators/:validator_pubkey/exit/:epoch`~~ Future works


## Additional Info

TBC
2020-10-02 09:42:19 +00:00
Geoffry Song
2cc20101d4 Wallet creation: Make mnemonic length configurable, default to 24 words. (#1697)
## Issue Addressed

Fixes #1665.

## Proposed Changes

`lighthouse account_manager wallet create` now generates a 24-word
mnemonic. The user can override this by passing `--mnemonic-length 12`
(or another legal bip39 length).

## Additional Info

CLI `--help`:
```
        --mnemonic-length <MNEMONIC_LENGTH>       The number of words to use for the mnemonic phrase. [default: 24]
```

In case of an invalid argument:
```
% lighthouse account_manager wallet create --mnemonic-length 25
error: Invalid value for '--mnemonic-length <MNEMONIC_LENGTH>': Mnemonic length must be one of 12, 15, 18, 21, 24
```
2020-10-02 07:51:50 +00:00
Paul Hauner
cdec3cec18
Implement standard eth2.0 API (#1569)
- Resolves #1550
- Resolves #824
- Resolves #825
- Resolves #1131
- Resolves #1411
- Resolves #1256
- Resolve #1177

- Includes the `ShufflingId` struct initially defined in #1492. That PR is now closed and the changes are included here, with significant bug fixes.
- Implement the https://github.com/ethereum/eth2.0-APIs in a new `http_api` crate using `warp`. This replaces the `rest_api` crate.
- Add a new `common/eth2` crate which provides a wrapper around `reqwest`, providing the HTTP client that is used by the validator client and for testing. This replaces the `common/remote_beacon_node` crate.
- Create a `http_metrics` crate which is a dedicated server for Prometheus metrics (they are no longer served on the same port as the REST API). We now have flags for `--metrics`, `--metrics-address`, etc.
- Allow the `subnet_id` to be an optional parameter for `VerifiedUnaggregatedAttestation::verify`. This means it does not need to be provided unnecessarily by the validator client.
- Move `fn map_attestation_committee` in `mod beacon_chain::attestation_verification` to a new `fn with_committee_cache` on the `BeaconChain` so the same cache can be used for obtaining validator duties.
- Add some other helpers to `BeaconChain` to assist with common API duties (e.g., `block_root_at_slot`, `head_beacon_block_root`).
- Change the `NaiveAggregationPool` so it can index attestations by `hash_tree_root(attestation.data)`. This is a requirement of the API.
- Add functions to `BeaconChainHarness` to allow it to create slashings and exits.
- Allow for `eth1::Eth1NetworkId` to go to/from a `String`.
- Add functions to the `OperationPool` to allow getting all objects in the pool.
- Add function to `BeaconState` to check if a committee cache is initialized.
- Fix bug where `seconds_per_eth1_block` was not transferring over from `YamlConfig` to `ChainSpec`.
- Add the `deposit_contract_address` to `YamlConfig` and `ChainSpec`. We needed to be able to return it in an API response.
- Change some uses of serde `serialize_with` and `deserialize_with` to a single use of `with` (code quality).
- Impl `Display` and `FromStr` for several BLS fields.
- Check for clock discrepancy when VC polls BN for sync state (with +/- 1 slot tolerance). This is not intended to be comprehensive, it was just easy to do.

- See #1434 for a per-endpoint overview.
- Seeking clarity here: https://github.com/ethereum/eth2.0-APIs/issues/75

- [x] Add docs for prom port to close #1256
- [x] Follow up on this #1177
- [x] ~~Follow up with #1424~~ Will fix in future PR.
- [x] Follow up with #1411
- [x] ~~Follow up with  #1260~~ Will fix in future PR.
- [x] Add quotes to all integers.
- [x] Remove `rest_types`
- [x] Address missing beacon block error. (#1629)
- [x] ~~Add tests for lighthouse/peers endpoints~~ Wontfix
- [x] ~~Follow up with validator status proposal~~ Tracked in #1434
- [x] Unify graffiti structs
- [x] ~~Start server when waiting for genesis?~~ Will fix in future PR.
- [x] TODO in http_api tests
- [x] Move lighthouse endpoints off /eth/v1
- [x] Update docs to link to standard

- ~~Blocked on #1586~~

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-10-01 11:12:36 +10:00
Pawan Dhananjay
8e20176337
Directory restructure (#1532)
Closes #1487
Closes #1427

Directory restructure in accordance with #1487. Also has temporary migration code to move the old directories into new structure.
Also extracts all default directory names and utility functions into a `directory` crate to avoid repetitio.

~Since `validator_definition.yaml` stores absolute paths, users will have to manually change the keystore paths or delete the file to get the validators picked up by the vc.~. `validator_definition.yaml` is migrated as well from the default directories.

Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-10-01 11:12:35 +10:00
Pawan Dhananjay
80ecafaae4 Add --staking flag (#1641)
## Issue Addressed

Closes #1472 

## Proposed Changes

Add `--staking` ~~and`staking-with-eth1-endpoint`~~ flag to improve UX for stakers.


Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-09-23 01:19:58 +00:00
realbigsean
1801dd1a34 Interactive account passwords (#1623)
## Issue Addressed

#1437

## Proposed Changes

- Make the `--wallet-password` flag optional and creates an interactive prompt if not provided.
- Make the `--wallet-name` flag optional and creates an interactive prompt if not provided.
- Add a minimum password requirement of a 12 character length.
- Update the `--stdin-passwords` flag to `--stdin-inputs` because we have non-password user inputs 

## Additional Info
2020-09-23 01:19:54 +00:00
Michael Sproul
62c8548ed0 Revert "Update BLST, add force-adx support (#1595)" (#1649)
This reverts commit 4fca306397.

Something in the BLST update is causing SIGILLs on aarch64 non-portable builds. While we debug the issue, I think it's best if we just revert the update.
2020-09-23 00:25:56 +00:00
Michael Sproul
4fca306397 Update BLST, add force-adx support (#1595)
## Issue Addressed

Closes #1504
Closes https://github.com/sigp/lighthouse/issues/1505

## Proposed Changes

* Update `blst` to the latest version, which is more portable and includes finer-grained compilation controls (see below).
* Detect the case where a binary has been explicitly compiled with ADX support but it's missing at runtime, and report a nicer error than `SIGILL`.

## Known Issues

* None. The previous issue with `make build-aarch64` (https://github.com/supranational/blst/issues/27), has been resolved.

## Additional Info

I think we should tweak our release process and our Docker builds so that we provide two options:

Binaries:

* `lighthouse`: compiled with `modern`/`force-adx`, for CPUs 2013 and newer
* `lighthouse-portable`: compiled with `portable` for older CPUs

Docker images:

* `sigp/lighthouse:latest`: multi-arch image with `modern` x86_64 and vanilla aarch64 binary
* `sigp/lighthouse:latest-portable`: multi-arch image with `portable` builds for x86_64 and aarch64

And relevant Docker images for the releases (as per https://github.com/sigp/lighthouse/pull/1574#issuecomment-687766141), tagged `v0.x.y` and `v0.x.y-portable`
2020-09-22 05:40:02 +00:00
Paul Hauner
dfe507715d Remove references to rust-docs (#1601)
## Issue Addressed

- Resolves #897
- Resolves #821

## Proposed Changes

Removes references to the rust docs that we're no long maintaining.

## Additional Info

NA
2020-09-10 00:24:41 +00:00
realbigsean
9cf8f45192 Mnemonic key recovery (#1579)
## Issue Addressed

N/A

## Proposed Changes

Add a  `lighthouse am wallet recover` command that recreates a wallet from a mnemonic but no validator keys.  Add a `lighthouse am validator recover` command which would directly create keys from a mnemonic for a given index and count.

## Additional Info


Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-09-08 12:17:51 +00:00
Pawan Dhananjay
87181204d0 Minor documentation fixes (#1297)
## Issue Addressed

N/A

## Proposed Changes

- Fix a wrong command in the validator generation example.
- Replace occurrences of 'passphrase' with 'password'. This is mostly because I felt that there was a lot of mixing of the two phrases in the documentation and the actual commands which is a bit confusing. Picked 'password' everywhere because it felt more appropriate but I don't mind changing it to 'passphrase' as long it's consistent everywhere.
2020-09-02 04:59:22 +00:00
Age Manning
8301a984eb Revert 1502 - Switching docker user to lighthouse (#1578)
## Issue Addressed

The lighthouse user has recently changed to `lighthouse` from root. 

This requires uses to change ownership of their current docker mounted volumes and the upgrade path is non-trivial. 
This reverts #1502 and we will include it in a major release in the future.

## Proposed Changes

N/A

## Additional Info

N/A
2020-09-01 01:32:02 +00:00
Maximilian Ehlers
7d71d98dc1 Creates a new lighthouse user and makes it the default user to be use… (#1502)
…d in the Docker image

## Issue Addressed
https://github.com/sigp/lighthouse/issues/1459

## Proposed Changes

- Create new `lighthouse` user and group in Docker container
- Set user as the default user
2020-08-31 07:52:26 +00:00
Nat
3cfd70d7fd Docs: Fix reference to incorrect password file. (#1556)
Leftover "mywallet.pass" -> "wally.pass"

Thanks @pecurliarly (from Discord)!
2020-08-21 03:50:37 +00:00
ladidan
e6f45524f9 Update key-management.md (#1515)
## Issue Addressed

consequent use of "wally"

## Proposed Changes

Please list or describe the changes introduced by this PR.

## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.
2020-08-13 05:25:51 +00:00
ladidan
61367efa64 Update key-management.md (#1508)
## Issue Addressed

minor documentation changes in order to have identical command prompts and description below

## Proposed Changes

adjust description "wally" to align with command prompt

## Additional Info

devs might give it a thought whether command line should be "mywallet"
I personally prefer "wally" for minimization reasons =)
2020-08-12 01:24:36 +00:00
Paul Hauner
b063df5bf9 Cross-compile to vendored x86_84, aarch64 (Raspberry Pi 4) (#1497)
## Issue Addressed

NA

## Proposed Changes

Adds support for using the [`cross`](https://github.com/rust-embedded/cross) project to produce cross-compiled binaries using Docker images.

Provides quite clean and simple cross-compiles cause all the complexity is hidden in Dockerfiles. It does require you to be in the `docker` group though.

## Details

- Adds shortcut commands to `Makefile`
- Ensures `reqwest` and `discv5` use vendored openssl libs (i.e., static not shared).
- Switches to a [commit](284f705964) of blst that has a renamed C function to avoid a collision with openssl (upstream issue: https://github.com/supranational/blst/issues/21).
- Updates `ring` to the latest satisfiable version, since an earlier version was causing issues with `cross`.
- Off-topic, but adds extra message about Windows support as suggested by Discord user.

## Additional Info

- ~~Blocked on #1495~~
- There are no tests in CI for this yet for a few reasons:
  - I'm hesitant to add more long-running tasks.
  - Short-term bitrot should be avoided since we'll use it each release.
  - In the long term I think it would be good to automate binary creation on a release.
- I observed the binaries increase in size from 50mb to 52mb after these changes.
2020-08-11 05:16:30 +00:00
Naoya Okanami
1d5d3e3ea7 Fix typo (#1476)
just a typo fix :)
managment -> management
2020-08-07 00:09:39 +00:00
Jan K
cfae5fbbc4 Update installation.md (#1467)
Install cmake on macOS

## Issue Addressed

Installation error on macOS

## Proposed Changes

Add instructions for installing `cmake` on macOS via homebrew.
2020-08-06 04:29:21 +00:00
Michael Sproul
c5722093d3 Clarify lack of Windows support (#1453)
## Issue Addressed

Replaces and closes #1446

## Proposed Changes

Clarify that Windows is only supported via WSL in the docs.
2020-08-04 06:20:07 +00:00
Michael Sproul
7d8acc20a0 Add a flag to make lighthouse portable across machines (#1423)
## Issue Addressed

Closes #1395

## Proposed Changes

* Add a feature to `lighthouse` and `lcli` called `portable` which enables the `portable` feature on our fork of BLST. This feature turns off the `-march=native` C compiler flag that produces binaries highly targeted to the host CPU's instruction set.
* Tweak the `Makefile` so that when the `PORTABLE` environment variable is set to `true`, it compiles with this feature.
* Temporarily enable `PORTABLE=true` in the Docker build so that the image on Docker Hub is portable. Eventually I think we should enable `PORTABLE=true` _only on Docker Hub_, so that users building locally can take advantage of the tasty compiler magic. This seems to be possible by setting a Docker Hub environment variable: https://docs.docker.com/docker-hub/builds/#environment-variables-for-builds

## Additional Info

Tested by compiling on a very new CPU (Intel Core i7-8550U) and copying the binary to a very old CPU (Intel Core i3 530). Before the portability fix, this produced the SIGILL crash described in #1395, and after the fix, it worked smoothly.

I'm in the process of testing the Docker build and running some benches to confirm that the performance penalty isn't too severe.
2020-07-31 05:00:39 +00:00
Age Manning
2ede9caaa6 Book modifications to include testnet param (#1418)
## Issue Addressed

N/A

## Proposed Changes

Update the lighthouse book to accommodate the new `--testnet` CLI param. 

## Additional Info
2020-07-31 02:14:50 +00:00
Paul Hauner
36d3d37cb4 Add support for multiple testnet flags (#1396)
## Issue Addressed

NA

## Proposed Changes

Allows for multiple "hardcoded" testnets.

## Additional Info

This PR is incomplete.

## TODO

- [x] Add flag to CLI, integrate with rest of Lighthouse.


Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-07-29 06:39:29 +00:00
Paul Hauner
eaa9f9744f Add EF launchpad import (#1381)
## Issue Addressed

NA

## Proposed Changes

Adds an integration for keys generated via https://github.com/ethereum/eth2.0-deposit (In reality keys are *actually* generated here: https://github.com/ethereum/eth2.0-deposit-cli).

## Additional Info

NA

## TODO

- [x] Docs
- [x] Tests

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-07-29 04:32:50 +00:00
Paul Hauner
e26da35cbf Introduce validator definition file for VC (#1357)
## Issue Addressed

NA

## Proposed Changes

- Introduces the `valdiator_definitions.yml` file which serves as an explicit list of validators that should be run by the validator client.
  - Removes `--strict` flag, split into `--strict-lockfiles` and `--disable-auto-discover`  
  - Adds a "Validator Management" page to the book.
- Adds the `common/account_utils` crate which contains some logic that was starting to duplicate across the codebase.

The new docs for this feature are the best description of it (apart from the code, I guess): 9cb87e93ce/book/src/validator-management.md

## API Changes

This change should be transparent for *most* existing users. If the `valdiator_definitions.yml` doesn't exist then it will be automatically generated using a method that will detect all the validators in their `validators_dir`.

Users will have issues if they are:

1. Using `--strict`.
1. Have keystores in their `~/.lighthouse/validators` directory that weren't being detected by the current keystore discovery method.

For users with (1), the VC will refuse to start because the `--strict` flag has been removed. They will be forced to review `--help` and choose an equivalent flag.

For users with (2), this seems fairly unlikely and since we're only in testnets there's no *real* value on the line here. I'm happy to take the risk, it would be a different case for mainnet.

## Additional Info

This PR adds functionality we will need for #1347.

## TODO

- [x] Reconsider flags
- [x] Move doc into a more reasonable chapter.
- [x] Check for compile warnings.
2020-07-22 09:34:55 +00:00
Age Manning
ac2ce2ba6b
Correct book links (#1338)
* Correct book links

* Add count to validator creation

* Correct more links
2020-07-07 17:28:59 +10:00
Pawan Dhananjay
314c077870
Fix altona deposit contract address (#1317) 2020-06-30 08:44:55 +10:00
Paul Hauner
66f183be02
Update docs for Altona (#1304) 2020-06-28 16:04:03 +10:00
Michael Sproul
305724770d
Bump all spec tags to v0.12.1 (#1275) 2020-06-19 11:18:27 +10:00
Michael Sproul
e6f97bf466
Merge remote-tracking branch 'origin/master' into spec-v0.12 2020-06-17 12:34:11 +10:00
Michael Sproul
39bf05e3e5
Update v0.12.0 to v0.12.1 (#1259) 2020-06-11 10:07:10 +10:00
Paul Hauner
d2983c13df
Add raspberry pi docs (#1252) 2020-06-10 17:11:57 +10:00
Michael Sproul
fe03ff0f21
Consensus updates for v0.12 (#1228)
* Update state processing for v0.12

* Fix EF test runners for v0.12

* Fix some tests

* Fix broken attestation verification test

* More test fixes

* Fix typo found in review
2020-06-03 14:56:54 +10:00
Paul Hauner
a214032e1f
Update docs to suit sigp/docker-compose updates (#1235) 2020-06-01 21:18:48 +10:00
Age Manning
cb26ddebb1
Doc updates (#1226)
* Keep wallet name and password file in sync with commands

* Fix create validator example to include required param --count

The current example fails with v0.1.2 fails with "Must supply either --count or --at-most"  This change proposes passing count=1, esp'ly since "Creating another validator is easy" a couple of paragraphs down.

* Fix broken links to wallet-create

Wallet links result in file not found.  May be caused by the lack of `.md`?

* Include node/health in TOC

* Remove unnecessary book files

Co-authored-by: J Burnett <jburnett@users.noreply.github.com>
2020-06-01 11:34:16 +10:00
Paul Hauner
ea4a52984c
Add /node/health endpoint (#1197)
* Start adding health endpoint

* Use psutil more

* Add get_health test

* Expose health to Prom

* Update comments

* Add /node/health to docs

* Update Prom naming
2020-05-28 11:24:24 +10:00
Paul Hauner
8bc82c573d
Update local testnet scripts, fix eth1 sim (#1184)
* Update local testnet scripts

* Add logs when decrypting validators

* Update comment

* Update account manager

* Make random key generation explicit

* Remove unnecessary clap constraint

* Only decrypt voting keypair for eth1 deposit

* Use insecure kdf for insecure keypairs

* Simplify local testnet keygen

* Update local testnet

* Fix eth1 sim

* Add eth1 sim to CI again

* Remove old local testnet docs

* Tidy

* Remove checks for existing validators

* Tidy

* Fix typos
2020-05-26 18:30:44 +10:00
Paul Hauner
ad4e5adabc
Integrate Witti testnet (#1193)
* Update for latest master

* Shift delay inside loop

* Clean up genesis service

* Tidy

* Tidy logs

* Address Michael's comments

* Add pre-genesis logging

* Remove est time till genesis

* Fix time formatting

* Tidy

* Update docs for Witti

* Update JS for Witti

* Ensure deposit data is 0x-prefixed hex

* Hard code witti testnet dir

* Add --auto-register warning

* Integrate local sigp testnet source

* Reword warning
2020-05-26 15:31:41 +10:00
Paul Hauner
f8cac1b822
Update docs to indicate that Schelsi is down (#1191)
* Update docs to indicate that Schelsi is down

* Fix discord link
2020-05-26 09:17:46 +10:00
Paul Hauner
681e013d31
Remove v0.1.2 install instructions (#1169) 2020-05-21 09:05:33 +10:00
ethers
0b49a8507e
Minor fix on wallet output text (#1176)
* Minor fix on wallet output text

* Fix "import" to "important" in docs
2020-05-21 09:04:48 +10:00
Paul Hauner
4331834003
Directory Restructure (#1163)
* Move tests -> testing

* Directory restructure

* Update Cargo.toml during restructure

* Update Makefile during restructure

* Fix arbitrary path
2020-05-18 21:24:23 +10:00
Paul Hauner
c571afb8d8
Wallet-based, encrypted key management (#1138)
* Update hashmap hashset to stable futures

* Adds panic test to hashset delay

* Port remote_beacon_node to stable futures

* Fix lcli merge conflicts

* Non rpc stuff compiles

* Remove padding

* Add error enum, zeroize more things

* Fix comment

* protocol.rs compiles

* Port websockets, timer and notifier to stable futures (#1035)

* Fix lcli

* Port timer to stable futures

* Fix timer

* Port websocket_server to stable futures

* Port notifier to stable futures

* Add TODOS

* Port remote_beacon_node to stable futures

* Partial eth2-libp2p stable future upgrade

* Finished first round of fighting RPC types

* Further progress towards porting eth2-libp2p adds caching to discovery

* Update behaviour

* Add keystore builder

* Remove keystore stuff from val client

* Add more tests, comments

* RPC handler to stable futures

* Update RPC to master libp2p

* Add more comments, test vectors

* Network service additions

* Progress on improving JSON validation

* More JSON verification

* Start moving JSON into own mod

* Remove old code

* Add more tests, reader/writers

* Tidy

* Move keystore into own file

* Move more logic into keystore file

* Tidy

* Tidy

* Fix the fallback transport construction (#1102)

* Allow for odd-character hex

* Correct warning

* Remove hashmap delay

* Compiling version of eth2-libp2p

* Update all crates versions

* Fix conversion function and add tests (#1113)

* Add more json missing field checks

* Use scrypt by default

* Tidy, address comments

* Test path and uuid in vectors

* Fix comment

* Add checks for kdf params

* Enforce empty kdf message

* Port validator_client to stable futures (#1114)

* Add PH & MS slot clock changes

* Account for genesis time

* Add progress on duties refactor

* Add simple is_aggregator bool to val subscription

* Start work on attestation_verification.rs

* Add progress on ObservedAttestations

* Progress with ObservedAttestations

* Fix tests

* Add observed attestations to the beacon chain

* Add attestation observation to processing code

* Add progress on attestation verification

* Add first draft of ObservedAttesters

* Add more tests

* Add observed attesters to beacon chain

* Add observers to attestation processing

* Add more attestation verification

* Create ObservedAggregators map

* Remove commented-out code

* Add observed aggregators into chain

* Add progress

* Finish adding features to attestation verification

* Ensure beacon chain compiles

* Link attn verification into chain

* Integrate new attn verification in chain

* Remove old attestation processing code

* Start trying to fix beacon_chain tests

* Split adding into pools into two functions

* Add aggregation to harness

* Get test harness working again

* Adjust the number of aggregators for test harness

* Fix edge-case in harness

* Integrate new attn processing in network

* Fix compile bug in validator_client

* Update validator API endpoints

* Fix aggreagation in test harness

* Fix enum thing

* Fix attestation observation bug:

* Patch failing API tests

* Start adding comments to attestation verification

* Remove unused attestation field

* Unify "is block known" logic

* Update comments

* Supress fork choice errors for network processing

* Add todos

* Tidy

* Add gossip attn tests

* Disallow test harness to produce old attns

* Comment out in-progress tests

* Partially address pruning tests

* Fix failing store test

* Add aggregate tests

* Add comments about which spec conditions we check

* Dont re-aggregate

* Split apart test harness attn production

* Fix compile error in network

* Make progress on commented-out test

* Fix skipping attestation test

* Add fork choice verification tests

* Tidy attn tests, remove dead code

* Remove some accidentally added code

* Fix clippy lint

* Rename test file

* Add block tests, add cheap block proposer check

* Rename block testing file

* Add observed_block_producers

* Tidy

* Switch around block signature verification

* Finish block testing

* Remove gossip from signature tests

* First pass of self review

* Fix deviation in spec

* Update test spec tags

* Start moving over to hashset

* Finish moving observed attesters to hashmap

* Move aggregation pool over to hashmap

* Make fc attn borrow again

* Fix rest_api compile error

* Fix missing comments

* Fix monster test

* Uncomment increasing slots test

* Address remaining comments

* Remove unsafe, use cfg test

* Remove cfg test flag

* Fix dodgy comment

* Revert "Update hashmap hashset to stable futures"

This reverts commit d432378a3cc5cd67fc29c0b15b96b886c1323554.

* Revert "Adds panic test to hashset delay"

This reverts commit 281502396fc5b90d9c421a309c2c056982c9525b.

* Ported attestation_service

* Ported duties_service

* Ported fork_service

* More ports

* Port block_service

* Minor fixes

* VC compiles

* Update TODOS

* Borrow self where possible

* Ignore aggregates that are already known.

* Unify aggregator modulo logic

* Fix typo in logs

* Refactor validator subscription logic

* Avoid reproducing selection proof

* Skip HTTP call if no subscriptions

* Rename DutyAndState -> DutyAndProof

* Tidy logs

* Print root as dbg

* Fix compile errors in tests

* Fix compile error in test

* Re-Fix attestation and duties service

* Minor fixes

Co-authored-by: Paul Hauner <paul@paulhauner.com>

* Expose json_keystore mod

* First commits on path derivation

* Progress with implementation

* More progress

* Passing intermediate test vectors

* Tidy, add comments

* Add DerivedKey structs

* Move key derivation into own crate

* Add zeroize structs

* Return error for empty seed

* Add tests

* Tidy

* First commits on path derivation

* Progress with implementation

* Move key derivation into own crate

* Start defining JSON wallet

* Add progress

* Split out encrypt/decrypt

* First commits on path derivation

* Progress with implementation

* More progress

* Passing intermediate test vectors

* Tidy, add comments

* Add DerivedKey structs

* Move key derivation into own crate

* Add zeroize structs

* Return error for empty seed

* Add tests

* Tidy

* Add progress

* Replace some password usage with slice

* First commits on path derivation

* Progress with implementation

* More progress

* Passing intermediate test vectors

* Tidy, add comments

* Add DerivedKey structs

* Move key derivation into own crate

* Add zeroize structs

* Return error for empty seed

* Add tests

* Tidy

* Add progress

* Expose PlainText struct

* First commits on path derivation

* Progress with implementation

* More progress

* Passing intermediate test vectors

* Tidy, add comments

* Add DerivedKey structs

* Move key derivation into own crate

* Add zeroize structs

* Return error for empty seed

* Add tests

* Tidy

* Add builder

* Expose consts, remove Password

* Minor progress

* Expose SALT_SIZE

* First compiling version

* Add test vectors

* Network crate update to stable futures

* Move dbg assert statement

* Port account_manager to stable futures (#1121)

* Port account_manager to stable futures

* Run async fns in tokio environment

* Port rest_api crate to stable futures (#1118)

* Port rest_api lib to stable futures

* Reduce tokio features

* Update notifier to stable futures

* Builder update

* Further updates

* Add mnemonic, tidy

* Convert self referential async functions

* Tidy

* Add testing

* Add first attempt at validator_dir

* Present pubkey field

* stable futures fixes (#1124)

* Fix eth1 update functions

* Fix genesis and client

* Fix beacon node lib

* Return appropriate runtimes from environment

* Fix test rig

* Refactor eth1 service update

* Upgrade simulator to stable futures

* Lighthouse compiles on stable futures

* Add first pass of wallet manager

* Progress with CLI

* Remove println debugging statement

* Tidy output

* Tidy 600 perms

* Update libp2p service, start rpc test upgrade

* Add validator creation flow

* Update network crate for new libp2p

* Start tidying, adding comments

* Update tokio::codec to futures_codec (#1128)

* Further work towards RPC corrections

* Correct http timeout and network service select

* Add wallet mgr testing

* Shift LockedWallet into own file

* Add comments to fs

* Start integration into VC

* Use tokio runtime for libp2p

* Revert "Update tokio::codec to futures_codec (#1128)"

This reverts commit e57aea924acf5cbabdcea18895ac07e38a425ed7.

* Upgrade RPC libp2p tests

* Upgrade secio fallback test

* Add lcli keypair upgrade command

* Upgrade gossipsub examples

* Clean up RPC protocol

* Test fixes (#1133)

* Correct websocket timeout and run on os thread

* Fix network test

* Add --secrets-dir to VC

* Remove --legacy-keys from VC

* Clean up PR

* Correct tokio tcp move attestation service tests

* Upgrade attestation service tests

* Fix sim

* Correct network test

* Correct genesis test

* Start docs

* Add progress for validator generation

* Tidy error messages

* Test corrections

* Log info when block is received

* Modify logs and update attester service events

* Stable futures: fixes to vc, eth1 and account manager (#1142)

* Add local testnet scripts

* Remove whiteblock script

* Rename local testnet script

* Move spawns onto handle

* Fix VC panic

* Initial fix to block production issue

* Tidy block producer fix

* Tidy further

* Add local testnet clean script

* Run cargo fmt

* Tidy duties service

* Tidy fork service

* Tidy ForkService

* Tidy AttestationService

* Tidy notifier

* Ensure await is not suppressed in eth1

* Ensure await is not suppressed in account_manager

* Use .ok() instead of .unwrap_or(())

* RPC decoding test for proto

* Update discv5 and eth2-libp2p deps

* Run cargo fmt

* Pre-build keystores for sim

* Fix lcli double runtime issue (#1144)

* Handle stream termination and dialing peer errors

* Correct peer_info variant types

* Add progress on new deposit flow

* Remove unnecessary warnings

* Handle subnet unsubscription removal and improve logigng

* Add logs around ping

* Upgrade discv5 and improve logging

* Handle peer connection status for multiple connections

* Improve network service logging

* Add more incomplete progress

* Improve logging around peer manager

* Upgrade swarm poll centralise peer management

* Identify clients on error

* Fix `remove_peer` in sync (#1150)

* remove_peer removes from all chains

* Remove logs

* Fix early return from loop

* Improved logging, fix panic

* Partially correct tests

* Add deposit command

* Remove old validator directory

* Start adding AM tests

* Stable futures: Vc sync (#1149)

* Improve syncing heuristic

* Add comments

* Use safer method for tolerance

* Fix tests

* Binary testing progress

* Progress with CLI tests

* Use constants for flags

* More account manager testing

* Improve CLI tests

* Move upgrade-legacy-keypairs into account man

* Use rayon for VC key generation

* Add comments to `validator_dir`

* Add testing to validator_dir

* Add fix to eth1-sim

* Check errors in eth1-sim

* Fix mutability issue

* Ensure password file ends in .pass

* Add more tests to wallet manager

* Tidy deposit

* Tidy account manager

* Tidy account manager

* Remove panic

* Generate keypairs earlier in sim

* Tidy eth1-sime

* Try to fix eth1 sim

* Address review comments

* Fix typo in CLI command

* Update docs

* Disable eth1 sim

* Remove eth1 sim completely

Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: pawanjay176 <pawandhananjay@gmail.com>
2020-05-18 19:01:45 +10:00
Michael Sproul
2d8e2dd7f5
Implement Slashing Protection (#1116)
* Implement slashing protection

Roll-up of #588 with some conflicts resolved

* WIP improvements

* Require slot uniqueness for blocks (rather than epochs)
* Native DB support for Slot and Epoch
* Simplify surrounding/surrounded-by queries

* Implement unified slashing protection database

A single SQL database saves on open file descriptors.

* Make slashing protection concurrency safe.

Revive tests, add parallel tests.

* Some simplifications

* Auto-registration, test clean-ups

* More tests, clean-ups, hardening

* Fix comments in BLS

* Optimise bulk validator registration

* Delete outdated tests

* Use bundled SQLite in slashing protection

* Auto-register validators in simulation

* Use real signing_root in slashing protection

* Update book for --auto-register

* Refine log messages and help flags

* Correct typo in Cargo.toml authors

* Fix merge conflicts

* Safer error handling in sqlite slot/epoch

* Address review comments

* Add attestation test mutating block root

Co-authored-by: pscott <scottpiriou@gmail.com>
2020-05-18 16:25:16 +10:00
Paul Hauner
90b3953dda
v0.1.2 (#1155)
* Version downgrade

* Start updating docs and version tags

Co-authored-by: Age Manning <Age@AgeManning.com>
2020-05-18 15:05:23 +10:00
Paul Hauner
d1864a8f01
Move docs over to Schelsi (#1129) 2020-05-12 14:21:21 +10:00
Paul Hauner
6c713d1e5f
Add note about building from testnet5 (#1094) 2020-05-04 08:04:24 +10:00
Age Manning
9e42a851e4
Adds peers and connected_peers to lighthouse http API (#1030) 2020-04-21 23:27:49 +10:00
Paul Hauner
7b86c9a08f
Update testnet tooling (#1001)
* Add progress on new deposits

* Add deposited command to account manager

* Remove old lcli::helpers mod

* Clean clap_utils

* Refactor lcli deposit contract commands to use IPC

* Make testnet optional for environment

* Use dbg formatting for deploy address

* Add command to generate bootnode enr

* Ensure lcli returns with 1 on error

* Ensure account manager returns 1 on error

* Disallow deposits to the zero address

* Update web3 in eth1 crate

* Ensure correct lighthouse dir is created

* Reduce deposit gas requirement

* Update cargo.lock

* Add progress on new deposits

* Add deposited command to account manager

* Remove old lcli::helpers mod

* Clean clap_utils

* Refactor lcli deposit contract commands to use IPC

* Add command to generate bootnode enr

* Ensure lcli returns with 1 on error

* Ensure account manager returns 1 on error

* Update web3 in eth1 crate

* Update Cargo.lock

* Move lcli out of main install script

* Change --limit to --at-least

* Change --datadir to --validator-dir

* Remove duplication in docs
2020-04-19 02:20:43 +00:00
Age Manning
62d7a71fd9
Merge latest master 2020-04-17 18:25:56 +10:00
Paul Hauner
1a3d1b3077
Fix issues with testnet dir, update docs (#992)
* Fix issues with testnet dir, update docs

* Remove "simple testnet" docs

* Tear out old "bn testnet" stuff

* Add back ClientGenesis::Interop

* Tidy

* Remove lighthouse-bootstrap module

* Fix bug with spec constant mismatch

* Ensure beacon-node.toml is written to correct dir

* Add -t alias for --testnet-dir

* Update book/src/local-testnets.md

Co-Authored-By: Age Manning <Age@AgeManning.com>

* Add --purge CLI flag

* Update purge docs

* Perform manual delete of files in purge

* Rename --purge to --purge-db

* Address Michael's comments

Co-authored-by: Age Manning <Age@AgeManning.com>
2020-04-17 17:49:29 +10:00
Age Manning
6edb4f655c
Update HTTP API docs (#1010) 2020-04-16 20:31:43 +10:00
Age Manning
d3dfd72f4d
Minor useability updates (#1005)
* Adds lcli to the makefile

* Update docs

* Set genesis fork version to default
2020-04-15 17:46:03 +10:00