First RESTful HTTP API (#399)
* Added generated code for REST API.
- Created a new crate rest_api, which will adapt the openapi generated code to Lighthouse
- Committed automatically generated code from openapi-generator-cli (via docker). Should hopfully not have to modify this at all, and do all changes in the rest_api crate.
* Removed openapi generated code, because it was the rust client, not the rust server.
* Added the correct rust-server code, automatically generated from openapi.
* Added generated code for REST API.
- Created a new crate rest_api, which will adapt the openapi generated code to Lighthouse
- Committed automatically generated code from openapi-generator-cli (via docker). Should hopfully not have to modify this at all, and do all changes in the rest_api crate.
* Removed openapi generated code, because it was the rust client, not the rust server.
* Added the correct rust-server code, automatically generated from openapi.
* Included REST API in configuratuion.
- Started adding the rest_api into the beacon node's dependencies.
- Set up configuration file for rest_api and integrated into main client config
- Added CLI flags for REST API.
* Futher work on REST API.
- Adding the dependencies to rest_api crate
- Created a skeleton BeaconNodeService, which will handle /node requests.
- Started the rest_api server definition, with the high level request handling logic.
* Added generated code for REST API.
- Created a new crate rest_api, which will adapt the openapi generated code to Lighthouse
- Committed automatically generated code from openapi-generator-cli (via docker). Should hopfully not have to modify this at all, and do all changes in the rest_api crate.
* Removed openapi generated code, because it was the rust client, not the rust server.
* Added the correct rust-server code, automatically generated from openapi.
* Included REST API in configuratuion.
- Started adding the rest_api into the beacon node's dependencies.
- Set up configuration file for rest_api and integrated into main client config
- Added CLI flags for REST API.
* Futher work on REST API.
- Adding the dependencies to rest_api crate
- Created a skeleton BeaconNodeService, which will handle /node requests.
- Started the rest_api server definition, with the high level request handling logic.
* WIP: Restructured REST API to use hyper_router and separate services.
* WIP: Fixing rust for REST API
* WIP: Fixed up many bugs in trying to get router to compile.
* WIP: Got the beacon_node to compile with the REST changes
* Basic API works!
- Changed CLI flags from rest-api* to api*
- Fixed port cli flag
- Tested, works over HTTP
* WIP: Moved things around so that we can get state inside the handlers.
* WIP: Significant API updates.
- Started writing a macro for getting the handler functions.
- Added the BeaconChain into the type map, gives stateful access to the beacon state.
- Created new generic error types (haven't figured out yet), to reduce code duplication.
- Moved common stuff into lib.rs
* WIP: Factored macros, defined API result and error.
- did more logging when creating HTTP responses
- Tried moving stuff into macros, but can't get macros in macros to compile.
- Pulled out a lot of placeholder code.
* Fixed macros so that things compile.
* Cleaned up code.
- Removed unused imports
- Removed comments
- Addressed all compiler warnings.
- Ran cargo fmt.
* Removed auto-generated OpenAPI code.
* Addressed Paul's suggestions.
- Fixed spelling mistake
- Moved the simple macros into functions, since it doesn't make sense for them to be macros.
- Removed redundant code & inclusions.
* Removed redundant validate_request function.
* Included graceful shutdown in Hyper server.
* Fixing the dropped exit_signal, which prevented the API from starting.
* Wrapped the exit signal, to get an API shutdown log line.
2019-07-31 08:29:41 +00:00
|
|
|
[package]
|
2020-09-29 03:46:54 +00:00
|
|
|
name = "http_api"
|
|
|
|
|
version = "0.1.0"
|
|
|
|
|
authors = ["Paul Hauner <paul@paulhauner.com>"]
|
2022-02-25 00:10:17 +00:00
|
|
|
edition = "2021"
|
2021-08-06 00:47:31 +00:00
|
|
|
autotests = false # using a single test binary compiles faster
|
First RESTful HTTP API (#399)
* Added generated code for REST API.
- Created a new crate rest_api, which will adapt the openapi generated code to Lighthouse
- Committed automatically generated code from openapi-generator-cli (via docker). Should hopfully not have to modify this at all, and do all changes in the rest_api crate.
* Removed openapi generated code, because it was the rust client, not the rust server.
* Added the correct rust-server code, automatically generated from openapi.
* Added generated code for REST API.
- Created a new crate rest_api, which will adapt the openapi generated code to Lighthouse
- Committed automatically generated code from openapi-generator-cli (via docker). Should hopfully not have to modify this at all, and do all changes in the rest_api crate.
* Removed openapi generated code, because it was the rust client, not the rust server.
* Added the correct rust-server code, automatically generated from openapi.
* Included REST API in configuratuion.
- Started adding the rest_api into the beacon node's dependencies.
- Set up configuration file for rest_api and integrated into main client config
- Added CLI flags for REST API.
* Futher work on REST API.
- Adding the dependencies to rest_api crate
- Created a skeleton BeaconNodeService, which will handle /node requests.
- Started the rest_api server definition, with the high level request handling logic.
* Added generated code for REST API.
- Created a new crate rest_api, which will adapt the openapi generated code to Lighthouse
- Committed automatically generated code from openapi-generator-cli (via docker). Should hopfully not have to modify this at all, and do all changes in the rest_api crate.
* Removed openapi generated code, because it was the rust client, not the rust server.
* Added the correct rust-server code, automatically generated from openapi.
* Included REST API in configuratuion.
- Started adding the rest_api into the beacon node's dependencies.
- Set up configuration file for rest_api and integrated into main client config
- Added CLI flags for REST API.
* Futher work on REST API.
- Adding the dependencies to rest_api crate
- Created a skeleton BeaconNodeService, which will handle /node requests.
- Started the rest_api server definition, with the high level request handling logic.
* WIP: Restructured REST API to use hyper_router and separate services.
* WIP: Fixing rust for REST API
* WIP: Fixed up many bugs in trying to get router to compile.
* WIP: Got the beacon_node to compile with the REST changes
* Basic API works!
- Changed CLI flags from rest-api* to api*
- Fixed port cli flag
- Tested, works over HTTP
* WIP: Moved things around so that we can get state inside the handlers.
* WIP: Significant API updates.
- Started writing a macro for getting the handler functions.
- Added the BeaconChain into the type map, gives stateful access to the beacon state.
- Created new generic error types (haven't figured out yet), to reduce code duplication.
- Moved common stuff into lib.rs
* WIP: Factored macros, defined API result and error.
- did more logging when creating HTTP responses
- Tried moving stuff into macros, but can't get macros in macros to compile.
- Pulled out a lot of placeholder code.
* Fixed macros so that things compile.
* Cleaned up code.
- Removed unused imports
- Removed comments
- Addressed all compiler warnings.
- Ran cargo fmt.
* Removed auto-generated OpenAPI code.
* Addressed Paul's suggestions.
- Fixed spelling mistake
- Moved the simple macros into functions, since it doesn't make sense for them to be macros.
- Removed redundant code & inclusions.
* Removed redundant validate_request function.
* Included graceful shutdown in Hyper server.
* Fixing the dropped exit_signal, which prevented the API from starting.
* Wrapped the exit signal, to get an API shutdown log line.
2019-07-31 08:29:41 +00:00
|
|
|
|
|
|
|
|
[dependencies]
|
2021-11-18 05:08:42 +00:00
|
|
|
warp = { version = "0.3.2", features = ["tls"] }
|
2020-10-05 08:22:19 +00:00
|
|
|
serde = { version = "1.0.116", features = ["derive"] }
|
2021-11-18 05:08:42 +00:00
|
|
|
tokio = { version = "1.14.0", features = ["macros","sync"] }
|
2021-03-01 01:58:05 +00:00
|
|
|
tokio-stream = { version = "0.1.3", features = ["sync"] }
|
2020-09-29 03:46:54 +00:00
|
|
|
types = { path = "../../consensus/types" }
|
|
|
|
|
hex = "0.4.2"
|
First RESTful HTTP API (#399)
* Added generated code for REST API.
- Created a new crate rest_api, which will adapt the openapi generated code to Lighthouse
- Committed automatically generated code from openapi-generator-cli (via docker). Should hopfully not have to modify this at all, and do all changes in the rest_api crate.
* Removed openapi generated code, because it was the rust client, not the rust server.
* Added the correct rust-server code, automatically generated from openapi.
* Added generated code for REST API.
- Created a new crate rest_api, which will adapt the openapi generated code to Lighthouse
- Committed automatically generated code from openapi-generator-cli (via docker). Should hopfully not have to modify this at all, and do all changes in the rest_api crate.
* Removed openapi generated code, because it was the rust client, not the rust server.
* Added the correct rust-server code, automatically generated from openapi.
* Included REST API in configuratuion.
- Started adding the rest_api into the beacon node's dependencies.
- Set up configuration file for rest_api and integrated into main client config
- Added CLI flags for REST API.
* Futher work on REST API.
- Adding the dependencies to rest_api crate
- Created a skeleton BeaconNodeService, which will handle /node requests.
- Started the rest_api server definition, with the high level request handling logic.
* Added generated code for REST API.
- Created a new crate rest_api, which will adapt the openapi generated code to Lighthouse
- Committed automatically generated code from openapi-generator-cli (via docker). Should hopfully not have to modify this at all, and do all changes in the rest_api crate.
* Removed openapi generated code, because it was the rust client, not the rust server.
* Added the correct rust-server code, automatically generated from openapi.
* Included REST API in configuratuion.
- Started adding the rest_api into the beacon node's dependencies.
- Set up configuration file for rest_api and integrated into main client config
- Added CLI flags for REST API.
* Futher work on REST API.
- Adding the dependencies to rest_api crate
- Created a skeleton BeaconNodeService, which will handle /node requests.
- Started the rest_api server definition, with the high level request handling logic.
* WIP: Restructured REST API to use hyper_router and separate services.
* WIP: Fixing rust for REST API
* WIP: Fixed up many bugs in trying to get router to compile.
* WIP: Got the beacon_node to compile with the REST changes
* Basic API works!
- Changed CLI flags from rest-api* to api*
- Fixed port cli flag
- Tested, works over HTTP
* WIP: Moved things around so that we can get state inside the handlers.
* WIP: Significant API updates.
- Started writing a macro for getting the handler functions.
- Added the BeaconChain into the type map, gives stateful access to the beacon state.
- Created new generic error types (haven't figured out yet), to reduce code duplication.
- Moved common stuff into lib.rs
* WIP: Factored macros, defined API result and error.
- did more logging when creating HTTP responses
- Tried moving stuff into macros, but can't get macros in macros to compile.
- Pulled out a lot of placeholder code.
* Fixed macros so that things compile.
* Cleaned up code.
- Removed unused imports
- Removed comments
- Addressed all compiler warnings.
- Ran cargo fmt.
* Removed auto-generated OpenAPI code.
* Addressed Paul's suggestions.
- Fixed spelling mistake
- Moved the simple macros into functions, since it doesn't make sense for them to be macros.
- Removed redundant code & inclusions.
* Removed redundant validate_request function.
* Included graceful shutdown in Hyper server.
* Fixing the dropped exit_signal, which prevented the API from starting.
* Wrapped the exit signal, to get an API shutdown log line.
2019-07-31 08:29:41 +00:00
|
|
|
beacon_chain = { path = "../beacon_chain" }
|
2020-09-29 03:46:54 +00:00
|
|
|
eth2 = { path = "../../common/eth2", features = ["lighthouse"] }
|
|
|
|
|
slog = "2.5.2"
|
2019-08-14 08:23:26 +00:00
|
|
|
network = { path = "../network" }
|
Rename eth2_libp2p to lighthouse_network (#2702)
## Description
The `eth2_libp2p` crate was originally named and designed to incorporate a simple libp2p integration into lighthouse. Since its origins the crates purpose has expanded dramatically. It now houses a lot more sophistication that is specific to lighthouse and no longer just a libp2p integration.
As of this writing it currently houses the following high-level lighthouse-specific logic:
- Lighthouse's implementation of the eth2 RPC protocol and specific encodings/decodings
- Integration and handling of ENRs with respect to libp2p and eth2
- Lighthouse's discovery logic, its integration with discv5 and logic about searching and handling peers.
- Lighthouse's peer manager - This is a large module handling various aspects of Lighthouse's network, such as peer scoring, handling pings and metadata, connection maintenance and recording, etc.
- Lighthouse's peer database - This is a collection of information stored for each individual peer which is specific to lighthouse. We store connection state, sync state, last seen ips and scores etc. The data stored for each peer is designed for various elements of the lighthouse code base such as syncing and the http api.
- Gossipsub scoring - This stores a collection of gossipsub 1.1 scoring mechanisms that are continuously analyssed and updated based on the ethereum 2 networks and how Lighthouse performs on these networks.
- Lighthouse specific types for managing gossipsub topics, sync status and ENR fields
- Lighthouse's network HTTP API metrics - A collection of metrics for lighthouse network monitoring
- Lighthouse's custom configuration of all networking protocols, RPC, gossipsub, discovery, identify and libp2p.
Therefore it makes sense to rename the crate to be more akin to its current purposes, simply that it manages the majority of Lighthouse's network stack. This PR renames this crate to `lighthouse_network`
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2021-10-19 00:30:39 +00:00
|
|
|
lighthouse_network = { path = "../lighthouse_network" }
|
2020-09-29 03:46:54 +00:00
|
|
|
eth1 = { path = "../eth1" }
|
2020-05-18 11:24:23 +00:00
|
|
|
state_processing = { path = "../../consensus/state_processing" }
|
2020-09-29 03:46:54 +00:00
|
|
|
lighthouse_version = { path = "../../common/lighthouse_version" }
|
2020-05-18 11:24:23 +00:00
|
|
|
lighthouse_metrics = { path = "../../common/lighthouse_metrics" }
|
2020-09-29 03:46:54 +00:00
|
|
|
lazy_static = "1.4.0"
|
|
|
|
|
warp_utils = { path = "../../common/warp_utils" }
|
2020-05-18 11:24:23 +00:00
|
|
|
slot_clock = { path = "../../common/slot_clock" }
|
2021-11-29 03:57:54 +00:00
|
|
|
eth2_ssz = "0.4.1"
|
2020-12-07 08:20:33 +00:00
|
|
|
bs58 = "0.4.0"
|
2020-12-04 00:18:58 +00:00
|
|
|
futures = "0.3.8"
|
2022-03-31 07:52:23 +00:00
|
|
|
execution_layer = {path = "../execution_layer"}
|
2022-04-04 00:26:16 +00:00
|
|
|
parking_lot = "0.12.0"
|
2022-02-18 05:32:00 +00:00
|
|
|
safe_arith = {path = "../../consensus/safe_arith"}
|
2022-05-16 08:35:59 +00:00
|
|
|
task_executor = { path = "../../common/task_executor" }
|
2022-06-29 04:50:37 +00:00
|
|
|
lru = "0.7.7"
|
2022-07-30 00:22:37 +00:00
|
|
|
tree_hash = "0.4.1"
|
2022-11-15 05:21:26 +00:00
|
|
|
sysinfo = "0.26.5"
|
|
|
|
|
system_health = { path = "../../common/system_health" }
|
|
|
|
|
directory = { path = "../../common/directory" }
|
Cache validator balances and allow them to be served over the HTTP API (#3863)
## Issue Addressed
#3804
## Proposed Changes
- Add `total_balance` to the validator monitor and adjust the number of historical epochs which are cached.
- Allow certain values in the cache to be served out via the HTTP API without requiring a state read.
## Usage
```
curl -X POST "http://localhost:5052/lighthouse/ui/validator_info" -d '{"indices": [0]}' -H "Content-Type: application/json" | jq
```
```
{
"data": {
"validators": {
"0": {
"info": [
{
"epoch": 172981,
"total_balance": 36566388519
},
...
{
"epoch": 172990,
"total_balance": 36566496513
}
]
},
"1": {
"info": [
{
"epoch": 172981,
"total_balance": 36355797968
},
...
{
"epoch": 172990,
"total_balance": 36355905962
}
]
}
}
}
}
```
## Additional Info
This requires no historical states to operate which mean it will still function on the freshly checkpoint synced node, however because of this, the values will populate each epoch (up to a maximum of 10 entries).
Another benefit of this method, is that we can easily cache any other values which would normally require a state read and serve them via the same endpoint. However, we would need be cautious about not overly increasing block processing time by caching values from complex computations.
This also caches some of the validator metrics directly, rather than pulling them from the Prometheus metrics when the API is called. This means when the validator count exceeds the individual monitor threshold, the cached values will still be available.
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-02-21 20:54:55 +00:00
|
|
|
eth2_serde_utils = "0.1.1"
|
2023-02-07 06:13:49 +00:00
|
|
|
operation_pool = { path = "../operation_pool" }
|
2022-02-21 23:21:02 +00:00
|
|
|
|
2019-11-25 04:48:24 +00:00
|
|
|
[dev-dependencies]
|
2020-09-29 03:46:54 +00:00
|
|
|
store = { path = "../store" }
|
|
|
|
|
environment = { path = "../../lighthouse/environment" }
|
2021-05-04 01:59:51 +00:00
|
|
|
sensitive_url = { path = "../../common/sensitive_url" }
|
Run fork choice before block proposal (#3168)
## Issue Addressed
Upcoming spec change https://github.com/ethereum/consensus-specs/pull/2878
## Proposed Changes
1. Run fork choice at the start of every slot, and wait for this run to complete before proposing a block.
2. As an optimisation, also run fork choice 3/4 of the way through the slot (at 9s), _dequeueing attestations for the next slot_.
3. Remove the fork choice run from the state advance timer that occurred before advancing the state.
## Additional Info
### Block Proposal Accuracy
This change makes us more likely to propose on top of the correct head in the presence of re-orgs with proposer boost in play. The main scenario that this change is designed to address is described in the linked spec issue.
### Attestation Accuracy
This change _also_ makes us more likely to attest to the correct head. Currently in the case of a skipped slot at `slot` we only run fork choice 9s into `slot - 1`. This means the attestations from `slot - 1` aren't taken into consideration, and any boost applied to the block from `slot - 1` is not removed (it should be). In the language of the linked spec issue, this means we are liable to attest to C, even when the majority voting weight has already caused a re-org to B.
### Why remove the call before the state advance?
If we've run fork choice at the start of the slot then it has already dequeued all the attestations from the previous slot, which are the only ones eligible to influence the head in the current slot. Running fork choice again is unnecessary (unless we run it for the next slot and try to pre-empt a re-org, but I don't currently think this is a great idea).
### Performance
Based on Prater testing this adds about 5-25ms of runtime to block proposal times, which are 500-1000ms on average (and spike to 5s+ sometimes due to state handling issues :cry: ). I believe this is a small enough penalty to enable it by default, with the option to disable it via the new flag `--fork-choice-before-proposal-timeout 0`. Upcoming work on block packing and state representation will also reduce block production times in general, while removing the spikes.
### Implementation
Fork choice gets invoked at the start of the slot via the `per_slot_task` function called from the slot timer. It then uses a condition variable to signal to block production that fork choice has been updated. This is a bit funky, but it seems to work. One downside of the timer-based approach is that it doesn't happen automatically in most of the tests. The test added by this PR has to trigger the run manually.
2022-05-20 05:02:11 +00:00
|
|
|
logging = { path = "../../common/logging" }
|
2022-06-30 00:49:21 +00:00
|
|
|
serde_json = "1.0.58"
|
2022-07-25 08:23:00 +00:00
|
|
|
proto_array = { path = "../../consensus/proto_array" }
|
2022-07-30 00:22:37 +00:00
|
|
|
unused_port = {path = "../../common/unused_port"}
|
2023-01-25 04:47:07 +00:00
|
|
|
genesis = { path = "../genesis" }
|
2021-08-06 00:47:31 +00:00
|
|
|
|
|
|
|
|
[[test]]
|
|
|
|
|
name = "bn_http_api_tests"
|
|
|
|
|
path = "tests/main.rs"
|
