lighthouse/book/src/validator-monitoring.md

Validator Monitoring

Lighthouse allows for fine-grained monitoring of specific validators using the "validator monitor". Generally users will want to use this function to track their own validators, however, it can be used for any validator, regardless of who controls it.

Monitoring is in the Beacon Node

Lighthouse performs validator monitoring in the Beacon Node (BN) instead of the Validator Client (VC). This is contrary to what some users may expect, but it has several benefits:

1. It keeps the VC simple. The VC handles cryptographic signing and the developers believe it should be doing as little additional work as possible.
2. The BN has a better knowledge of the chain and network. Communicating all this information to the VC is impractical, we can provide more information when monitoring with the BN.
3. It is more flexible:
   • Users can use a local BN to observe some validators running in a remote location.
   • Users can monitor validators that are not their own.

How to Enable Monitoring

The validator monitor is always enabled in Lighthouse, but it might not have any enrolled validators. There are two methods for a validator to be enrolled for additional monitoring; automatic and manual.

Automatic

When the --validator-monitor-auto flag is supplied, any validator which uses the beacon_committee_subscriptions API endpoint will be enrolled for additional monitoring. All active validators will use this endpoint each epoch, so you can expect it to detect all local and active validators within several minutes after start up.

Example

lighthouse bn --staking --validator-monitor-auto

Manual

The --validator-monitor-pubkeys flag can be used to specify validator public keys for monitoring. This is useful when monitoring validators that are not directly attached to this BN.

Note: when monitoring validators that aren't connected to this BN, supply the --subscribe-all-subnets --import-all-attestations flags to ensure the BN has a full view of the network. This is not strictly necessary, though.

Example

Monitor the mainnet validators at indices 0 and 1:

lighthouse bn --validator-monitor-pubkeys 0x933ad9491b62059dd065b560d256d8957a8c402cc6e8d8ee7290ae11e8f7329267a8811c397529dac52ae1342ba58c95,0xa1d1ad0714035353258038e964ae9675dc0252ee22cea896825c01458e1807bfad2f9969338798548d9858a571f7425c

Observing Monitoring

Enrolling a validator for additional monitoring results in:

• Additional logs to be printed during BN operation.
• Additional Prometheus metrics from the BN.

Logging

Lighthouse will create logs for the following events for each monitored validator:

• A block from the validator is observed.
• An unaggregated attestation from the validator is observed.
• An unaggregated attestation from the validator is included in an aggregate.
• An unaggregated attestation from the validator is included in a block.
• An aggregated attestation from the validator is observed.
• An exit for the validator is observed.
• A slashing (proposer or attester) is observed which implicates that validator.

Example

Jan 18 11:50:03.896 INFO Unaggregated attestation                validator: 0, src: gossip, slot: 342248, epoch: 10695, delay_ms: 891, index: 12, head: 0x5f9d603c04b5489bf2de3708569226fd9428eb40a89c75945e344d06c7f4f86a, service: beacon

Jan 18 11:32:55.196 INFO Attestation included in aggregate       validator: 0, src: gossip, slot: 342162, epoch: 10692, delay_ms: 2193, index: 10, head: 0x9be04ecd04bf82952dad5d12c62e532fd13a8d42afb2e6ee98edaf05fc7f9f30, service: beacon

Jan 18 11:21:09.808 INFO Attestation included in block           validator: 1, slot: 342102, epoch: 10690, inclusion_lag: 0 slot(s), index: 7, head: 0x422bcd14839e389f797fd38b01e31995f91bcaea3d5d56457fc6aac76909ebac, service: beacon

Metrics

The ValidatorMonitor dashboard contains all/most of the metrics exposed via the validator monitor.