Add a stack for prom node exporter and its dashboard in monitoring stack (#696)

* Add a stack for Prometheus node exporter

* Add node exporter dashboard to monitoring stack
This commit is contained in:
prathamesh0 2023-12-21 15:15:03 +05:30 committed by GitHub
parent 38622fb33c
commit a57b0cdd26
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 23876 additions and 6 deletions

View File

@ -0,0 +1,16 @@
version: '3.8'
services:
node-exporter:
image: prom/node-exporter:latest
restart: unless-stopped
command:
- '--path.rootfs=/host'
- '--collector.systemd'
- '--collector.processes'
network_mode: host
pid: host
ports:
- 9100
volumes:
- '/:/host:ro,rslave'

View File

@ -11,6 +11,13 @@ scrape_configs:
static_configs: static_configs:
- targets: ['localhost:9090'] - targets: ['localhost:9090']
- job_name: node
static_configs:
# Add node-exporter targets to be monitored below
# - targets: ['example-host:9100']
# labels:
# instance: 'my-host'
- job_name: 'blackbox' - job_name: 'blackbox'
scrape_interval: 10s scrape_interval: 10s
metrics_path: /probe metrics_path: /probe

View File

@ -2,9 +2,12 @@
* Instructions to setup and run a Prometheus server and Grafana dashboard * Instructions to setup and run a Prometheus server and Grafana dashboard
* Comes with the following built-in exporters / dashboards: * Comes with the following built-in exporters / dashboards:
* [Prometheus Blackbox Exporter](https://grafana.com/grafana/dashboards/7587-prometheus-blackbox-exporter/) - for tracking HTTP endpoints
* [NodeJS Application Dashboard](https://grafana.com/grafana/dashboards/11159-nodejs-application-dashboard/) - for default NodeJS metrics
* Chain Head Exporter - for tracking chain heads given external ETH RPC endpoints * Chain Head Exporter - for tracking chain heads given external ETH RPC endpoints
* Watchers dashboard
* [Prometheus Blackbox](https://grafana.com/grafana/dashboards/7587-prometheus-blackbox-exporter/) - for tracking HTTP endpoints
* [NodeJS Application Dashboard](https://grafana.com/grafana/dashboards/11159-nodejs-application-dashboard/) - for default NodeJS metrics
* [PostgreSQL Database](https://grafana.com/grafana/dashboards/9628-postgresql-database/) - for monitoring Postgres dbs
* [Node Exporter Full](https://grafana.com/grafana/dashboards/1860-node-exporter-full/) - for monitoring system metrics
* See [monitoring-watchers.md](./monitoring-watchers.md) for an example usage of the stack with pre-configured dashboards for watchers * See [monitoring-watchers.md](./monitoring-watchers.md) for an example usage of the stack with pre-configured dashboards for watchers
## Setup ## Setup
@ -72,7 +75,20 @@ laconic-so --stack monitoring deploy create --spec-file monitoring-spec.yml --de
- targets: ['<METRICS_ENDPOINT_HOST>:<METRICS_ENDPOINT_PORT>'] - targets: ['<METRICS_ENDPOINT_HOST>:<METRICS_ENDPOINT_PORT>']
``` ```
* Blackbox: update the `blackbox` job to add any endpoints to be monitored on the Blackbox dashboard: * Node exporter: update the `node` job to add any node-exporter targets to be monitored:
```yml
...
- job_name: 'node'
...
static_configs:
# Add node-exporter targets to be monitored below
- targets: [example-host:9100]
labels:
instance: 'my-host'
```
* Blackbox (in-stack exporter): update the `blackbox` job to add any endpoints to be monitored on the Blackbox dashboard:
```yml ```yml
... ...
@ -85,7 +101,7 @@ laconic-so --stack monitoring deploy create --spec-file monitoring-spec.yml --de
- <HTTP_ENDPOINT_2> - <HTTP_ENDPOINT_2>
``` ```
* Postgres: * Postgres (in-stack exporter):
* Update the `postgres` job to add Postgres db targets to be monitored: * Update the `postgres` job to add Postgres db targets to be monitored:
```yml ```yml
@ -111,6 +127,8 @@ Place the dashboard json files in grafana dashboards config directory (`monitori
Set the following env variables in the deployment env config file (`monitoring-deployment/config.env`): Set the following env variables in the deployment env config file (`monitoring-deployment/config.env`):
```bash ```bash
# For chain-head exporter
# External ETH RPC endpoint (ethereum) # External ETH RPC endpoint (ethereum)
# (Optional, default: https://mainnet.infura.io/v3) # (Optional, default: https://mainnet.infura.io/v3)
CERC_ETH_RPC_ENDPOINT= CERC_ETH_RPC_ENDPOINT=

View File

@ -1,6 +1,6 @@
# Monitoring Watchers # Monitoring Watchers
Instructions to setup and run monitoring stack with pre-configured watcher dashboards Instructions to setup and run monitoring stack with configured Watchers dashboard
## Create a deployment ## Create a deployment
@ -98,7 +98,7 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers
chain: 'filecoin' chain: 'filecoin'
``` ```
Add scrape config as done above for any additional watcher to add it to the watcher dashboard. Add scrape config as done above for any additional watcher to add it to the Watchers dashboard.
### Env ### Env

View File

@ -0,0 +1,17 @@
# node-exporter
## Start the stack
```bash
laconic-so --stack node-exporter deploy up
```
* The host node's metrics can be accessed at `http://localhost:9100/metrics`
## Clean up
Stop the node-exporter running in background:
```bash
laconic-so --stack node-exporter deploy down
```

View File

@ -0,0 +1,6 @@
version: "0.1"
name: node-exporter
repos:
containers:
pods:
- node-exporter