Add a stack for prom node exporter and its dashboard in monitoring stack (#696 )

* Add a stack for Prometheus node exporter * Add node exporter dashboard to monitoring stack
[WIP] Use templating for watcher dashboard and add Postgres exporter (#695 )
2023-12-21 15:15:03 +05:30 · 2023-12-21 13:41:36 +05:30
22 changed files with 29087 additions and 20084 deletions
@@ -0,0 +1,16 @@
+version: '3.8'
+
+services:
+  node-exporter:
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    command:
+      - '--path.rootfs=/host'
+      - '--collector.systemd'
+      - '--collector.processes'
+    network_mode: host
+    pid: host
+    ports:
+      - 9100
+    volumes:
+      - '/:/host:ro,rslave'
@@ -43,5 +43,15 @@ services:
    extra_hosts:
      - "host.docker.internal:host-gateway"

+  postgres-exporter:
+    image: quay.io/prometheuscommunity/postgres-exporter
+    restart: always
+    volumes:
+      - ../config/monitoring/postgres-exporter.yml:/postgres_exporter.yml
+    ports:
+      - '9187'
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
 volumes:
  prometheus_data:
@@ -0,0 +1,8 @@
+auth_modules:
+  foo:
+    type: userpass
+    userpass:
+      username: username
+      password: password
+    options:
+      sslmode: disable
@@ -11,6 +11,13 @@ scrape_configs:
    static_configs:
      - targets: ['localhost:9090']

+  - job_name: node
+    static_configs:
+      # Add node-exporter targets to be monitored below
+      # - targets: ['example-host:9100']
+      #   labels:
+      #     instance: 'my-host'
+
  - job_name: 'blackbox'
    scrape_interval: 10s
    metrics_path: /probe
@@ -40,3 +47,21 @@ scrape_configs:
    static_configs:
      - targets: ['chain-head-exporter:5000']

+  - job_name: 'postgres'
+    scrape_interval: 30s
+    scrape_timeout: 30s
+    static_configs:
+      # Add DB targets below
+      # - targets: [example-server:5432]
+        # labels:
+          # instance: 'example-label'
+    metrics_path: /probe
+    params:
+      auth_module: [foo]
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: postgres-exporter:9187
@@ -2,9 +2,12 @@

 * Instructions to setup and run a Prometheus server and Grafana dashboard
 * Comes with the following built-in exporters / dashboards:
-  * [Prometheus Blackbox Exporter](https://grafana.com/grafana/dashboards/7587-prometheus-blackbox-exporter/) - for tracking HTTP endpoints
-  * [NodeJS Application Dashboard](https://grafana.com/grafana/dashboards/11159-nodejs-application-dashboard/) - for default NodeJS metrics
  * Chain Head Exporter - for tracking chain heads given external ETH RPC endpoints
+  * Watchers dashboard
+  * [Prometheus Blackbox](https://grafana.com/grafana/dashboards/7587-prometheus-blackbox-exporter/) - for tracking HTTP endpoints
+  * [NodeJS Application Dashboard](https://grafana.com/grafana/dashboards/11159-nodejs-application-dashboard/) - for default NodeJS metrics
+  * [PostgreSQL Database](https://grafana.com/grafana/dashboards/9628-postgresql-database/) - for monitoring Postgres dbs
+  * [Node Exporter Full](https://grafana.com/grafana/dashboards/1860-node-exporter-full/) - for monitoring system metrics
 * See [monitoring-watchers.md](./monitoring-watchers.md) for an example usage of the stack with pre-configured dashboards for watchers

 ## Setup
@@ -72,7 +75,20 @@ laconic-so --stack monitoring deploy create --spec-file monitoring-spec.yml --de
      - targets: ['<METRICS_ENDPOINT_HOST>:<METRICS_ENDPOINT_PORT>']
  ```

-* Also update the `blackbox` job to add any endpoints to be monitored on the Blackbox dashboard:
+* Node exporter: update the `node` job to add any node-exporter targets to be monitored:
+
+  ```yml
+  ...
+  - job_name: 'node'
+    ...
+    static_configs:
+      # Add node-exporter targets to be monitored below
+      - targets: [example-host:9100]
+        labels:
+          instance: 'my-host'
+  ```
+
+* Blackbox (in-stack exporter): update the `blackbox` job to add any endpoints to be monitored on the Blackbox dashboard:

  ```yml
  ...
@@ -85,6 +101,21 @@ laconic-so --stack monitoring deploy create --spec-file monitoring-spec.yml --de
        - <HTTP_ENDPOINT_2>
  ```

+* Postgres (in-stack exporter):
+  * Update the `postgres` job to add Postgres db targets to be monitored:
+
+    ```yml
+    ...
+    - job_name: 'postgres'
+      ...
+      static_configs:
+        # Add DB targets below
+        - targets: [example-server:5432]
+          labels:
+            instance: 'example-db'
+    ```
+  * Add database credentials to be used in `auth_modules` in the postgres-exporter config file (`monitoring-deployment/config/monitoring/postgres-exporter.yml`)
+
 Note: Use `host.docker.internal` as host to access ports on the host machine

 ### Grafana Config
@@ -96,6 +127,8 @@ Place the dashboard json files in grafana dashboards config directory (`monitori
 Set the following env variables in the deployment env config file (`monitoring-deployment/config.env`):

  ```bash
+  # For chain-head exporter
+
  # External ETH RPC endpoint (ethereum)
  # (Optional, default: https://mainnet.infura.io/v3)
  CERC_ETH_RPC_ENDPOINT=
@@ -1,6 +1,6 @@
 # Monitoring Watchers

-Instructions to setup and run monitoring stack with pre-configured watcher dashboards
+Instructions to setup and run monitoring stack with configured Watchers dashboard

 ## Create a deployment

@@ -35,8 +35,6 @@ laconic-so --stack monitoring deploy create --spec-file monitoring-watchers-spec

 ## Configure

-### Prometheus Config
-
 Add the following scrape configs to prometheus config file (`monitoring-watchers-deployment/config/monitoring/prometheus/prometheus.yml`) in the deployment folder:

  ```yml
@@ -55,27 +53,35 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers
      - targets: ['AZIMUTH_WATCHER_HOST:AZIMUTH_WATCHER_PORT']
        labels:
          instance: 'azimuth'
+          chain: 'ethereum'
      - targets: ['CENSURES_WATCHER_HOST:CENSURES_WATCHER_PORT']
        labels:
          instance: 'censures'
+          chain: 'ethereum'
      - targets: ['CLAIMS_WATCHER_HOST:CLAIMS_WATCHER_PORT']
        labels:
          instance: 'claims'
+          chain: 'ethereum'
      - targets: ['CONDITIONAL_STAR_RELEASE_WATCHER_HOST:CONDITIONAL_STAR_RELEASE_WATCHER_PORT']
        labels:
          instance: 'conditional_star_release'
+          chain: 'ethereum'
      - targets: ['DELEGATED_SENDING_WATCHER_HOST:DELEGATED_SENDING_WATCHER_PORT']
        labels:
-          instance: 'delegated_sending_watcher'
+          instance: 'delegated_sending'
+          chain: 'ethereum'
      - targets: ['ECLIPTIC_WATCHER_HOST:ECLIPTIC_WATCHER_PORT']
        labels:
          instance: 'ecliptic'
+          chain: 'ethereum'
      - targets: ['LINEAR_STAR_WATCHER_HOST:LINEAR_STAR_WATCHER_PORT']
        labels:
          instance: 'linear_star_release'
+          chain: 'ethereum'
      - targets: ['POLLS_WATCHER_HOST:POLLS_WATCHER_PORT']
        labels:
          instance: 'polls'
+          chain: 'ethereum'

  - job_name: sushi
    scrape_interval: 20s
@@ -85,18 +91,14 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers
      - targets: ['SUSHISWAP_WATCHER_HOST:SUSHISWAP_WATCHER_PORT']
        labels:
          instance: 'sushiswap'
+          chain: 'filecoin'
      - targets: ['MERKLE_SUSHISWAP_WATCHER_HOST:MERKLE_SUSHISWAP_WATCHER_PORT']
        labels:
          instance: 'merkl_sushiswap'
+          chain: 'filecoin'
  ```

-### Grafana Config
-
-In the deployment folder, copy over the pre-configured watcher dashboard JSON files to grafana dashboards config directory:
-
-```bash
-cp -r monitoring-watchers-deployment/config/monitoring/grafana/watcher-dashboards/* monitoring-watchers-deployment/config/monitoring/grafana/dashboards/
-```
+Add scrape config as done above for any additional watcher to add it to the Watchers dashboard.

 ### Env

@@ -0,0 +1,17 @@
+# node-exporter
+
+## Start the stack
+
+```bash
+laconic-so --stack node-exporter deploy up
+```
+
+* The host node's metrics can be accessed at `http://localhost:9100/metrics`
+
+## Clean up
+
+Stop the node-exporter running in background:
+
+```bash
+laconic-so --stack node-exporter deploy down
+```
@@ -0,0 +1,6 @@
+version: "0.1"
+name: node-exporter
+repos:
+containers:
+pods:
+  - node-exporter