Update monitoring stack with additional dashboards and watcher metrics (#693)

* Include retry jobs and update default refresh intervals * Add prometheus blackbox exporter and it's dashboard * Add NodeJS application dashboard * Allow UI updates * Update watcher dashboards for upstream and external chain heads * Update watcher dashboards with watcher config metrics * Upgrade sushiswap and azimuth watchers * Removed fixed title size values * Update instructions * Update instructions for env config * Update instructions with setup
2023-12-21 09:26:37 +05:30
parent 42b92f7e23
commit 4a1a46facc
26 changed files with 12143 additions and 231 deletions
@@ -40,20 +40,36 @@ network:
  ports:
    watcher-db:
     - 0.0.0.0:15432:5432
+    azimuth-watcher-job-runner:
+     - 0.0.0.0:9000:9000
    azimuth-watcher-server:
     - 0.0.0.0:3001:3001
+    censures-watcher-job-runner:
+     - 0.0.0.0:9002:9002
    censures-watcher-server:
     - 0.0.0.0:3002:3002
+    claims-watcher-job-runner:
+     - 0.0.0.0:9004:9004
    claims-watcher-server:
     - 0.0.0.0:3003:3003
+    conditional-star-release-watcher-job-runner:
+     - 0.0.0.0:9006:9006
    conditional-star-release-watcher-server:
     - 0.0.0.0:3004:3004
+    delegated-sending-watcher-job-runner:
+     - 0.0.0.0:9008:9008
    delegated-sending-watcher-server:
     - 0.0.0.0:3005:3005
+    ecliptic-watcher-job-runner:
+     - 0.0.0.0:9010:9010
    ecliptic-watcher-server:
     - 0.0.0.0:3006:3006
+    linear-star-release-watcher-job-runner:
+     - 0.0.0.0:9012:9012
    linear-star-release-watcher-server:
     - 0.0.0.0:3007:3007
+    polls-watcher-job-runner:
+     - 0.0.0.0:9014:9014
    polls-watcher-server:
     - 0.0.0.0:3008:3008
    gateway-server:
@@ -1,7 +1,7 @@
 version: "1.0"
 name: azimuth
 repos:
-  - github.com/cerc-io/azimuth-watcher-ts@v0.1.2
+  - github.com/cerc-io/azimuth-watcher-ts@v0.1.3
 containers:
  - cerc/watcher-azimuth
 pods:
@@ -1,8 +1,26 @@
 # monitoring

 * Instructions to setup and run a Prometheus server and Grafana dashboard
+* Comes with the following built-in exporters / dashboards:
+  * [Prometheus Blackbox Exporter](https://grafana.com/grafana/dashboards/7587-prometheus-blackbox-exporter/) - for tracking HTTP endpoints
+  * [NodeJS Application Dashboard](https://grafana.com/grafana/dashboards/11159-nodejs-application-dashboard/) - for default NodeJS metrics
+  * Chain Head Exporter - for tracking chain heads given external ETH RPC endpoints
 * See [monitoring-watchers.md](./monitoring-watchers.md) for an example usage of the stack with pre-configured dashboards for watchers

+## Setup
+
+Clone required repositories:
+
+```bash
+laconic-so --stack monitoring setup-repositories --git-ssh --pull
+```
+
+Build the container images:
+
+```bash
+laconic-so --stack monitoring build-containers
+```
+
 ## Create a deployment

 First, create a spec file for the deployment, which will map the stack's ports and volumes to the host:
@@ -43,7 +61,7 @@ laconic-so --stack monitoring deploy create --spec-file monitoring-spec.yml --de

 ### Prometheus Config

-Add desired scrape configs to prometheus config file (`monitoring-deployment/config/monitoring/prometheus/prometheus.yml`) in the deployment folder; for example:
+* Add desired scrape configs to prometheus config file (`monitoring-deployment/config/monitoring/prometheus/prometheus.yml`) in the deployment folder; for example:

  ```yml
  ...
@@ -54,12 +72,43 @@ Add desired scrape configs to prometheus config file (`monitoring-deployment/con
      - targets: ['<METRICS_ENDPOINT_HOST>:<METRICS_ENDPOINT_PORT>']
  ```

+* Also update the `blackbox` job to add any endpoints to be monitored on the Blackbox dashboard:
+
+  ```yml
+  ...
+  - job_name: 'blackbox'
+    ...
+    static_configs:
+      # Add URLs to be monitored below
+      - targets:
+        - <HTTP_ENDPOINT_1>
+        - <HTTP_ENDPOINT_2>
+  ```
+
 Note: Use `host.docker.internal` as host to access ports on the host machine

 ### Grafana Config

 Place the dashboard json files in grafana dashboards config directory (`monitoring-deployment/config/monitoring/grafana/dashboards`) in the deployment folder

+### Env
+
+Set the following env variables in the deployment env config file (`monitoring-deployment/config.env`):
+
+  ```bash
+  # External ETH RPC endpoint (ethereum)
+  # (Optional, default: https://mainnet.infura.io/v3)
+  CERC_ETH_RPC_ENDPOINT=
+
+  # Infura key to be used
+  # (Optional, used with ETH_RPC_ENDPOINT if provided)
+  CERC_INFURA_KEY=
+
+  # External ETH RPC endpoint (filecoin)
+  # (Optional, default: https://api.node.glif.io/rpc/v1)
+  CERC_FIL_RPC_ENDPOINT=
+  ```
+
 ## Start the stack

 Start the deployment:
@@ -4,7 +4,7 @@ Instructions to setup and run monitoring stack with pre-configured watcher dashb

 ## Create a deployment

-First, create a spec file for the deployment, which will map the stack's ports and volumes to the host:
+After completing [setup](./README.md#setup), create a spec file for the deployment, which will map the stack's ports and volumes to the host:

 ```bash
 laconic-so --stack monitoring deploy init --output monitoring-watchers-spec.yml
@@ -41,7 +41,14 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers

  ```yml
  ...
+  - job_name: 'blackbox'
+    ...
+    static_configs:
+      - targets:
+        - <AZIMUTH_GATEWAY_GQL_ENDPOINT>
+  ...
  - job_name: azimuth
+    scrape_interval: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
@@ -71,6 +78,7 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers
          instance: 'polls'

  - job_name: sushi
+    scrape_interval: 20s
    metrics_path: /metrics
    scheme: http
    static_configs:
@@ -90,6 +98,15 @@ In the deployment folder, copy over the pre-configured watcher dashboard JSON fi
 cp -r monitoring-watchers-deployment/config/monitoring/grafana/watcher-dashboards/* monitoring-watchers-deployment/config/monitoring/grafana/dashboards/
 ```

+### Env
+
+Set the following env variables in the deployment env config file (`monitoring-watchers-deployment/config.env`):
+
+  ```bash
+  # Infura key to be used
+  CERC_INFURA_KEY=
+  ```
+
 ## Start the stack

 Start the deployment:
@@ -1,7 +1,9 @@
 version: "0.1"
 name: monitoring
 repos:
+  - github.com/cerc-io/watcher-ts@v0.2.79
 containers:
+  - cerc/watcher-ts
 pods:
  - prom-server
  - grafana
@@ -2,7 +2,7 @@ version: "1.0"
 name: sushiswap-v3
 description: "SushiSwap v3 watcher stack"
 repos:
-  - github.com/cerc-io/sushiswap-v3-watcher-ts@v0.1.4
+  - github.com/cerc-io/sushiswap-v3-watcher-ts@v0.1.5
 containers:
  - cerc/watcher-sushiswap-v3
 pods: