[WIP] Use templating for watcher dashboard and add Postgres exporter (#695)

* Add Postgres exporter and it's dashboard

* Use templating for watcher dashboard

* Add subgraph related panels to watcher dashboard

* Remove individual watcher dashboards and update instructions
This commit is contained in:
prathamesh0 2023-12-21 13:41:36 +05:30 committed by GitHub
parent 4a1a46facc
commit 38622fb33c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
18 changed files with 5214 additions and 20081 deletions

View File

@ -43,5 +43,15 @@ services:
extra_hosts: extra_hosts:
- "host.docker.internal:host-gateway" - "host.docker.internal:host-gateway"
postgres-exporter:
image: quay.io/prometheuscommunity/postgres-exporter
restart: always
volumes:
- ../config/monitoring/postgres-exporter.yml:/postgres_exporter.yml
ports:
- '9187'
extra_hosts:
- "host.docker.internal:host-gateway"
volumes: volumes:
prometheus_data: prometheus_data:

View File

@ -0,0 +1,8 @@
auth_modules:
foo:
type: userpass
userpass:
username: username
password: password
options:
sslmode: disable

View File

@ -40,3 +40,21 @@ scrape_configs:
static_configs: static_configs:
- targets: ['chain-head-exporter:5000'] - targets: ['chain-head-exporter:5000']
- job_name: 'postgres'
scrape_interval: 30s
scrape_timeout: 30s
static_configs:
# Add DB targets below
# - targets: [example-server:5432]
# labels:
# instance: 'example-label'
metrics_path: /probe
params:
auth_module: [foo]
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: postgres-exporter:9187

View File

@ -72,7 +72,7 @@ laconic-so --stack monitoring deploy create --spec-file monitoring-spec.yml --de
- targets: ['<METRICS_ENDPOINT_HOST>:<METRICS_ENDPOINT_PORT>'] - targets: ['<METRICS_ENDPOINT_HOST>:<METRICS_ENDPOINT_PORT>']
``` ```
* Also update the `blackbox` job to add any endpoints to be monitored on the Blackbox dashboard: * Blackbox: update the `blackbox` job to add any endpoints to be monitored on the Blackbox dashboard:
```yml ```yml
... ...
@ -85,6 +85,21 @@ laconic-so --stack monitoring deploy create --spec-file monitoring-spec.yml --de
- <HTTP_ENDPOINT_2> - <HTTP_ENDPOINT_2>
``` ```
* Postgres:
* Update the `postgres` job to add Postgres db targets to be monitored:
```yml
...
- job_name: 'postgres'
...
static_configs:
# Add DB targets below
- targets: [example-server:5432]
labels:
instance: 'example-db'
```
* Add database credentials to be used in `auth_modules` in the postgres-exporter config file (`monitoring-deployment/config/monitoring/postgres-exporter.yml`)
Note: Use `host.docker.internal` as host to access ports on the host machine Note: Use `host.docker.internal` as host to access ports on the host machine
### Grafana Config ### Grafana Config

View File

@ -35,8 +35,6 @@ laconic-so --stack monitoring deploy create --spec-file monitoring-watchers-spec
## Configure ## Configure
### Prometheus Config
Add the following scrape configs to prometheus config file (`monitoring-watchers-deployment/config/monitoring/prometheus/prometheus.yml`) in the deployment folder: Add the following scrape configs to prometheus config file (`monitoring-watchers-deployment/config/monitoring/prometheus/prometheus.yml`) in the deployment folder:
```yml ```yml
@ -55,27 +53,35 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers
- targets: ['AZIMUTH_WATCHER_HOST:AZIMUTH_WATCHER_PORT'] - targets: ['AZIMUTH_WATCHER_HOST:AZIMUTH_WATCHER_PORT']
labels: labels:
instance: 'azimuth' instance: 'azimuth'
chain: 'ethereum'
- targets: ['CENSURES_WATCHER_HOST:CENSURES_WATCHER_PORT'] - targets: ['CENSURES_WATCHER_HOST:CENSURES_WATCHER_PORT']
labels: labels:
instance: 'censures' instance: 'censures'
chain: 'ethereum'
- targets: ['CLAIMS_WATCHER_HOST:CLAIMS_WATCHER_PORT'] - targets: ['CLAIMS_WATCHER_HOST:CLAIMS_WATCHER_PORT']
labels: labels:
instance: 'claims' instance: 'claims'
chain: 'ethereum'
- targets: ['CONDITIONAL_STAR_RELEASE_WATCHER_HOST:CONDITIONAL_STAR_RELEASE_WATCHER_PORT'] - targets: ['CONDITIONAL_STAR_RELEASE_WATCHER_HOST:CONDITIONAL_STAR_RELEASE_WATCHER_PORT']
labels: labels:
instance: 'conditional_star_release' instance: 'conditional_star_release'
chain: 'ethereum'
- targets: ['DELEGATED_SENDING_WATCHER_HOST:DELEGATED_SENDING_WATCHER_PORT'] - targets: ['DELEGATED_SENDING_WATCHER_HOST:DELEGATED_SENDING_WATCHER_PORT']
labels: labels:
instance: 'delegated_sending_watcher' instance: 'delegated_sending'
chain: 'ethereum'
- targets: ['ECLIPTIC_WATCHER_HOST:ECLIPTIC_WATCHER_PORT'] - targets: ['ECLIPTIC_WATCHER_HOST:ECLIPTIC_WATCHER_PORT']
labels: labels:
instance: 'ecliptic' instance: 'ecliptic'
chain: 'ethereum'
- targets: ['LINEAR_STAR_WATCHER_HOST:LINEAR_STAR_WATCHER_PORT'] - targets: ['LINEAR_STAR_WATCHER_HOST:LINEAR_STAR_WATCHER_PORT']
labels: labels:
instance: 'linear_star_release' instance: 'linear_star_release'
chain: 'ethereum'
- targets: ['POLLS_WATCHER_HOST:POLLS_WATCHER_PORT'] - targets: ['POLLS_WATCHER_HOST:POLLS_WATCHER_PORT']
labels: labels:
instance: 'polls' instance: 'polls'
chain: 'ethereum'
- job_name: sushi - job_name: sushi
scrape_interval: 20s scrape_interval: 20s
@ -85,18 +91,14 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers
- targets: ['SUSHISWAP_WATCHER_HOST:SUSHISWAP_WATCHER_PORT'] - targets: ['SUSHISWAP_WATCHER_HOST:SUSHISWAP_WATCHER_PORT']
labels: labels:
instance: 'sushiswap' instance: 'sushiswap'
chain: 'filecoin'
- targets: ['MERKLE_SUSHISWAP_WATCHER_HOST:MERKLE_SUSHISWAP_WATCHER_PORT'] - targets: ['MERKLE_SUSHISWAP_WATCHER_HOST:MERKLE_SUSHISWAP_WATCHER_PORT']
labels: labels:
instance: 'merkl_sushiswap' instance: 'merkl_sushiswap'
chain: 'filecoin'
``` ```
### Grafana Config Add scrape config as done above for any additional watcher to add it to the watcher dashboard.
In the deployment folder, copy over the pre-configured watcher dashboard JSON files to grafana dashboards config directory:
```bash
cp -r monitoring-watchers-deployment/config/monitoring/grafana/watcher-dashboards/* monitoring-watchers-deployment/config/monitoring/grafana/dashboards/
```
### Env ### Env