Setup basic alerting for watchers in monitoring stack #698

Merged
prathamesh0 merged 5 commits from pm-grafana-alerts into main 2024-01-08 11:55:30 +00:00
Showing only changes of commit ce7d91d1bb - Show all commits

View File

@ -35,6 +35,8 @@ laconic-so --stack monitoring deploy create --spec-file monitoring-watchers-spec
## Configure
### Prometheus scrape config
Add the following scrape configs to prometheus config file (`monitoring-watchers-deployment/config/monitoring/prometheus/prometheus.yml`) in the deployment folder:
```yml
@ -100,6 +102,27 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers
Add scrape config as done above for any additional watcher to add it to the Watchers dashboard.
### Grafana alerts config
Place the pre-configured watcher alerts rules in Grafana provisioning directory:
```bash
cp monitoring-watchers-deployment/config/monitoring/watcher-alert-rules.yml monitoring-watchers-deployment/config/monitoring/grafana/provisioning/alerting/
```
Update the alerting contact points config (`monitoring-watchers-deployment/config/monitoring/grafana/provisioning/alerting/contactpoints.yml`) with desired contact points
Add corresponding routes to the notification policies config (`monitoring-watchers-deployment/monitoring/grafana/provisioning/alerting/policies.yaml`) with appropriate object-matchers:
```yml
...
routes:
- receiver: SlackNotifier
object_matchers:
# Add matchers below
- ['grafana_folder', '=', 'WatcherAlerts']
```
### Env
Set the following env variables in the deployment env config file (`monitoring-watchers-deployment/config.env`):