Reduce alert pending period for watcher alerts

Use classic condition for watcher alert rules
Add http status code check to alert rule
2024-04-22 18:48:56 +05:30 · 2024-04-22 18:47:34 +05:30 · 2024-04-15 18:42:26 +05:30 · 2024-04-15 17:51:17 +05:30 · 2024-04-15 17:02:50 +05:30
6 changed files with 270 additions and 114 deletions
--- a/stack_orchestrator/data/config/monitoring/blackbox-alert-rules.yml
+++ b/stack_orchestrator/data/config/monitoring/blackbox-alert-rules.yml
@ -0,0 +1,147 @@
+apiVersion: 1
+groups:
+    - orgId: 1
+      name: blackbox
+      folder: BlackboxAlerts
+      interval: 30s
+      rules:
+        # Azimuth Gateway endpoint
+        - uid: azimuth_gateway
+          title: azimuth_gateway_endpoint_tracking
+          condition: condition
+          data:
+            - refId: probe
+              relativeTimeRange:
+                from: 600
+                to: 0
+              datasourceUid: PBFA97CFB590B2093
+              model:
+                editorMode: code
+                expr: probe_success{destination="azimuth_gateway"}
+                instant: true
+                intervalMs: 1000
+                legendFormat: __auto
+                maxDataPoints: 43200
+                range: false
+                refId: probe
+            - refId: http_status_code
+              relativeTimeRange:
+                from: 600
+                to: 0
+              datasourceUid: PBFA97CFB590B2093
+              model:
+                editorMode: code
+                expr: probe_http_status_code{destination="azimuth_gateway"}
+                instant: true
+                intervalMs: 1000
+                legendFormat: __auto
+                maxDataPoints: 43200
+                range: false
+                refId: http_status_code
+            - refId: condition
+              relativeTimeRange:
+                from: 600
+                to: 0
+              datasourceUid: __expr__
+              model:
+                conditions:
+                    - evaluator:
+                        params:
+                            - 0
+                            - 0
+                        type: gt
+                      operator:
+                        type: and
+                      query:
+                        params: []
+                      reducer:
+                        params: []
+                        type: avg
+                      type: query
+                datasource:
+                    name: Expression
+                    type: __expr__
+                    uid: __expr__
+                expression: ${probe} != 1 ||  ${http_status_code} != 200
+                intervalMs: 1000
+                maxDataPoints: 43200
+                refId: condition
+                type: math
+          noDataState: Alerting
+          execErrState: Alerting
+          for: 5m
+          annotations:
+            summary: Probe failed for Azimuth gateway endpoint
+          labels:
+            probe_success: '{{ index $values "probe" }}'
+          isPaused: false
+        # Laconicd GQL endpoint
+        - uid: laconicd_gql
+          title: laconicd_gql_endpoint_tracking
+          condition: condition
+          data:
+            - refId: probe
+              relativeTimeRange:
+                from: 600
+                to: 0
+              datasourceUid: PBFA97CFB590B2093
+              model:
+                editorMode: code
+                expr: probe_success{destination="laconicd_gql"}
+                instant: true
+                intervalMs: 1000
+                legendFormat: __auto
+                maxDataPoints: 43200
+                range: false
+                refId: probe
+            - refId: http_status_code
+              relativeTimeRange:
+                from: 600
+                to: 0
+              datasourceUid: PBFA97CFB590B2093
+              model:
+                editorMode: code
+                expr: probe_http_status_code{destination="laconicd_gql"}
+                instant: true
+                intervalMs: 1000
+                legendFormat: __auto
+                maxDataPoints: 43200
+                range: false
+                refId: http_status_code
+            - refId: condition
+              relativeTimeRange:
+                from: 600
+                to: 0
+              datasourceUid: __expr__
+              model:
+                conditions:
+                    - evaluator:
+                        params:
+                            - 0
+                            - 0
+                        type: gt
+                      operator:
+                        type: and
+                      query:
+                        params: []
+                      reducer:
+                        params: []
+                        type: avg
+                      type: query
+                datasource:
+                    name: Expression
+                    type: __expr__
+                    uid: __expr__
+                expression: ${probe} != 1 ||  ${http_status_code} != 200
+                intervalMs: 1000
+                maxDataPoints: 43200
+                refId: condition
+                type: math
+          noDataState: Alerting
+          execErrState: Alerting
+          for: 5m
+          annotations:
+            summary: Probe failed for Laconicd GQL endpoint
+          labels:
+            probe_success: '{{ index $values "probe" }}'
+          isPaused: false
--- a/stack_orchestrator/data/config/monitoring/grafana/dashboards/laconicd-dashboard.json
+++ b/stack_orchestrator/data/config/monitoring/grafana/dashboards/laconicd-dashboard.json
@ -49,7 +49,7 @@
      },
      "gridPos": {
        "h": 3,
-        "w": 3,
+        "w": 4,
        "x": 0,
        "y": 0
      },
--- a/stack_orchestrator/data/config/monitoring/prometheus/prometheus.yml
+++ b/stack_orchestrator/data/config/monitoring/prometheus/prometheus.yml
@ -24,9 +24,10 @@ scrape_configs:
    params:
      module: [http_2xx]
    static_configs:
-      # Add URLs to be monitored below
-      - targets:
-        # - https://github.com
+      # Add URLs for targets to be monitored below
+      # - targets: [https://github.com]
+      #   labels:
+      #     destination: 'github'
    relabel_configs:
      - source_labels: [__address__]
        regex: (.*)(:80)?
--- a/stack_orchestrator/data/config/monitoring/watcher-alert-rules.yml
+++ b/stack_orchestrator/data/config/monitoring/watcher-alert-rules.yml
@ -59,29 +59,29 @@ groups:
              conditions:
                - evaluator:
                    params:
-                      - 0
+                      - 15
                      - 0
                    type: gt
                  operator:
-                    type: and
+                    type: when
                  query:
-                    params: []
+                    params:
+                      - diff
                  reducer:
                    params: []
-                    type: avg
+                    type: last
                  type: query
              datasource:
                name: Expression
                type: __expr__
                uid: __expr__
-              expression: ${diff} >= 16
-              intervalMs: 1000
-              maxDataPoints: 43200
+              expression: ""
+              hide: false
              refId: condition
-              type: math
+              type: classic_conditions
        noDataState: Alerting
        execErrState: Alerting
-        for: 15m
+        for: 5m
        annotations:
          summary: Watcher {{ index $labels "instance" }} of group {{ index $labels "job" }} is falling behind external head by {{ index $values "diff" }}
        isPaused: false
@ -135,29 +135,29 @@ groups:
              conditions:
                - evaluator:
                    params:
-                      - 0
+                      - 15
                      - 0
                    type: gt
                  operator:
-                    type: and
+                    type: when
                  query:
-                    params: []
+                    params:
+                      - diff
                  reducer:
                    params: []
-                    type: avg
+                    type: last
                  type: query
              datasource:
                name: Expression
                type: __expr__
                uid: __expr__
-              expression: ${diff} >= 16
-              intervalMs: 1000
-              maxDataPoints: 43200
+              expression: ""
+              hide: false
              refId: condition
-              type: math
+              type: classic_conditions
        noDataState: Alerting
        execErrState: Alerting
-        for: 15m
+        for: 5m
        annotations:
          summary: Watcher {{ index $labels "instance" }} of group {{ index $labels "job" }} is falling behind external head by {{ index $values "diff" }}
        isPaused: false
@ -211,29 +211,29 @@ groups:
              conditions:
                - evaluator:
                    params:
-                      - 0
+                      - 15
                      - 0
                    type: gt
                  operator:
-                    type: and
+                    type: when
                  query:
-                    params: []
+                    params:
+                      - diff
                  reducer:
                    params: []
-                    type: avg
+                    type: last
                  type: query
              datasource:
                name: Expression
                type: __expr__
                uid: __expr__
-              expression: ${diff} >= 16
-              intervalMs: 1000
-              maxDataPoints: 43200
+              expression: ""
+              hide: false
              refId: condition
-              type: math
+              type: classic_conditions
        noDataState: Alerting
        execErrState: Alerting
-        for: 15m
+        for: 5m
        annotations:
          summary: Watcher {{ index $labels "instance" }} of group {{ index $labels "job" }} is falling behind external head by {{ index $values "diff" }}
        isPaused: false
@ -287,29 +287,29 @@ groups:
              conditions:
                - evaluator:
                    params:
-                      - 0
+                      - 15
                      - 0
                    type: gt
                  operator:
-                    type: and
+                    type: when
                  query:
-                    params: []
+                    params:
+                      - diff
                  reducer:
                    params: []
-                    type: avg
+                    type: last
                  type: query
              datasource:
                name: Expression
                type: __expr__
                uid: __expr__
-              expression: ${diff} >= 16
-              intervalMs: 1000
-              maxDataPoints: 43200
+              expression: ""
+              hide: false
              refId: condition
-              type: math
+              type: classic_conditions
        noDataState: Alerting
        execErrState: Alerting
-        for: 15m
+        for: 5m
        annotations:
          summary: Watcher {{ index $labels "instance" }} of group {{ index $labels "job" }} is falling behind external head by {{ index $values "diff" }}
        isPaused: false
@ -363,29 +363,29 @@ groups:
              conditions:
                - evaluator:
                    params:
-                      - 0
+                      - 15
                      - 0
                    type: gt
                  operator:
-                    type: and
+                    type: when
                  query:
-                    params: []
+                    params:
+                      - diff
                  reducer:
                    params: []
-                    type: avg
+                    type: last
                  type: query
              datasource:
                name: Expression
                type: __expr__
                uid: __expr__
-              expression: ${diff} >= 16
-              intervalMs: 1000
-              maxDataPoints: 43200
+              expression: ""
+              hide: false
              refId: condition
-              type: math
+              type: classic_conditions
        noDataState: Alerting
        execErrState: Alerting
-        for: 15m
+        for: 5m
        annotations:
          summary: Watcher {{ index $labels "instance" }} of group {{ index $labels "job" }} is falling behind external head by {{ index $values "diff" }}
        isPaused: false
@ -439,29 +439,29 @@ groups:
              conditions:
                - evaluator:
                    params:
-                      - 0
+                      - 15
                      - 0
                    type: gt
                  operator:
-                    type: and
+                    type: when
                  query:
-                    params: []
+                    params:
+                      - diff
                  reducer:
                    params: []
-                    type: avg
+                    type: last
                  type: query
              datasource:
                name: Expression
                type: __expr__
                uid: __expr__
-              expression: ${diff} >= 16
-              intervalMs: 1000
-              maxDataPoints: 43200
+              expression: ""
+              hide: false
              refId: condition
-              type: math
+              type: classic_conditions
        noDataState: Alerting
        execErrState: Alerting
-        for: 15m
+        for: 5m
        annotations:
          summary: Watcher {{ index $labels "instance" }} of group {{ index $labels "job" }} is falling behind external head by {{ index $values "diff" }}
        isPaused: false
@ -515,29 +515,29 @@ groups:
              conditions:
                - evaluator:
                    params:
-                      - 0
+                      - 15
                      - 0
                    type: gt
                  operator:
-                    type: and
+                    type: when
                  query:
-                    params: []
+                    params:
+                      - diff
                  reducer:
                    params: []
-                    type: avg
+                    type: last
                  type: query
              datasource:
                name: Expression
                type: __expr__
                uid: __expr__
-              expression: ${diff} >= 16
-              intervalMs: 1000
-              maxDataPoints: 43200
+              expression: ""
+              hide: false
              refId: condition
-              type: math
+              type: classic_conditions
        noDataState: Alerting
        execErrState: Alerting
-        for: 15m
+        for: 5m
        annotations:
          summary: Watcher {{ index $labels "instance" }} of group {{ index $labels "job" }} is falling behind external head by {{ index $values "diff" }}
        isPaused: false
@ -591,29 +591,29 @@ groups:
              conditions:
                - evaluator:
                    params:
-                      - 0
+                      - 15
                      - 0
                    type: gt
                  operator:
-                    type: and
+                    type: when
                  query:
-                    params: []
+                    params:
+                      - diff
                  reducer:
                    params: []
-                    type: avg
+                    type: last
                  type: query
              datasource:
                name: Expression
                type: __expr__
                uid: __expr__
-              expression: ${diff} >= 16
-              intervalMs: 1000
-              maxDataPoints: 43200
+              expression: ""
+              hide: false
              refId: condition
-              type: math
+              type: classic_conditions
        noDataState: Alerting
        execErrState: Alerting
-        for: 15m
+        for: 5m
        annotations:
          summary: Watcher {{ index $labels "instance" }} of group {{ index $labels "job" }} is falling behind external head by {{ index $values "diff" }}
        isPaused: false
@ -669,29 +669,29 @@ groups:
              conditions:
                - evaluator:
                    params:
-                      - 0
+                      - 15
                      - 0
                    type: gt
                  operator:
-                    type: and
+                    type: when
                  query:
-                    params: []
+                    params:
+                      - diff
                  reducer:
                    params: []
-                    type: avg
+                    type: last
                  type: query
              datasource:
                name: Expression
                type: __expr__
                uid: __expr__
-              expression: ${diff} >= 16
-              intervalMs: 1000
-              maxDataPoints: 43200
+              expression: ""
+              hide: false
              refId: condition
-              type: math
+              type: classic_conditions
        noDataState: Alerting
        execErrState: Alerting
-        for: 15m
+        for: 5m
        annotations:
          summary: Watcher {{ index $labels "instance" }} of group {{ index $labels "job" }} is falling behind external head by {{ index $values "diff" }}
        isPaused: false
@ -745,29 +745,29 @@ groups:
              conditions:
                - evaluator:
                    params:
-                      - 0
+                      - 15
                      - 0
                    type: gt
                  operator:
-                    type: and
+                    type: when
                  query:
-                    params: []
+                    params:
+                      - diff
                  reducer:
                    params: []
-                    type: avg
+                    type: last
                  type: query
              datasource:
                name: Expression
                type: __expr__
                uid: __expr__
-              expression: ${diff} >= 16
-              intervalMs: 1000
-              maxDataPoints: 43200
+              expression: ""
+              hide: false
              refId: condition
-              type: math
+              type: classic_conditions
        noDataState: Alerting
        execErrState: Alerting
-        for: 15m
+        for: 5m
        annotations:
          summary: Watcher {{ index $labels "instance" }} of group {{ index $labels "job" }} is falling behind external head by {{ index $values "diff" }}
        isPaused: false
@ -823,29 +823,29 @@ groups:
              conditions:
                - evaluator:
                    params:
-                      - 0
+                      - 15
                      - 0
                    type: gt
                  operator:
-                    type: and
+                    type: when
                  query:
-                    params: []
+                    params:
+                      - diff
                  reducer:
                    params: []
-                    type: avg
+                    type: last
                  type: query
              datasource:
                name: Expression
                type: __expr__
                uid: __expr__
-              expression: ${diff} >= 16
-              intervalMs: 1000
-              maxDataPoints: 43200
+              expression: ""
+              hide: false
              refId: condition
-              type: math
+              type: classic_conditions
        noDataState: Alerting
        execErrState: Alerting
-        for: 15m
+        for: 5m
        annotations:
          summary: Watcher {{ index $labels "instance" }} of group {{ index $labels "job" }} is falling behind external head by {{ index $values "diff" }}
        isPaused: false
--- a/stack_orchestrator/data/stacks/monitoring/README.md
+++ b/stack_orchestrator/data/stacks/monitoring/README.md
@ -123,6 +123,7 @@ laconic-so --stack monitoring deploy create --spec-file monitoring-spec.yml --de
  ```yml
  ...
  - job_name: laconicd
+    ...
    static_configs:
      - targets: ['example-host:1317']
  ...
--- a/stack_orchestrator/data/stacks/monitoring/monitoring-watchers.md
+++ b/stack_orchestrator/data/stacks/monitoring/monitoring-watchers.md
@ -44,11 +44,16 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers
  - job_name: 'blackbox'
    ...
    static_configs:
-      - targets:
-        - <AZIMUTH_GATEWAY_GQL_ENDPOINT>
-        - <LACONICD_GQL_ENDPOINT>
+      - targets: [<AZIMUTH_GATEWAY_GQL_ENDPOINT>]
+        labels:
+          # Add destination label for pre-configured alerts
+          destination: 'azimuth_gateway'
+      - targets: [<LACONICD_GQL_ENDPOINT>]
+        labels:
+          destination: 'laconicd_gql'
  ...
  - job_name: laconicd
+    ...
    static_configs:
      - targets: ['LACONICD_REST_HOST:LACONICD_REST_PORT']
  ...
@ -119,10 +124,12 @@ Add scrape config as done above for any additional watcher to add it to the Watc

 ### Grafana alerts config

-Place the pre-configured watcher alerts rules in Grafana provisioning directory:
+Place the pre-configured watcher and blackbox endpoint alerts rules in Grafana provisioning directory:

  ```bash
  cp monitoring-watchers-deployment/config/monitoring/watcher-alert-rules.yml monitoring-watchers-deployment/config/monitoring/grafana/provisioning/alerting/
+
+  cp monitoring-watchers-deployment/config/monitoring/blackbox-alert-rules.yml monitoring-watchers-deployment/config/monitoring/grafana/provisioning/alerting/
  ```

 Update the alerting contact points config (`monitoring-watchers-deployment/config/monitoring/grafana/provisioning/alerting/contactpoints.yml`) with desired contact points
@ -135,7 +142,7 @@ Add corresponding routes to the notification policies config (`monitoring-watche
      - receiver: SlackNotifier
      object_matchers:
        # Add matchers below
-        - ['grafana_folder', '=', 'WatcherAlerts']
+        - ['grafana_folder', '=~', 'WatcherAlerts|BlackboxAlerts']
  ```

 ### Env
Author	SHA1	Message	Date
Prathamesh Musale	68132c3305	Reduce alert pending period for watcher alerts	2024-04-22 18:48:56 +05:30
Prathamesh Musale	cfc411bfe4	Use classic condition for watcher alert rules	2024-04-22 18:47:34 +05:30
Prathamesh Musale	abf4d39a22	Add http status code check to alert rule	2024-04-15 18:42:26 +05:30
Prathamesh Musale	23d527720f	Add a custom label to blackbox metrics and update instructions	2024-04-15 17:51:17 +05:30
Prathamesh Musale	1746f7366c	Add alerts on blackbox metrics for monitoring endpoints	2024-04-15 17:02:50 +05:30