Add readme for monitoring testnet services

stack_orchestrator/data/config/monitoring/blackbox.yml

@@ -5,3 +5,4 @@ modules:
     http:
       valid_status_codes: [] #default to 2xx
       method: GET
+      preferred_ip_protocol: ip4

stack_orchestrator/data/stacks/monitoring/monitoring-testnet.md

@@ -0,0 +1,128 @@
+# Monitoring Testnet
+
+Instructions to setup and run monitoring stack for testnet services
+
+## Create a deployment
+
+After completing [setup](./README.md#setup), create a spec file for the deployment, which will map the stack's ports and volumes to the host:
+
+```bash
+laconic-so --stack monitoring deploy init --output monitoring-testnet-spec.yml
+```
+
+### Ports
+
+Edit `network` in spec file to map container ports to same ports in host:
+
+```
+...
+network:
+  ports:
+    prometheus:
+      - '9090:9090'
+    grafana:
+      - '3000:3000'
+...
+```
+
+---
+
+Once you've made any needed changes to the spec file, create a deployment from it:
+
+```bash
+laconic-so --stack monitoring deploy create --spec-file monitoring-testnet-spec.yml --deployment-dir monitoring-testnet-deployment
+```
+
+## Configure
+
+### Prometheus scrape config
+
+Add the following scrape configs to prometheus config file (`monitoring-testnet-deployment/config/monitoring/prometheus/prometheus.yml`) in the deployment folder:
+
+  ```yml
+  ...
+  - job_name: 'blackbox'
+    ...
+    static_configs:
+      - targets:
+        - https://wallet.laconic.com
+        - https://laconicd-sapo.laconic.com
+        - https://console-sapo.laconic.com
+        - https://fixturenet-eth.laconic.com
+        - https://deploy.laconic.com
+        - https://deploy-backend.laconic.com/staging/version
+        - https://container-registry.apps.vaasl.io
+        - https://webapp-deployer-api.apps.vaasl.io
+        - https://webapp-deployer-ui.apps.vaasl.io
+  ...
+  - job_name: laconicd
+    ...
+    static_configs:
+      - targets: ['LACONICD_REST_HOST:LACONICD_REST_PORT']
+      # Example: 'host.docker.internal:3317'
+  ```
+
+### Grafana alerts config
+
+Place the pre-configured alerts rules in Grafana provisioning directory:
+
+  ```bash
+  # watcher alert rules
+  cp monitoring-testnet-deployment/config/monitoring/testnet-alert-rules.yml monitoring-testnet-deployment/config/monitoring/grafana/provisioning/alerting/
+  ```
+
+Update the alerting contact points config (`monitoring-testnet-deployment/config/monitoring/grafana/provisioning/alerting/contactpoints.yml`) with desired contact points
+
+Add corresponding routes to the notification policies config (`monitoring-testnet-deployment/config/monitoring/grafana/provisioning/alerting/policies.yml`) with appropriate object-matchers:
+
+  ```yml
+  ...
+    routes:
+      - receiver: SlackNotifier
+      object_matchers:
+        # Add matchers below
+        - ['grafana_folder', '=~', 'TestnetAlerts']
+  ```
+
+### Env
+
+Set the following env variables in the deployment env config file (`monitoring-testnet-deployment/config.env`):
+
+  ```bash
+  # Grafana server host URL to be used
+  # (Optional, default: http://localhost:3000)
+  GF_SERVER_ROOT_URL=
+  ```
+
+## Start the stack
+
+Start the deployment:
+
+```bash
+laconic-so deployment --dir monitoring-testnet-deployment start
+```
+
+* List and check the health status of all the containers using `docker ps` and wait for them to be `healthy`
+
+* Grafana should now be visible at http://localhost:3000 with configured dashboards
+
+## Clean up
+
+To stop monitoring services running in the background, while preserving data:
+
+```bash
+# Only stop the docker containers
+laconic-so deployment --dir monitoring-watchers-deployment stop
+
+# Run 'start' to restart the deployment
+```
+
+To stop monitoring services and also delete data:
+
+```bash
+# Stop the docker containers
+laconic-so deployment --dir monitoring-watchers-deployment stop --delete-volumes
+
+# Remove deployment directory (deployment will have to be recreated for a re-run)
+rm -rf monitoring-watchers-deployment
+```