diff --git a/stack_orchestrator/data/config/monitoring/blackbox.yml b/stack_orchestrator/data/config/monitoring/blackbox.yml index 4a9d72eb..d24e7574 100644 --- a/stack_orchestrator/data/config/monitoring/blackbox.yml +++ b/stack_orchestrator/data/config/monitoring/blackbox.yml @@ -5,3 +5,4 @@ modules: http: valid_status_codes: [] #default to 2xx method: GET + preferred_ip_protocol: ip4 diff --git a/stack_orchestrator/data/stacks/monitoring/monitoring-testnet.md b/stack_orchestrator/data/stacks/monitoring/monitoring-testnet.md new file mode 100644 index 00000000..e20fab13 --- /dev/null +++ b/stack_orchestrator/data/stacks/monitoring/monitoring-testnet.md @@ -0,0 +1,128 @@ +# Monitoring Testnet + +Instructions to setup and run monitoring stack for testnet services + +## Create a deployment + +After completing [setup](./README.md#setup), create a spec file for the deployment, which will map the stack's ports and volumes to the host: + +```bash +laconic-so --stack monitoring deploy init --output monitoring-testnet-spec.yml +``` + +### Ports + +Edit `network` in spec file to map container ports to same ports in host: + +``` +... +network: + ports: + prometheus: + - '9090:9090' + grafana: + - '3000:3000' +... +``` + +--- + +Once you've made any needed changes to the spec file, create a deployment from it: + +```bash +laconic-so --stack monitoring deploy create --spec-file monitoring-testnet-spec.yml --deployment-dir monitoring-testnet-deployment +``` + +## Configure + +### Prometheus scrape config + +Add the following scrape configs to prometheus config file (`monitoring-testnet-deployment/config/monitoring/prometheus/prometheus.yml`) in the deployment folder: + + ```yml + ... + - job_name: 'blackbox' + ... + static_configs: + - targets: + - https://wallet.laconic.com + - https://laconicd-sapo.laconic.com + - https://console-sapo.laconic.com + - https://fixturenet-eth.laconic.com + - https://deploy.laconic.com + - https://deploy-backend.laconic.com/staging/version + - https://container-registry.apps.vaasl.io + - https://webapp-deployer-api.apps.vaasl.io + - https://webapp-deployer-ui.apps.vaasl.io + ... + - job_name: laconicd + ... + static_configs: + - targets: ['LACONICD_REST_HOST:LACONICD_REST_PORT'] + # Example: 'host.docker.internal:3317' + ``` + +### Grafana alerts config + +Place the pre-configured alerts rules in Grafana provisioning directory: + + ```bash + # watcher alert rules + cp monitoring-testnet-deployment/config/monitoring/testnet-alert-rules.yml monitoring-testnet-deployment/config/monitoring/grafana/provisioning/alerting/ + ``` + +Update the alerting contact points config (`monitoring-testnet-deployment/config/monitoring/grafana/provisioning/alerting/contactpoints.yml`) with desired contact points + +Add corresponding routes to the notification policies config (`monitoring-testnet-deployment/config/monitoring/grafana/provisioning/alerting/policies.yml`) with appropriate object-matchers: + + ```yml + ... + routes: + - receiver: SlackNotifier + object_matchers: + # Add matchers below + - ['grafana_folder', '=~', 'TestnetAlerts'] + ``` + +### Env + +Set the following env variables in the deployment env config file (`monitoring-testnet-deployment/config.env`): + + ```bash + # Grafana server host URL to be used + # (Optional, default: http://localhost:3000) + GF_SERVER_ROOT_URL= + ``` + +## Start the stack + +Start the deployment: + +```bash +laconic-so deployment --dir monitoring-testnet-deployment start +``` + +* List and check the health status of all the containers using `docker ps` and wait for them to be `healthy` + +* Grafana should now be visible at http://localhost:3000 with configured dashboards + +## Clean up + +To stop monitoring services running in the background, while preserving data: + +```bash +# Only stop the docker containers +laconic-so deployment --dir monitoring-watchers-deployment stop + +# Run 'start' to restart the deployment +``` + +To stop monitoring services and also delete data: + +```bash +# Stop the docker containers +laconic-so deployment --dir monitoring-watchers-deployment stop --delete-volumes + +# Remove deployment directory (deployment will have to be recreated for a re-run) +rm -rf monitoring-watchers-deployment +```