Monitoring

Heron does not run Prometheus or Grafana itself. It provides metrics, a dashboard and alert rules to use with your own Grafana, self-hosted or Grafana Cloud.

Quick start with Grafana Cloud

1. In your Grafana Cloud stack, open the Prometheus data source details. Note the remote write endpoint and the instance ID.

2. Create an access policy token with the metrics:write scope.

3. In .env:

   HERON_STATION_NAME=radio1
   HERON_METRICS_REMOTE_WRITE_URL=https://prometheus-prod-XX-prod-REGION.grafana.net/api/prom/push
   HERON_METRICS_REMOTE_WRITE_USERNAME=123456
   HERON_METRICS_REMOTE_WRITE_PASSWORD=glc_…

4. docker compose up -d. Each service logs pushing metrics with remote write.

5. In Grafana, import monitoring/grafana/heron-dashboard.json from the repository and pick your Prometheus data source.

6. Load the alert rules and set up a contact point.

Sending the metrics

Remote write (push). With HERON_METRICS_REMOTE_WRITE_URL set, every service pushes its metrics every 15 s. The station only makes outgoing HTTPS requests, so it works behind NAT. Prometheus 2.33+ (with --web.enable-remote-write-receiver), Grafana Mimir, VictoriaMetrics and Thanos Receive all accept it. A failed push is never retried and never stops a service.

Scraping (pull). Leave the URL empty and scrape /metrics on each service:

Service	Port
heron-engine	8081 (scrape every 5 s)
heron-api	8080
heron-analyzer	8082
heron-librarian	8083

These ports have no authentication. Join Heron’s Docker network (heron_default) from your Prometheus, or publish them on 127.0.0.1 or a private network only, never on the Internet.

Every series carries job (the service) and station (HERON_STATION_NAME).

The dashboard

monitoring/grafana/heron-dashboard.json (Grafana 12 or later) shows:

• On air: dead air, silence, outputs connected, tracks queued, master level, fallback tracks and silence skips;
• Audio engine health: underruns, dropped audio, reconnections, the clock’s lateness and drift;
• Server clock, control plane, library and running versions.

A healthy station shows no dead air, every output connected, two or more tracks queued, a flat clock drift, and zero underruns and dropped audio.

Alerts

monitoring/prometheus/alerts.yml is a standard Prometheus rule file. Load it with mimirtool (Grafana Cloud, Mimir) or list it under rule_files (Prometheus).

Critical alerts mean listeners hear the problem, or soon will: the engine is down, dead air, silence on air, an output disconnected for 2 min, dropped audio. Warnings mean the air is safe but something needs a look: fallback tracks playing, silence skips, underruns, unstable outputs, an unsynchronized server clock, a service or the API’s link to the engine down, failing analyses or scans, many failed sign-ins.

Health checks

Every service answers GET /healthz:

{"service": "heron-engine", "version": "0.1.0", "status": "ok"}

It answers 503 when the service is unhealthy, and lists warnings for problems that need attention, such as an unsynchronized clock or nothing left to play. Docker’s health checks use it.

From outside, also watch what listeners get: an HTTP check of your stream URL (Uptime Kuma, UptimeRobot, Grafana Synthetic Monitoring) catches DNS, TLS, network and stream server problems.

Keep the clock right

Run NTP on the host (chrony or systemd-timesyncd). The engine and the web interface warn when the server clock is not synchronized.

Logs

Containers log one JSON object per line. Set the level with RUST_LOG in .env, and follow a service with:

docker compose logs -f heron-engine

Sentry

Set HERON_SENTRY_DSN to report the services’ errors to Sentry (sentry.io or self-hosted), and HERON_WEB_SENTRY_DSN, in a project of its own, for the web interface.