Which signal reaches a human, and how
The homelab produces three fundamentally different kinds of "something is wrong" signals: metrics crossing a threshold, an external probe failing, and a policy violation being recorded. Only one of those currently ends in a push notification. This page traces each path end-to-end — source, transport, and the exact point where a human enters the loop (or doesn't).
It crosses every section of the docs, so it lives in topics/. The actors:
| Actor | Where it lives in docs |
|---|---|
| victoria-stack (vmagent, vmsingle, vlsingle, vmalert, Alertmanager, Grafana) | apps/monitoring |
| grafana-to-ntfy (webhook → ntfy bridge) | apps/monitoring |
| ntfy (push notifications to phones) | apps/ntfy |
| Gatus (external uptime probes) | apps/gatus |
| Kyverno (produces PolicyReports) | platform/kyverno |
| policy-reporter (renders PolicyReports) | apps/policy-reporter |
| Flux, k8up, cert-manager (the alert subjects) | foundation/flux · platform/k8up · platform/cert-manager |
The paths, top-down
One push path, two pull paths. Everything that pages a human funnels through a single Alertmanager route.
Path 1 — metric threshold to phone
How metrics get in
Four ingestion routes feed vmsingle, all described on apps/monitoring:
| Route | Who uses it | Defined in |
|---|---|---|
Annotation-driven scraping (prometheus.io/scrape: "true" on pods/services) | Flux controllers, the k8up operator, any opted-in app | inlineScrapeConfig in victoria-stack-values.yaml |
| Chart built-in jobs | kubelet, kube-apiserver, etcd, kube-state-metrics, node-exporter | the victoria-metrics-k8s-stack chart |
A VMPodScrape CR | Envoy Gateway data-plane pods (:19001/stats/prometheus, relabeled per gateway) | k8s/apps/talos/monitoring/envoy-traffic/scrape-envoy.yaml |
| Push | k8up restic job pods, via BACKUP_PROMURL pointing at vmsingle's /api/v1/import/prometheus | k8s/platform/talos/controllers/k8up/k8up-values.yaml |
Scraping is annotation-driven by design — the Prometheus-operator CRDs are not installed, and the chart's converter is disabled (disable_prometheus_converter: true). CNPG instance pods get a dedicated scrape job keyed on the cnpg.io/cluster label instead of per-cluster PodMonitors.
The rules
vmalert evaluates two sets of rules. The chart's community-maintained defaultRules are all enabled (minus the control-plane groups Talos doesn't expose), and three hand-written VMRules live in k8s/apps/talos/monitoring/:
| VMRule | Alert | Fires when | Severity |
|---|---|---|---|
flux-alerts | FluxReconciliationFailure | a Kustomization/HelmRelease/source is not-Ready for 15m (gotk_reconcile_condition) | critical |
k8up-alerts | K8upBackupLastErrors | k8up_backup_restic_last_errors > 0 | critical |
k8up-alerts | K8upBackupFailed | any k8up job failed within 25h | critical |
k8up-alerts | K8upBackupNotRunning | a namespace has a Schedule but ran no job in 25h | critical |
cert-manager-alerts | CertManagerCertExpiringSoon | a cert expires in < 14 days — auto-renewal is broken | critical |
cert-manager-alerts | CertManagerCertNotReady | a Certificate is not Ready for 1h | critical |
Every hand-written rule is severity: critical on purpose — see the routing below. The subjects tie this page to the rest of the stack: the Flux rule guards the GitOps loop, the k8up rules guard the backup tiers, the cert-manager rules guard the TLS edge.
Because vmalert routes queries through an internal vmauth in front of both vmsingle and vlsingle, LogsQL rules would work exactly like MetricsQL rules — log-based alerting is wired, just not used yet.
The routing
Alert routing is a VMAlertmanagerConfig (general-notifications, in k8s/apps/talos/monitoring/victoria-stack/alertmanager-config.yaml):
route:
receiver: "null"
group_by: ["alertname", "job", "severity"]
routes:
- receiver: "ntfy"
matchers: ['severity="critical"']
continue: true
- receiver: "mail"
matchers: ['severity="critical"']
continue: true
Read that carefully — it encodes the whole notification policy:
severity="critical"fans out to both receivers (continue: truemeans the first match doesn't stop evaluation): a push notification and an email for every critical alert.- Everything else lands in the
"null"receiver. The communitydefaultRulesship plenty ofwarning/infoalerts; they show up in the Alertmanager UI (alertmanager.talos.int.kueber.eu) and Grafana, but they never page anyone. Noise stays on the dashboard; only critical crosses into a pocket.
The mail receiver sends to talos@johanneskueber.com (with send_resolved: true) through the all-inkl SMTP relay (w0097b9d.kasserver.com:587, from noreply@kueber.eu), credentials mounted from the SOPS-encrypted allinkl-smtp-credentials secret.
The last hop — grafana-to-ntfy → ntfy
Alertmanager's webhook format is not ntfy's format, so a tiny bridge sits between them: the grafana-to-ntfy Deployment in the monitoring namespace (kittyandrew/grafana-to-ntfy, digest-pinned). Alertmanager POSTs to http://grafana-to-ntfy.monitoring.svc.cluster.local:8080 with basic auth from the ntfy-auth secret; the bridge authenticates the incoming call (BAUTH_USER/BAUTH_PASS), reformats the payload, and pushes it to ntfy with its own credentials (NTFY_URL, NTFY_BAUTH_USER, NTFY_BAUTH_PASS — all SOPS-encrypted, and the target topic is part of the encrypted NTFY_URL).
ntfy itself runs with NTFY_ENABLE_LOGIN: "true" and NTFY_AUTH_DEFAULT_ACCESS: deny-all — no anonymous publishing or reading, so the alert topic can't be spammed or snooped. Phones subscribe through the public ntfy.web.kueber.eu HTTPRoute; the auth and cache databases sit on a k8up-backed Longhorn PVC.
Path 2 — endpoint down → Gatus
Gatus is the outside-in check: it probes the public URLs (*.web.kueber.eu, *.int.kueber.eu), so every probe exercises DNS, the edge chain, Envoy Gateway, TLS, and the app itself — not just an in-cluster Service. A NetBird sidecar gives the pod VPN reach so the .int endpoints resolve too. Endpoints live in plain ConfigMap YAML under k8s/apps/talos/gatus/config/, grouped into Web (Nextcloud, Outline, Gitea, Immich, Vaultwarden, …), Arr, Int, and Talos (Longhorn UI). Most conditions are [STATUS] == 200; the arr apps assert [STATUS] == 401 — the auth wall responding is the healthy signal.
Two honest caveats, straight from the manifests:
- Gatus is currently a pull path. The config files contain no
alerting:block — a failing endpoint turns red ongatus.web.kueber.eu(uptime history in a CNPG PostgreSQL cluster), but nothing pushes to ntfy. Wiring Gatus's ntfy alerting provider is the obvious next step; today the dashboard is the destination. - Its metrics aren't collected.
metrics: trueis set inglobal.yaml, but the gatus Service carries noprometheus.io/scrapeannotation, so vmagent never picks it up — meaning no VMRule can fire ongatus_results_*either.
Gatus also runs inside the cluster it monitors. It catches an app, route, or cert being broken while the cluster lives; it cannot report the cluster itself being gone.
Path 3 — policy violation → policy-reporter
Kyverno validates admissions and background-scans existing workloads, writing results into PolicyReport/ClusterPolicyReport resources. policy-reporter reads those live from the API (no persistence) and renders them at policy-reporter.talos.int.kueber.eu, with the Kyverno plugin enabled for report enrichment.
This is the third path, and it is entirely pull: metrics.enabled: true exposes Prometheus metrics, but no VMRule references them and the chart's notification targets (the upstream chart supports pushing findings to webhooks) are not configured. A new policy violation reaches a human only when one opens the UI. That is a reasonable trade-off for policy findings — they are rarely urgent the way a failed backup is — but it's a choice, not an accident of wiring.
Summary — signal by signal
| Signal | Transport | Where a human sees it | Push or pull |
|---|---|---|---|
Metric threshold, severity: critical | vmagent → vmsingle → vmalert → Alertmanager → grafana-to-ntfy → ntfy and → SMTP | Phone notification + talos@johanneskueber.com inbox | Push |
| Metric threshold, warning/info | same, until Alertmanager's "null" receiver | Alertmanager UI, Grafana | Pull |
| Public endpoint down / TLS broken | Gatus probe every interval | gatus.web.kueber.eu dashboard | Pull |
| Policy violation | Kyverno → PolicyReport → policy-reporter | policy-reporter.talos.int.kueber.eu UI | Pull |
| Logs (errors, stack traces) | vlagent → vlsingle | Grafana LogsQL queries | Pull |
What can break, and where to look
| Symptom | Most likely cause | Where to look first |
|---|---|---|
| No notifications at all, cluster fine | grafana-to-ntfy down (ntfy leg) and SMTP relay creds rotated (mail leg) — two independent legs rarely die together, check both | monitoring namespace pods; Alertmanager UI at alertmanager.talos.int.kueber.eu |
| Push missing, email arrives | grafana-to-ntfy or ntfy down, or ntfy-auth/grafana-to-ntfy secrets drifted | grafana-to-ntfy logs; apps/ntfy |
| Email missing, push arrives | all-inkl SMTP relay or allinkl-smtp-credentials | Alertmanager logs |
| An alert that "should" have fired never did | it's severity: warning — routed to "null" by design | the routing in alertmanager-config.yaml |
| No alerts and no fresh metrics | vmagent or vmsingle down — the whole path is blind | Grafana; flux get all -n monitoring |
| Gatus all red, apps actually fine | edge chain problem, not app problem — that's the point of probing public URLs | topics/envoy-gateway-proxy-protocol-v2, fabric/netbird |
| Gatus dashboard empty / no history | its CNPG cluster is down | platform/cloudnative-pg |
| Whole cluster down | silence — every path above, ntfy included, runs in this cluster; nothing self-hosted can report its own host's death | your own noticing; this is the known gap |
Why this is a topic, not a runbook
Each component page documents its piece — the monitoring page the stack, the Gatus page the probes, the ntfy page the delivery. What none of them shows is the routing decision in the middle: one Alertmanager config decides that critical means "phone + inbox" and everything else means "dashboard", and two of the three signal families never enter that funnel at all. When an incident retrospective asks "why didn't we get paged?", this page is the answer key: find the signal in the summary table, and either it was below critical, on a pull path, or the cluster hosting the entire pipeline was the thing that died.