Monitoring

Unified VictoriaMetrics + VictoriaLogs observability stack with Grafana, deployed from a single chart.

This namespace deploys the full observability stack for the homelab cluster from one Helm chart — victoria-metrics-k8s-stack. Both metrics and logs stay inside the VictoriaMetrics family: VictoriaMetrics stores metrics, VictoriaLogs stores logs, and Grafana queries both through provisioned datasources. Self-hosting this stack avoids cloud observability costs while providing full access to all cluster telemetry.

The stack was consolidated onto VictoriaMetrics-only in July 2026: Loki, Fluent Bit, and the OpenTelemetry Collector were retired in favour of vlsingle + vlagent, and the Prometheus operator CRDs were dropped in favour of annotation-driven scraping.

Alternatives considered

Cloud Hosted

Tool	Open Source	Free Tier	Monthly Cost
Grafana Cloud	Yes	Limited	From $19/mo
Datadog	No	No	From $15/host
New Relic	No	Limited	Pay-as-you-go

Installation

Architecture

• HelmRelease: victoria-stack — the victoria-metrics-k8s-stack chart (nameOverride: vm), which bundles every component below into one release:
  • victoria-metrics-operator — reconciles the VM CRDs; admission webhooks are served with cert-manager certificates. The Prometheus-operator converter is disabled because those CRDs are not installed in this cluster.
  • vmsingle — single-node metrics storage, 3-month retention, 40Gi Longhorn-encrypted PVC.
  • vlsingle — single-node VictoriaLogs storage, 8-day retention, 20Gi Longhorn-encrypted PVC.
  • vmagent — scrapes metrics and remote-writes to vmsingle. Scraping is annotation-driven (see Usage) plus built-in jobs for kubelet, kube-apiserver, etcd, kube-state-metrics, and node-exporter.
  • vlagent — DaemonSet log collector (k8sCollector) that tails /var/log/containers on every node and ships to vlsingle. No third-party collector.
  • vmalert — evaluates alerting rules; with both stores enabled it routes queries through an internal vmauth, so MetricsQL and LogsQL rules both work.
  • alertmanager — routes alerts to email over the all-inkl SMTP relay.
  • grafana — dashboards, with the VictoriaMetrics metrics + logs datasource plugins and a dashboard sidecar that discovers grafana_dashboard-labelled ConfigMaps cluster-wide.
• Additional: grafana-to-ntfy Deployment proxies alerts to ntfy; a k8up-alerts VMRule alerts on failed/missing Restic backups.
• Storage: Longhorn-encrypted PVCs only — vmsingle (40Gi) and vlsingle (20Gi). No object storage.
• Networking: HTTPRoutes for Grafana, vmsingle (metrics), vmagent, vlsingle (logs), and Alertmanager — see Usage.

Security

• vmsingle and vlsingle run as runAsNonRoot: true, UID/GID 1000.
• vlagent runs as a DaemonSet with blanket NoSchedule/NoExecute tolerations; /var/log is mounted read-only with HostToContainer propagation (a non-default logsPath stops the operator from injecting its own hostPath mounts, which would otherwise wedge Longhorn dm-crypt volumes on detach).
• All secrets SOPS-encrypted with age.

Updates

Chart and images managed by Renovate. grafana-to-ntfy is digest-pinned.

Data Management

• PVCs: Longhorn-encrypted PVCs for vmsingle (metrics, ~27Gi at current ingest) and vlsingle (logs).
• Retention: 3 months of metrics, 8 days of logs. Metrics retention was tuned down from 8 months, which overflowed the 40Gi volume.
• Backups: No k8up schedule — observability data is transient and durability rests on Longhorn replication.

User Management

Grafana OIDC configured via GF_AUTH_GENERIC_OAUTH_* env vars from a SOPS-encrypted grafana secret. Users authenticate through the cluster's OIDC provider.

Configuration Management

• Chart values live in the victoria-stack-values ConfigMap (referenced via valuesFrom).
• Grafana OIDC credentials, SMTP credentials, and ntfy auth come from SOPS-encrypted secrets.
• Alertmanager routing and the k8up-alerts rules are defined alongside the release.

Usage

Grafana is the primary entry point — dashboards, ad-hoc MetricsQL queries against VictoriaMetrics, and LogsQL queries against VictoriaLogs, all through provisioned datasources. The VM component UIs are also exposed directly:

Service	Host	Backend
Grafana	grafana.talos.int.kueber.eu	victoria-stack-grafana
Metrics (vmsingle)	metrics.talos.int.kueber.eu	vmsingle-vm-stack:8428
Agent (vmagent)	agent.talos.int.kueber.eu	vmagent-vm-stack:8429
Logs (vlsingle)	logs.talos.int.kueber.eu	vlsingle-vm-stack:9428
Alertmanager	alertmanager.talos.int.kueber.eu	vmalertmanager-vm-stack:9093

To have an app scraped, annotate its Pods (or Service) with prometheus.io/scrape: "true", optionally prometheus.io/port and prometheus.io/path. vmagent discovers them automatically — no ServiceMonitor/PodMonitor needed. Alerts fire through Alertmanager to email and are mirrored to ntfy via the grafana-to-ntfy proxy.

Cluster-specific deviations from the above live in the per-cluster README — see k8s/apps/monitoring/talos/README.md.

Cluster Deployment

App URLs

• agent.talos.int.kueber.eu
• alertmanager.talos.int.kueber.eu
• grafana.talos.int.kueber.eu
• logs.talos.int.kueber.eu
• metrics.talos.int.kueber.eu

Depends on

• Envoy Gateway

Monitoring — Talos cluster

Deviations from defaults

• kube-state-metrics image pinned by digest (victoria-stack/victoria-stack-values.yaml) — verify-kubernetes-addon-images runs with verifyDigest: true, so the tag-only subchart default is rejected at admission on the next rollout.

Kubernetes Metadata

• HelmRelease: victoria-metrics-k8s-stack@0.96.0
• HelmRepo: victoria-stack (oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-k8s-stack)
• Image: kittyandrew/grafana-to-ntfy:2026.9.16@sha256:ae86166ae4449cd0aa5158045018452cc91e0684bb27334d7ec59f472cdaa4c8

Rendered manifests (kustomize build)

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/force: enabled
  labels:
    app.kubernetes.io/name: grafana-to-ntfy
  name: grafana-to-ntfy
  namespace: monitoring
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: grafana-to-ntfy
  strategy:
    rollingUpdate: null
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: grafana-to-ntfy
    spec:
      automountServiceAccountToken: false
      containers:
        - envFrom:
            - secretRef:
                name: grafana-to-ntfy
          image: kittyandrew/grafana-to-ntfy:2026.9.16@sha256:ae86166ae4449cd0aa5158045018452cc91e0684bb27334d7ec59f472cdaa4c8
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          name: grafana-to-ntfy
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            failureThreshold: 5
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          resources:
            limits:
              memory: 64Mi
            requests:
              cpu: 25m
              memory: 64Mi
      priorityClassName: tier-1
      terminationGracePeriodSeconds: 60