Skip to main content

backups-cnpg (component)

A Kustomize component that gives an app's k8up backups a logical database dump. It drops a single PreBackupPod into the namespace; before every run of the backups schedules, k8up runs pg_dump inside that pod against the app's CNPG cluster and writes the output into the same Restic snapshot as the app's PVCs, as one .sql file. Filesystem copies of a live Postgres data directory are not restorable with confidence; a dump is.

The component exists for the Talos cluster only (k8s/components/backups-cnpg/talos). There is no edge overlay, because the edge cluster has no k8up.

What it composes​

One object, PreBackupPod cnpg-app-pgdump:

  • Pod template running the same ghcr.io/cloudnative-pg/postgresql image tag as the postgresql-trixie ClusterImageCatalog the CNPG component points at, so Renovate bumps client and server in one PR and pg_dump never drifts ahead of or behind the server. The container's own command is sleep infinity; the pod only idles until k8up executes into it.
  • Connection from the CNPG-generated Secret. PGHOST, PGPORT, PGDATABASE, PGUSER and PGPASSWORD come from the cnpg-app Secret that the operator creates for the cluster named cnpg. The host in it is cnpg-rw, so the dump always comes from the current primary.
  • backupCommand — a short shell loop that waits on pg_isready and then execs pg_dump -d app --clean. --clean prefixes the dump with DROP statements, so it restores into a database that already has the schema.
  • fileExtension: .sql — the object lands in the snapshot as <namespace>-cnpg-pgdump.sql (namespace plus container name), restorable with plain psql.
  • Identity of the CNPG instance pods: runAsUser/runAsGroup 26 (postgres), non-root, all capabilities dropped, RuntimeDefault seccomp. The pod needs the network and stdout, nothing else.

How an app uses it​

Apps pull it in from their backups/ sub-component, next to the backups component that provides the Schedule objects:

# k8s/apps/<app>/talos/backups/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

components:
  - ../../../../components/backups/talos
  - ../../../../components/backups-cnpg/talos

Nothing needs patching. The component assumes what the cnpg component provides: a Cluster named cnpg (hence the cnpg-app Secret and the cnpg-rw Service) and the default app database. An app whose data lives in a differently named database would need its own PreBackupPod.

Twenty-two apps use it today, among them Gitea, Immich, Keycloak, Memos, Nextcloud, Paperless, Prefect and Vaultwarden — every app that combines the cnpg and backups components.

How a run works​

  1. The k8up backup Job for a schedule finds the PreBackupPod objects in its namespace and starts the pod from the template.
  2. It runs backupCommand in that pod through the Kubernetes API (pods/exec) and streams the command's stdout into Restic. This is why the backups component ships an egress-k8up-apiserver NetworkPolicy: without API-server egress from the job pods, every backup that has a pre-backup pod fails.
  3. The per-PVC part of the run follows, and the pre-backup pod is torn down afterwards. The pod exists only for the duration of the run.

The pre-backup pod itself reaches cnpg-rw on 5432 through the intra-namespace allowance of the app-network-policy baseline; no extra policy is needed.

Why wait on pg_isready, and why never retry pg_dump​

The manifest records the reason in place. Roughly once or twice a week a pre-backup pod could not resolve cnpg-rw ("Temporary failure in name resolution"), and the whole run was lost even though the next scheduled run succeeded. The backupCommand therefore polls pg_isready -q — up to twelve times, five seconds apart — before it dumps exactly once. The short-lived variant of that failure becomes a non-event; a genuinely unreachable database still fails, just later and with the wait visible in the log.

The wait is on pg_isready, never on pg_dump, and the choice is load-bearing: the command's stdout is the Restic stream. Retrying pg_dump after a partial dump would append a second dump to the first and silently corrupt the snapshot. pg_isready -q writes nothing to stdout, so it cannot pollute that stream either.

The wait is 60 seconds, down from the 300 it started as. Evidence from the 2026-09-24 Nextcloud failure: the fault lasted hours, so every one of k8up's retry attempts burned the full wait before failing anyway. The long wait bought nothing and made the run hold its repository longer. Sixty seconds still covers the case the wait is for — a database that is a moment behind its pre-backup pod.

Why the whole run is capped at 30 minutes​

The cap does not live in this component but in the Schedule objects of the backups component, and it exists because of this one: backup.activeDeadlineSeconds: 1800. k8up hardcodes backoffLimit: 6 on the Jobs it creates and does not expose it, so a pre-backup that keeps failing retries for hours — Nextcloud's 2026-09-24 pg_dump ran from 02:26 until BackoffLimitExceeded at 05:00. k8up serialises jobs per repository, so that also held the repository and pushed the week's check 33 minutes late.

Thirty minutes is deliberately generous, not tight. The longest successful k8up job across the retained history took 142 seconds, with a median around 22, and a deadline kill is the same "killed backup pod" condition that truncated repository objects in August 2026. It is a backstop against a multi-hour hang, not a throughput limit. k8up sets the deadline on every Job it spawns, so it applies to the pre-backup and the per-PVC Jobs alike.

Operational notes​

  • A failed pre-backup is quiet. The per-PVC Jobs of the same run still succeed, so the snapshot holds the app's volume data and no database. The monitoring rules carry a dedicated K8upPreBackupFailed alert for exactly this, because the next successful run restores cover and clears the alert — check the gap before it disappears. Judge backup health from snapshots.k8up.io and k8up_schedule_last_job_succeeded, not from the retained Backup objects: successfulJobsHistoryLimit: 0 deletes successes at once while failures stay.
  • Logs. The pre-backup pod is gone after the run; the backup Job relays the pod's stderr into its own log, which is where a pg_dump error shows up.
  • Restore is the PostgreSQL backup & restore runbook: fetch <namespace>-cnpg-pgdump.sql from the snapshot and stream it into the primary with psql. The disaster-recovery drill covers the from-nothing case.
  • Image coupling. When Renovate bumps the postgresql-trixie catalog, the same PR bumps this pod's image; the tags stay identical by design. Do not pin them apart.
  • Removing the component keeps the PVC backups running but drops the dump from every future snapshot. A snapshot without the dump is no longer a complete restore point for the app.

Cluster-specific deviations from the above live next to the manifests — see k8s/components/backups-cnpg/talos.

Cluster Deployment​

Depends on