Skip to main content

Three-tier backups, end-to-end

The homelab follows the 3-2-1 backup rule — three copies of every important byte, on two different media, with at least one off-site. This page traces the full path a byte takes from "written to a Longhorn PVC" to "rotated cold drive on a shelf", and what it costs to recover from each tier.

TierWhereDriverRPO (target)RTO (target)
LiveLonghorn PVC on the Talos clusterthe workload itselfn/an/a
WarmTwo Restic repositories per app: Hetzner Object Storage and Backblaze B2k8up → Restic≤ 24 hminutes-to-an-hour
HotOff-site Synology DS723+ ("Maresa")Syncthingsecondsseconds
ColdTwo rotating, encrypted WD Elements drivesmanual restic copy + rsync1–4 weekshours

What gets backed up vs. what doesn't​

A workload's volume is backed up when its PersistentVolumeClaim carries the annotation k8up.io/backup: "true". The matching Schedule resource in the backups component picks it up. A volume without the annotation is deliberately not backed up — regenerable state (search indices, transcode caches), bulk media that's mirrored separately, or anything where the snapshot cost outweighs the data's value.

Databases are not backed up as volumes. For every CNPG cluster the backups-cnpg component adds a PreBackupPod that streams pg_dump into the same run, so the snapshot holds a logical dump rather than a file-system view of a running database. The one MongoDB instance does the same with a k8up.io/backupcommand annotation that runs mongodump.

Synced datasets (the /volume1/backup/* paths on Maresa) use a different mechanism — Syncthing replicates the upstream peer continuously. Neither k8up nor Restic is involved on that path; the cluster is the source of truth and Maresa is a live replica.

Warm tier — k8up → Restic → Hetzner and Backblaze​

This is the primary recovery path. Every app has two Schedules from the backups component, schedule-hetzner and schedule-backblaze. They run independently, once a day each, into separate repositories at two providers, so the warm tier survives the loss of either one; Backup location: Backblaze covers the second target. Restic's content-addressed store gives:

  • Deduplication within each app's repository — daily snapshots of mostly unchanged volumes cost little more than the changes.
  • End-to-end encryption before the bytes leave the cluster. The repository password lives in a SOPS-encrypted Secret per app; the providers see only ciphertext.
  • Pruning policy that keeps the last 5 snapshots, 7 daily, 4 weekly, 12 monthly and 5 yearly. A weekly restic check verifies each repository and a weekly prune applies the retention.

Restoring from warm: a k8up Restore resource writes a volume snapshot back into a PVC, and the PostgreSQL backup & restore runbook replays a database dump through pv. Hetzner is the repository to restore from first; Backblaze holds the same data and is the fallback when Hetzner is unavailable. RTO is dominated by transfer speed from the provider — a 100 GB volume is on the order of 15–30 minutes.

Hot tier — Syncthing to Maresa​

The hot tier exists for the last few minutes of work that the warm-tier snapshot hasn't caught yet, plus for bulk datasets (media libraries, etc.) where a continuous replica is cheaper than scheduled snapshots.

  • Bidirectional by default. A correction made on Maresa flows back to the source. Useful for the archive pool.
  • Versioning per folder ("Staggered" policy) gives hour-by-hour, then daily, then weekly file history, even on Maresa. So a delete on the upstream doesn't immediately destroy the replica's previous state.
  • No central server. Each peer holds its own keys; the Synology can be unreachable for days without breaking the upstream.

Maresa is in a different physical location from the cluster — the "off-site" leg of 3-2-1. The Syncthing folder on Maresa is also the source for the cold tier below.

Restoring from hot: copy the relevant directory back from Maresa over Syncthing (or just over the NetBird mesh directly). RTO measured in seconds for individual files; minutes for large datasets at LAN-ish speeds.

Cold tier — encrypted rotating drives​

Two WD Elements drives, each formatted as a VeraCrypt volume, rotated monthly to a different physical location. One drive is always offline; the other is in transit or being written. See Hardware → Cold storage for the drive list and the encryption setup.

  • Air-gapped. A cluster-wide ransomware event can't touch a drive that's sitting on a shelf with no USB plugged in.
  • Encrypted at rest. A lost or stolen drive is useless without the passphrase.
  • Targeted, not exhaustive. Only the "important" subset of /volume1/backup/* makes it onto the cold tier — irreplaceable content (photos, documents, source code) — not bulk media that can be re-acquired.

The cold tier is the slowest, lowest-RPO, highest-RTO tier. It's the answer to every other tier has been compromised — not the path you reach for to roll back a deploy.

When each tier matters​

Failure modeTier that recovers it
Bad deploy, app data corrupted todaywarm (last night's Restic snapshot; everything since is lost)
App data lost on a single PVC, more than 24 h agowarm (daily/weekly snapshot)
Cluster unreachable, but Maresa OKhot (Syncthing replica is reachable directly)
Hetzner outagewarm (the Backblaze repository)
Repo password lost or mistakenly rotatedhot or cold, whichever has the freshest copy
Cluster and both warm providers lostcold (offline drive)
Drive theft / lossthe other drive in rotation

The whole point of having three tiers is that any single compromise leaves the other two intact.

Operational rules of thumb​

  • Don't tier-shift on a deploy. Backups happen on their own k8up Schedules, unrelated to GitOps. Resist the urge to "snapshot before deploy" — see topics/gitops-flow for why.
  • Periodically rotate the cold drives. A monthly cadence is what's documented; further drift turns the cold tier from a safety net into a museum.
  • Test restores. A backup that's never been restored is a hope, not a backup. The postgres-restore runbook is the canonical drill; pick an app's PVC quarterly and run through it on a throwaway namespace.
  • Watch repo password handling. Lose the Restic repository password and the warm tier is cryptographically gone — that's the failure mode the cold tier exists to survive.

See also​