Skip to main content

Disaster recovery drill

This is the end-to-end path from the Talos production cluster is gone to every app is running again with its data. The trigger doesn't matter — four dead Proxmox nodes, a botched upgrade, ransomware, an exuberant rm — the procedure is the same.

Two properties of the setup shape the whole procedure:

  • The cluster's state is not backed up; it is rebuilt. There is no etcd snapshot. Everything Kubernetes knows comes back from Git through Flux, and everything apps know comes back from the Restic repositories. A recovered cluster is a new cluster with the old data.
  • Most apps are delivered from Gitea, and Gitea runs on the cluster. Flux reads the platform, Gitea and the runner straight from the Git repository on Codeberg, but 53 app Kustomizations pull signed OCI artifacts from Gitea's registry. So the order is fixed: platform, then Gitea, then everything else.

Expect most of a working day. Most of it is waiting for downloads and reconciliation, not active work.

Prerequisites — what survives a full disaster​

The recovery only works because these live outside the production cluster:

  1. The Git repository. Flux follows the Codeberg mirror (ssh://git@codeberg.org/johnnycube/homelab.git), not Gitea, precisely so that a dead cluster cannot take its own source of truth with it. The workstation clone is the second copy and may be a few commits ahead of the mirror.
  2. The age keys. There are four, see SOPS. The talos cluster key decrypts every Secret Flux applies, the machine-secrets key decrypts talsecret.sops.yaml for Talos, and the OpenTofu key decrypts the provider credentials. Outside the workstation they exist printed on paper in a safe and on the cold-storage drives.
  3. The Flux deploy key for the Codeberg repository. If it is lost, generate a new one and register it on Codeberg; nothing else depends on it.
  4. The backup repositories. Two per app, on Hetzner Object Storage and on Backblaze B2, see Backups end to end. Their Restic passwords and the S3 credentials are SOPS-encrypted in Git, so the age key is all it takes to read them.
  5. The OpenTofu state in the tofu-state-558a4a64 bucket on Hetzner. It also holds the state of the per-app workspaces, which is why the rebuilt cluster adopts the existing backup buckets and keys instead of trying to create them again.
  6. The cold-tier drives, for the unlikely case that both object-storage providers are unrecoverable.

Without the Git repository or the talos age key there is no recovery, only a fresh start. Everything else on the list can be replaced or worked around.

The procedure​

1. Compute​

OpenTofu recreates the seven Proxmox VMs: three control planes and four workers. If Proxmox itself is gone, that is a hypervisor reinstall first, which this drill does not cover.

Each environment keeps its credentials SOPS-encrypted next to the code; decrypt them, then go through the tofu.sh wrapper, which loads .env:

cd tofu/environment/production
sops -d --input-type dotenv --output-type dotenv .env.sops > .env
sops -d --input-type binary --output-type binary .auto.tfvars.sops > .auto.tfvars

./tofu.sh init
./tofu.sh plan -out=plan
./tofu.sh apply plan

The VMs boot the Talos nocloud image their disk was cloned from and wait in maintenance mode, each on the address from its cloud-init drive.

2. Talos​

topf renders the machine configs from talos/talos, decrypting talsecret.sops.yaml on the way, so the new cluster has the same CA and the same identity as the old one.

cd talos/talos
topf render && topf apply --dry-run     # look before you leap
topf apply --auto-bootstrap             # joins the nodes, bootstraps etcd on the first control plane
topf upgrade                            # installs the SecureBoot metal installer the cluster is meant to run

topf talosconfig > talosconfig
topf kubeconfig > ~/.kube/config

The upgrade is not optional. A node that was only applied still runs the nocloud image; the metal installer is what gives it the NIC names the Cilium L2 policy expects. Disk encryption is sealed to each VM's vTPM and can only be enabled on empty partitions, which fresh VMs have.

kubectl get nodes should now list seven nodes, NotReady until Flux installs the CNI.

3. Trust​

One Secret has to exist before Flux can do anything useful: the talos cluster's age key.

kubectl create namespace flux-system
kubectl -n flux-system create secret generic sops-age \
  --from-file=age.agekey=/path/to/recovered/talos.agekey

Nothing else is seeded by hand. Restic passwords, object-storage credentials, registry pull secrets and NetBird setup keys are all SOPS files in Git; Flux decrypts them with this key, and External Secrets fans the shared ones out.

Without the key every Kustomization that contains a Secret fails to decrypt and the cluster idles.

4. Flux​

Bootstrap against Codeberg, with the same path the cluster always used:

flux bootstrap git \
  --url=ssh://git@codeberg.org/johnnycube/homelab.git \
  --branch=main \
  --path=k8s/clusters/talos \
  --private-key-file=/path/to/flux_deploy_key

If the workstation clone is ahead of the mirror, push it to Codeberg first; otherwise the cluster comes back a few commits in the past.

Flux verifies the commit signature of every revision it applies (commit signature verification); the keys it checks against are part of the same directory, so nothing extra is needed as long as the HEAD being bootstrapped was signed by one of them.

5. Wait for the platform to converge​

flux get kustomizations --watch

The platform Kustomizations are Git-sourced and come up on their own: Cilium first, then storage, certificates, gateways, the operators, and finally infra-configs, which everything else waits for. Image pulls dominate the wait.

At this point the app Kustomizations are expected to be failing. Their OCIRepository sources point at gitea.web.kueber.eu, which does not exist yet. Leave them; they retry.

Public traffic reaches the cluster through the edge cluster, which is unaffected by a loss of the production site and resumes forwarding as soon as the gateways answer.

6. Bring Gitea back​

Gitea and its runner are the two apps Flux reads from Git instead of from the registry, so they are deployed by now, with an empty volume and an empty database. Restore both from the Hetzner repository.

Stop Gitea, find the bucket and pick the snapshots:

NS=gitea
kubectl scale deployment/gitea -n $NS --replicas=0
BUCKET=$(kubectl get schedule schedule-hetzner -n $NS -o jsonpath='{.spec.backend.s3.bucket}')
kubectl get snapshots.k8up.io -n $NS \
  -o custom-columns=ID:.spec.id,DATE:.spec.date,PATHS:.spec.paths,REPO:.spec.repository

On a fresh cluster the snapshot list fills once k8up has synced the repository. Restore the data volume with a Restore resource; this is the shape that works here, including the root security context the job needs to write files of any owner:

apiVersion: k8up.io/v1
kind: Restore
metadata:
  name: restore-gitea-data
  namespace: gitea
spec:
  snapshot: "<snapshot id>"
  restoreMethod:
    folder:
      claimName: gitea-data
  podSecurityContext:
    runAsNonRoot: false
    runAsUser: 0
    seccompProfile:
      type: RuntimeDefault
  backend:
    repoPasswordSecretRef:
      name: k8up-restic-password
      key: password
    s3:
      endpoint: hel1.your-objectstorage.com
      bucket: <bucket from the schedule>
      accessKeyIDSecretRef:
        name: k8up-s3-credentials
        key: AWS_ACCESS_KEY_ID
      secretAccessKeySecretRef:
        name: k8up-s3-credentials
        key: AWS_SECRET_ACCESS_KEY

Then replay the database dump with the PostgreSQL backup & restore runbook and scale Gitea back up.

The registry's packages live on the same gitea-data volume, so the OCI artifacts return with it and the app sources turn Ready within their five-minute poll. If an artifact is missing, the build-oci-apps workflow rebuilds and signs all of them from the repository; start it by hand from the Actions tab once the runner is registered again.

7. Restore the apps​

With the registry back, Flux deploys every app, each with empty volumes and an empty database. Go through them in order of importance, Keycloak first since it gates every login. Per app the steps are the ones just used for Gitea:

  1. Scale the workload to zero.
  2. Apply one Restore per backed-up PVC. The backups component page explains which volumes are included; kubectl get snapshots.k8up.io shows the paths.
  3. Replay the database dump, if the app has one.
  4. Scale the workload back up.

Restores into different namespaces run in parallel; k8up serialises the jobs that share a repository. To restore from Backblaze instead, swap the endpoint for s3.eu-central-003.backblazeb2.com and take the key pair from the workspace-outputs Secret, as shown in the runbook.

Apps without a backup come back empty by design; their pages say so under Data Management.

8. Verify​

Gatus going green is the first signal, not the last. Before calling it done:

  • Log in to one app through Keycloak and open real data in two or three others.
  • Check that the first scheduled backups of the new cluster succeed: kubectl get snapshots.k8up.io -A should show fresh dates the next morning.
  • Confirm flux get kustomizations has no entry that is not Ready.

If a specific app doesn't come up:

  • Pod stuck Pending — PVC not bound yet, or Longhorn is still building replicas.
  • Pod stuck CrashLoopBackOff — most often a database that hasn't been restored yet.
  • HTTPRoute 503 — Envoy Gateway is up but the backend isn't healthy. Same root cause as the above.

The edge cluster​

The edge cluster is rebuilt the same way, in miniature: tofu/environment/edge recreates the Hetzner server and reattaches the floating IP, topf runs from talos/edge, the sops-age Secret holds the edge age key, and Flux is bootstrapped with --path=k8s/clusters/edge.

What does not come back is data. The edge cluster has no k8up and no backup target today, and two of its apps hold state on the node's local disk: Seitenraum and Openbeehive. Losing the server means losing their databases and uploads.

While the edge cluster is down, every public hostname is down with it, including the ones served by the production cluster, because all public traffic enters there.

DNS​

Public zones are hosted at ALL-INKL and described in OpenTofu; they are unaffected by either cluster going away. The internal records under *.int.kueber.eu are NetBird DNS records created by each app's workspace, so they reappear as the apps do. The handful of *.maresa.int.kueber.eu rewrites live in AdGuard Home on the Synology and are maintained in its UI; they are outside this drill and outside Git.

Practice schedule​

The honest advice: practice it before you need it.

  • Quarterly: restore one random PVC and one database into a throwaway namespace and verify the data.
  • Yearly: run the full procedure against a sandbox cluster to catch documentation drift.

When the actual disaster hits, the documentation written a year ago will not be the documentation that is needed. The drill is what keeps the gap small enough to close in hours.

See also​