Volume restore

How to get a volume's contents back. Databases have their own page, PostgreSQL backup & restore; this one covers everything that is backed up as files, plus the two special cases that are not: MongoDB and the SILO object store.

Which copy to restore from

Situation	Restore from	Why
Something broke in the last three days and the volume itself is healthy	A Longhorn snapshot	Local, instant, no download. Taken daily at 02:00, three kept.
Older damage, a lost or corrupted volume, a lost node set	The Restic repository on Hetzner	The primary backup. Daily, with weekly, monthly and yearly history.
Hetzner is unavailable	The Restic repository on Backblaze	Same data, second provider.

Longhorn snapshots live on the same disks as the volume. They undo a mistake; they are not a backup.

Restore from Restic

Every backed-up PVC appears in the app's repository under the path /data/<pvc-name>. The steps below use Navidrome and its navidrome-data volume as the example.

1. Pick the snapshot

NS=navidrome
kubectl get snapshots.k8up.io -n $NS \
  -o custom-columns=ID:.spec.id,DATE:.spec.date,PATHS:.spec.paths,REPO:.spec.repository

ID                 DATE                   PATHS                    REPO
f1192932b4b0…      2026-09-29T02:13:38Z   [/data/navidrome-data]   s3:hel1.your-objectstorage.com/workload-talos-navidrome-49cfe503
f018e229b6f9…      2026-10-04T12:12:05Z   [/data/navidrome-data]   s3:s3.eu-central-003.backblazeb2.com/workload-talos-navidrome-49cfe503

Each snapshot covers one path, so an app with three volumes has three snapshots per run. Note the ID of the one you want and which repository it is in; the first eight characters are enough.

2. Stop the workload

Nothing may write to the volume while it is restored:

kubectl scale deployment/navidrome -n $NS --replicas=0

3. Apply a Restore

apiVersion: k8up.io/v1
kind: Restore
metadata:
  name: restore-navidrome-data
  namespace: navidrome
spec:
  snapshot: "f1192932"
  restoreMethod:
    folder:
      claimName: navidrome-data
  podSecurityContext:
    runAsNonRoot: false
    runAsUser: 0
    seccompProfile:
      type: RuntimeDefault
  backend:
    repoPasswordSecretRef:
      name: k8up-restic-password
      key: password
    s3:
      endpoint: hel1.your-objectstorage.com
      bucket: workload-talos-navidrome-49cfe503
      accessKeyIDSecretRef:
        name: k8up-s3-credentials
        key: AWS_ACCESS_KEY_ID
      secretAccessKeySecretRef:
        name: k8up-s3-credentials
        key: AWS_SECRET_ACCESS_KEY

The bucket name is the one in the snapshot's repository column, or kubectl get schedule schedule-hetzner -n $NS -o jsonpath='{.spec.backend.s3.bucket}'. The job runs as root on purpose: it has to recreate files with whatever owner they had.

For a snapshot in the Backblaze repository, three fields change:

    s3:
      endpoint: s3.eu-central-003.backblazeb2.com
      bucket: workload-talos-navidrome-49cfe503
      accessKeyIDSecretRef:
        name: workspace-outputs
        key: backup_key_schedule_backblaze_id
      secretAccessKeySecretRef:
        name: workspace-outputs
        key: backup_key_schedule_backblaze_secret

4. Wait, then start the workload

kubectl get restore -n $NS -w
kubectl logs -n $NS -l k8upjob=true --tail=20
kubectl scale deployment/navidrome -n $NS --replicas=1

Delete the Restore object afterwards; it has done its job and would otherwise sit in the namespace.

What a restore does not do

Restic writes the snapshot's files into the volume and overwrites what is there, but it does not delete files that exist only in the volume. After a restore over a damaged volume, leftovers from the damaged state can remain. When that matters, restore into an empty volume instead: delete and recreate the PVC first, or use the drill below and swap the claims.

k8up runs the jobs of one repository one after another. A restore started while that app's backup, check or prune is running waits for it.

Drill: restore into a scratch volume

The same procedure, without touching the application. Create an empty claim next to the real one, restore into it, and look at the result:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restore-drill
  namespace: navidrome
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-encrypted
  resources:
    requests:
      storage: 2Gi

Point the Restore at claimName: restore-drill, then mount the claim in a throwaway pod and compare file counts and a few known files against the live volume. Delete the pod, the claim and the Restore when done. This is the quarterly exercise the disaster recovery page asks for.

MongoDB

The one MongoDB instance, used by Spotify, is not backed up as a volume. A k8up.io/backupcommand annotation streams a gzipped mongodump archive into the repository as /spotify-mongodb.gz. Restoring it follows the database pattern: fetch the file with Restic, replay it into the running pod.

Set up the Restic environment as in the PostgreSQL runbook, then:

NS=spotify
restic dump --path /spotify-mongodb.gz latest /spotify-mongodb.gz > spotify-mongodb.gz

kubectl scale deployment/server -n $NS --replicas=0
kubectl exec -i -n $NS mongodb-0 -- sh -c \
  'mongorestore --username="$MONGO_INITDB_ROOT_USERNAME" --password="$MONGO_INITDB_ROOT_PASSWORD" \
     --authenticationDatabase=admin --archive --gzip --drop' < spotify-mongodb.gz
kubectl scale deployment/server -n $NS --replicas=1

--drop removes each collection before restoring it, so the result matches the dump instead of merging with what is there.

SILO object store

Apps that use the minio component keep their objects on the volume data-minio-0 of the StatefulSet minio. It is backed up as files like any other PVC, so the Restic procedure above applies with two substitutions: scale the StatefulSet instead of a Deployment, and restore into data-minio-0.

kubectl scale statefulset/minio -n <namespace> --replicas=0
# apply the Restore with claimName: data-minio-0
kubectl scale statefulset/minio -n <namespace> --replicas=1

Restore the app's database from the same night. Objects and the rows that reference them only fit together when they come from the same backup run.

Longhorn snapshot revert

For a mistake made since the last few nights, reverting the volume is faster than any download. Longhorn only reverts a volume that is attached in maintenance mode, so the workload has to be down:

1. Scale the workload to zero and wait until the volume shows Detached in the Longhorn UI.
2. Attach the volume to any node with Maintenance ticked.
3. Open the volume, select the snapshot (named after the snapshot recurring job and its timestamp) and choose Revert.
4. Detach the volume and scale the workload back up.

Everything written after that snapshot is gone. If the data in between matters, take a manual snapshot first; it costs nothing and keeps the current state reachable.

Database volumes are the exception. A CNPG cluster has two instances with one volume each, and reverting one of them only makes the operator re-clone it from the other. Use the PostgreSQL runbook for databases.