Backblaze B2 backup location

The warm tier writes to two independent restic targets: Hetzner Object Storage and Backblaze B2 (eu-central-003) — deliberately different providers, so no single incident can reach the workloads and all their backups at once. This page documents the Backblaze target and, more importantly, how the three operators involved hand work to each other so that no bucket and no per-app credential is ever created by hand.

Concern	Owner	Where
Declaring "this app backs up to B2"	k8up `Schedule` (shared component)	`k8s/components/backups/base/schedule-backblaze.yaml`
Per-app bucket naming	Kustomize `PrefixSuffixTransformer` chain	`k8s/components/backups/talos/` + per-app `backups/`
Creating the bucket + minting a bucket-scoped key	Crossplane `TofuWorkspace` composition	`k8s/platform/configs/talos/crossplane/composition-tofu-workspace.yaml`
Delivering the minted key to k8up	OpenTofu workspace connection secret	`workspace-outputs` (per namespace)
Distributing the one human-managed account key	External Secrets `ClusterExternalSecret`	`k8s/platform/configs/talos/secrets-hub/ces-backblaze-account-creds.yaml`

The flow

The Schedule is the single source of truth. Its spec.backend.s3.bucket — already rewritten per app by the transformer chain to workload-talos-<app>-<id> — names the bucket the composition creates; its endpoint decides which provider block handles it. Both Schedules in a namespace produce the same bucket name on different providers, which keeps restore procedures symmetric.

The Schedule

schedule-backblaze sits next to schedule-hetzner in the shared base component, so every app that includes the backups component gets it automatically — same PVC annotations, same CNPG pre-backup dumps, same per-app restic password (k8up-restic-password), separate repository:

s3:
  endpoint: s3.eu-central-003.backblazeb2.com
  bucket: workload            # transformer chain appends -talos-<app>-<id>
  accessKeyIDSecretRef:
    name: workspace-outputs
    key: backup_key_schedule_backblaze_id
  secretAccessKeySecretRef:
    name: workspace-outputs
    key: backup_key_schedule_backblaze_secret

workspace-outputs is not in Git — it is produced by the workspace (below). The region comes from the k8up-backblaze-config Secret (AWS_DEFAULT_REGION: eu-central-003) via backend.envFrom; that Secret is intentionally plaintext and allowlisted in scripts/check-unencrypted-secrets.sh.

One app-level detail: audiobookshelf overrides the backup-job security context. Its patch applies to all Schedules via a target: kind: Schedule selector in apps/talos/audiobookshelf/backups/kustomization.yaml, so both targets run with the same UID.

Provisioning: pure b2 provider

The composition (see the TofuWorkspace pattern) splits the namespace's Schedules by endpoint. Hetzner schedules keep the aminueza/minio S3 path; Backblaze schedules render against the first-party Backblaze/b2 provider — bucket and credential:

resource "b2_bucket" "backup_bucket_schedule_backblaze" {
  bucket_name = "workload-talos-<app>-<id>"
  bucket_type = "allPrivate"
  lifecycle { prevent_destroy = true }
}

resource "b2_application_key" "backup_key_schedule_backblaze" {
  key_name     = "workload-talos-<app>-<id>"
  bucket_ids   = [b2_bucket.backup_bucket_schedule_backblaze.bucket_id]
  capabilities = ["listBuckets", "listFiles", "readFiles", "writeFiles", "deleteFiles"]
}

The two outputs (..._id, ..._secret — the latter sensitive = true) land in the workspace connection secret, named {{ $xr.metadata.name }}-outputs → workspace-outputs, because every namespace's TofuWorkspace is named workspace.

listBuckets is not optional: bucket-restricted B2 keys without it break restic's S3 backend (and most S3 SDKs) on the compatibility API.

Credential model

There is exactly one human-managed credential, and a deliberate asymmetry in who holds what:

Credential	Scope	Held by	Managed by
`backblaze-account-credentials`	account-wide, management capabilities only (`listBuckets, readBuckets, writeBuckets, listKeys, writeKeys, deleteKeys` — no file access)	workspace env (`TF_VAR_backblaze_*`) in `secrets.tofu-workspace: "true"` namespaces	SOPS in `secrets-hub`, fanned out by `ces-backblaze-account-creds.yaml`
per-bucket application key	one bucket, file operations only	`workspace-outputs` in the app namespace, consumed by k8up	minted/rotated by the workspace
restic repository password	one repo	`k8up-restic-password`, per app	SOPS, per app (`apps/talos/<app>/backups/secret-k8up.yaml`)

Consequences:

The provisioning key cannot read or delete backup data — it lacks all file capabilities.
A leaked per-app key is confined to its own bucket and cannot escalate (bucket-restricted keys cannot hold writeKeys).
The account key can escalate by minting itself a stronger key (writeKeys), so treat it as account-root regardless of trimmed capabilities — which is why it is fenced to workspace namespaces instead of riding with the k8up credentials the way k8up-s3-credentials does for Hetzner.
The account key must be created with the B2 CLI/API (b2 key create workspace-provisioner listBuckets,readBuckets,writeBuckets,listKeys,writeKeys,deleteKeys); console-created keys can never carry writeKeys/deleteKeys.

Adding the hub secret also required whitelisting it in the secrets-hub Role — the ClusterSecretStore reads hub Secrets through a resourceNames list in k8s/platform/configs/talos/secrets-hub/rbac.yaml. A missing entry surfaces as SecretSyncedError on every ExternalSecret, not as a store-level error.

Bootstrap ordering

None. The first k8up run in a namespace fails on the missing workspace-outputs; the workspace reconciles (10-minute poll), creates bucket and key, publishes the secret; the next scheduled run succeeds. Every controller retries — no orchestration is needed or wanted.

Verifying

# workspace rendered and applied
kubectl get workspaces.opentofu.m.upbound.io -A

# minted key arrived (two backup_key_* entries expected)
kubectl get secret workspace-outputs -n gitea -o jsonpath='{.data}' | jq keys

# both schedules active, snapshots landing on B2
kubectl get schedules -n gitea
kubectl get backups -n gitea --sort-by=.metadata.creationTimestamp | tail -3

Operational notes

First-backup init race (CNPG namespaces). On an empty repo, the backup job pod and k8up's prebackup exec runner both race restic init; the loser can leave an orphan key that poisons the repo (Fatal: config or key <id> is damaged: ciphertext verification failed, crashlooping *-prebackup-* pod). Fix: list keys/ in the bucket, delete the key object that doesn't match the surviving config's timestamp, and re-run the backup (the first run also lacks the pg_dump snapshot). Prevention: seed new repos with a one-off restic init before their first scheduled run — the race only exists against an empty repo. The backupcommand annotation on CNPG pods is not an alternative: inheritedMetadata would annotate replicas too, and dumps race replication recovery.
Account region is fixed. A B2 account is pinned to one region at signup; b2_bucket has no region argument. The Schedule endpoint and AWS_DEFAULT_REGION must match the account's region (eu-central-003).
Key rotation ≠ sops edit. Rotating a per-app key means recreating the b2_application_key resource (taint or remove from state) — the new key flows to workspace-outputs automatically. Rotating the account key is a sops edit in secrets-hub.
Import by bucket ID. Adopting a pre-existing B2 bucket into workspace state uses the bucket ID, not the name (unlike the minio/S3 resources).
Bucket names are globally unique on B2. The -<8-hex> suffix from the transformer chain makes collisions unlikely; a collision surfaces as a workspace apply error.
prevent_destroy is set on the bucket — removing an app's backups requires an out-of-band destroy, by design.

The flow​

The Schedule​

Provisioning: pure b2 provider​

Credential model​

Bootstrap ordering​

Verifying​

Operational notes​

See also​