SOPS / age key rotation

The age key in flux-system/sops-age is the load-bearing piece of the secrets pipeline. Every committed Secret resource in the repo is encrypted to it; Flux's kustomize-controller decrypts inline at apply time. Rotating it is a tightrope walk — do it wrong and the cluster stops reconciling because the next Secret Flux sees is encrypted to a key it doesn't have.

This page is the procedure that's been used successfully and what to watch for.

When to rotate

Three reasons to actually do it:

Suspected key compromise. A workstation seizure, a stolen backup drive, an accidental paste — anything that means the private key may have left the safe set of hands.
Periodic hygiene. Once a year, even when nothing's wrong. The drill keeps the procedure rehearsed; the rotated key is freshly generated rather than worn-in.
Onboarding a second key for redundancy. Adding a new recipient and not removing the old one — same procedure, simpler — is also covered below.

Don't rotate as part of any other change. Rotating the key + changing manifests in the same flight is how you end up with a cluster that can't decrypt and can't roll back in one step.

Trust boundary

There is not one age key but four, each scoped to a part of the repository: the talos cluster's secrets, the edge cluster's secrets, the Talos machine secrets and the OpenTofu credentials. SOPS → Keys lists which rule file selects which. They rotate independently, and the procedure below is written for one of them at a time.

A cluster key has three copies that must all stay in sync after a rotation:

Copy	Used by
`flux-system/sops-age` Secret on the cluster	Flux kustomize-controller — decrypts in-flight
Operator's local `~/.config/sops/age/keys.txt`	`sops` CLI on the workstation — encrypts new Secrets, decrypts for editing
Printed copy + cold-drive copy (offline)	Disaster recovery — see DR drill, prerequisites

A rotation is "done" only when all three reflect the new key. The machine-secrets key and the OpenTofu key never reach a cluster, so for those the first row and step 4 fall away, and step 5 is a local decrypt instead of a reconciliation: workstation and offline copies are all there is.

Procedure

1. Generate a new key, locally

age-keygen -o /tmp/age-new.key

This emits a public/private pair. Take note of the public recipient (age1…); you'll add it as a SOPS recipient in step 2.

2. Add the new key as a second recipient

Edit the rule file of the key being rotated. For a cluster key that is the matching rule in both k8s/apps/.sops.yaml and k8s/platform/.sops.yaml; for the other two it is talos/.sops.yaml or tofu/.sops.yaml. The relevant block looks like:

creation_rules:
  - path_regex: "[^/]+/talos/"
    encrypted_regex: "^(data|stringData)$"
    age: age1OLD…   # existing recipient

Change it to both:

creation_rules:
  - path_regex: "[^/]+/talos/"
    encrypted_regex: "^(data|stringData)$"
    age: >-
      age1OLD…,
      age1NEW…

This is the key insight: don't remove the old recipient yet. Every Secret in the repo can now be re-encrypted to both recipients, so either key can decrypt it. The cluster (still holding the old key) keeps reconciling.

3. Re-encrypt every committed Secret

Walk the files that key protects and sops updatekeys each one. For the talos cluster key:

for f in $(grep -rl '^sops:' k8s --include='*.yaml' | grep '/talos/'); do
  sops updatekeys -y "$f"
done

For the edge key the filter is /edge/. The machine-secrets key covers talos/*/talsecret.sops.yaml, the OpenTofu key the .env.sops and .auto.tfvars.sops files under tofu/environment/.

updatekeys reads each file, decrypts it with whichever recipient you currently hold, and re-encrypts to whatever the .sops.yaml block now says — i.e. both keys.

Commit the result. Flux pulls the new commit; nothing changes in the actual ciphertext payload's content, only its envelope. Reconciliation stays green.

4. Install the new key on the cluster

Update the in-cluster Secret to add the new key alongside the old:

# Get the current key
kubectl -n flux-system get secret sops-age -o jsonpath='{.data.age\.agekey}' \
  | base64 -d > /tmp/current.key

# Concatenate
cat /tmp/current.key /tmp/age-new.key > /tmp/both.key

# Patch the Secret
kubectl -n flux-system create secret generic sops-age \
  --from-file=age.agekey=/tmp/both.key \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart Flux to pick up the new key (or wait for next reconciliation)
kubectl -n flux-system rollout restart deployment kustomize-controller

sops and age both accept multiple private keys in ~/.config/sops/age/keys.txt and try each in order. Same on the cluster — Flux now decrypts using whichever key the file was encrypted to.

5. Verify

Force a reconciliation and watch for Failed Kustomizations:

flux reconcile kustomization --all
flux get kustomizations -A | grep -v True

If anything went wrong, the old key still works — you can roll back by reverting the .sops.yaml change and sops updatekeys-ing again.

6. Remove the old recipient

Once verification is clean and you've slept on it (literally — don't do this in the same session as steps 1–5), drop the old key.

Edit the same rule file(s) as in step 2:

creation_rules:
  - encrypted_regex: '^(data|stringData)$'
    age: age1NEW…

Re-encrypt:

for f in $(grep -rl '^sops:' k8s --include='*.yaml' | grep '/talos/'); do
  sops updatekeys -y "$f"
done

Commit. Flux pulls; the new ciphertexts can only be opened by the new key. The old key still works locally for now (because keys.txt and the cluster Secret both still contain it), but Flux no longer needs it.

7. Wipe the old key

Now and only now, drop the old key everywhere:

# Local — leave only the new key in keys.txt
mv ~/.config/sops/age/keys.txt ~/.config/sops/age/keys.txt.bak
cp /tmp/age-new.key ~/.config/sops/age/keys.txt

# Cluster
kubectl -n flux-system create secret generic sops-age \
  --from-file=age.agekey=/tmp/age-new.key \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n flux-system rollout restart deployment kustomize-controller

8. Update the offline copies

The DR drill assumes the offline copies of the age key match production. Update them — see the DR drill prerequisites for which copies exist, and make sure the offline set still holds all four keys, not only the one just rotated.

What can go wrong, and the failure mode

Mistake	Visible symptom	Fix
Removed old key from `.sops.yaml` before re-encrypting	`sops` complains on every `updatekeys`; commit not possible	Add the old key back temporarily, run `updatekeys`, then proceed
Removed cluster key before commit reached cluster	`flux get kustomizations` shows `Failed` decrypting Secrets	Add the old key back to the in-cluster Secret; restart kustomize-ctl
Forgot to update offline copies	DR drill silently fails on step 3	Caught during the yearly DR rehearsal
Lost both old and new key with re-encrypted ciphertexts	The warm-tier Restic password is now unreachable; cold tier	The cold-tier drives are the only recovery path

The first two are recoverable in minutes. The last one is the whole reason the cold tier exists.

Frequency

Annually as documented hygiene.
Immediately on suspected compromise.
Never opportunistically tied to another change.

The drill takes about 30 minutes once you've done it twice; longer the first time as you re-read this page.

When to rotate​

Trust boundary​

Procedure​

1. Generate a new key, locally​

2. Add the new key as a second recipient​

3. Re-encrypt every committed Secret​

4. Install the new key on the cluster​

5. Verify​

6. Remove the old recipient​

7. Wipe the old key​

8. Update the offline copies​

What can go wrong, and the failure mode​

Frequency​

See also​