Skip to main content

SOPS / age key rotation

The age key in flux-system/sops-age is the load-bearing piece of the secrets pipeline. Every committed Secret resource in the repo is encrypted to it; Flux's kustomize-controller decrypts inline at apply time. Rotating it is a tightrope walk — do it wrong and the cluster stops reconciling because the next Secret Flux sees is encrypted to a key it doesn't have.

This page is the procedure that's been used successfully and what to watch for.

When to rotate​

Three reasons to actually do it:

  1. Suspected key compromise. A workstation seizure, a stolen backup drive, an accidental paste — anything that means the private key may have left the safe set of hands.
  2. Periodic hygiene. Once a year, even when nothing's wrong. The drill keeps the procedure rehearsed; the rotated key is freshly generated rather than worn-in.
  3. Onboarding a second key for redundancy. Adding a new recipient and not removing the old one — same procedure, simpler — is also covered below.

Don't rotate as part of any other change. Rotating the key + changing manifests in the same flight is how you end up with a cluster that can't decrypt and can't roll back in one step.

Trust boundary​

The age key has three copies that must all stay in sync after a rotation:

CopyUsed by
flux-system/sops-age Secret on the clusterFlux kustomize-controller — decrypts in-flight
Operator's local ~/.config/sops/age/keys.txtsops CLI on the workstation — encrypts new Secrets, decrypts for editing
Printed copy + cold-drive copy (offline)Disaster recovery — see DR drill, step 3

A rotation is "done" only when all three reflect the new key.

Procedure​

1. Generate a new key, locally​

age-keygen -o /tmp/age-new.key

This emits a public/private pair. Take note of the public recipient (age1…); you'll add it as a SOPS recipient in step 2.

2. Add the new key as a second recipient​

Edit .sops.yaml at the repo root. The relevant block looks like:

creation_rules:
  - encrypted_regex: '^(data|stringData)$'
    age: age1OLD…   # existing recipient

Change it to both:

creation_rules:
  - encrypted_regex: '^(data|stringData)$'
    age: >-
      age1OLD…,
      age1NEW…

This is the key insight: don't remove the old recipient yet. Every Secret in the repo can now be re-encrypted to both recipients, so either key can decrypt it. The cluster (still holding the old key) keeps reconciling.

3. Re-encrypt every committed Secret​

Walk the repo and sops updatekeys every encrypted file:

for f in $(grep -rl 'sops:' --include='*.yaml' --include='*.sops.yaml'); do
  sops updatekeys -y "$f"
done

updatekeys reads each file, decrypts it with whichever recipient you currently hold, and re-encrypts to whatever the .sops.yaml block now says — i.e. both keys.

Commit the result. Flux pulls the new commit; nothing changes in the actual ciphertext payload's content, only its envelope. Reconciliation stays green.

4. Install the new key on the cluster​

Update the in-cluster Secret to add the new key alongside the old:

# Get the current key
kubectl -n flux-system get secret sops-age -o jsonpath='{.data.age\.agekey}' \
  | base64 -d > /tmp/current.key

# Concatenate
cat /tmp/current.key /tmp/age-new.key > /tmp/both.key

# Patch the Secret
kubectl -n flux-system create secret generic sops-age \
  --from-file=age.agekey=/tmp/both.key \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart Flux to pick up the new key (or wait for next reconciliation)
kubectl -n flux-system rollout restart deployment kustomize-controller

sops and age both accept multiple private keys in ~/.config/sops/age/keys.txt and try each in order. Same on the cluster — Flux now decrypts using whichever key the file was encrypted to.

5. Verify​

Force a reconciliation and watch for Failed Kustomizations:

flux reconcile kustomization --all
flux get kustomizations -A | grep -v True

If anything went wrong, the old key still works — you can roll back by reverting the .sops.yaml change and sops updatekeys-ing again.

6. Remove the old recipient​

Once verification is clean and you've slept on it (literally — don't do this in the same session as steps 1–5), drop the old key.

Edit .sops.yaml:

creation_rules:
  - encrypted_regex: '^(data|stringData)$'
    age: age1NEW…

Re-encrypt:

for f in $(grep -rl 'sops:' --include='*.yaml' --include='*.sops.yaml'); do
  sops updatekeys -y "$f"
done

Commit. Flux pulls; the new ciphertexts can only be opened by the new key. The old key still works locally for now (because keys.txt and the cluster Secret both still contain it), but Flux no longer needs it.

7. Wipe the old key​

Now and only now, drop the old key everywhere:

# Local — leave only the new key in keys.txt
mv ~/.config/sops/age/keys.txt ~/.config/sops/age/keys.txt.bak
cp /tmp/age-new.key ~/.config/sops/age/keys.txt

# Cluster
kubectl -n flux-system create secret generic sops-age \
  --from-file=age.agekey=/tmp/age-new.key \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n flux-system rollout restart deployment kustomize-controller

8. Update the offline copies​

The DR drill assumes the offline copies of the age key match production. Update them — see DR drill, step 3 for which copies exist and where they live.

What can go wrong, and the failure mode​

MistakeVisible symptomFix
Removed old key from .sops.yaml before re-encryptingsops complains on every updatekeys; commit not possibleAdd the old key back temporarily, run updatekeys, then proceed
Removed cluster key before commit reached clusterflux get kustomizations shows Failed decrypting SecretsAdd the old key back to the in-cluster Secret; restart kustomize-ctl
Forgot to update offline copiesDR drill silently fails on step 3Caught during the yearly DR rehearsal
Lost both old and new key with re-encrypted ciphertextsThe warm-tier Restic password is now unreachable; cold tierThe cold-tier drives are the only recovery path

The first two are recoverable in minutes. The last one is the whole reason the cold tier exists.

Frequency​

  • Annually as documented hygiene.
  • Immediately on suspected compromise.
  • Never opportunistically tied to another change.

The drill takes about 30 minutes once you've done it twice; longer the first time as you re-read this page.

See also​