SOPS

Every secret in this repository is committed encrypted. SOPS encrypts the values of a file and leaves its structure readable, age supplies the key pairs, and Flux decrypts on the cluster with a private key that never enters Git.

SOPS workflow for kubernetes

This page is the everyday workflow. Replacing a key is covered in SOPS / age key rotation, and getting a cluster back with nothing but the offline copies in Disaster recovery.

What gets encrypted

For Kubernetes Secrets, only the data and stringData values are encrypted. Kind, name and namespace stay in clear text, so diffs and reviews still show which Secret changed:

apiVersion: v1
kind: Secret
metadata:
    name: sopstest
data:
    foo: ENC[AES256_GCM,data:UZY1VQ==,iv:54ce6x…,tag:Ms906P…,type:str]
sops:
    age:
        - recipient: age1…
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            …
            -----END AGE ENCRYPTED FILE-----
    encrypted_regex: ^(data|stringData)$
    mac: ENC[AES256_GCM,data:Hhu+4T…,type:str]

The mac covers the whole file. Editing an encrypted file by hand, even in a clear-text field, invalidates it; always go through sops.

Keys

There is not one key but four, each scoped to the part of the repository it protects. The single .sops.yaml at the repository root picks the recipient by the file's path relative to the root, so sops works the same from any directory inside the repository and needs no flags.

Protects	Rule (`path_regex`)	Files	Decrypted by
Secrets of the talos cluster	`^k8s/(apps\|platform)/[^/]+/talos/`	every `secret*.yaml` under a `talos/` directory	Flux on talos, workstation
Secrets of the edge cluster	`^k8s/(apps\|platform)/[^/]+/edge/`	every `secret*.yaml` under an `edge/` directory	Flux on edge, workstation
Talos machine secrets	`^talos/`	`talos/<cluster>/talsecret.sops.yaml`	workstation only, when `topf` renders machine configs
OpenTofu credentials	`^tofu/`	`.env.sops` and `.auto.tfvars.sops` per environment	workstation only, when an environment is planned or applied

The rules deliberately have no catch-all. A secret placed outside a cluster directory, in base/ for example, fails to encrypt instead of silently getting the wrong key.

The same file sets stores.yaml.indent: 2, so sops writes secrets with the two-space indentation used everywhere else in the repository. A file edited by absolute path from outside the repository comes back four-space; the sops-indent commit hook (scripts/sops-reindent.py, also run by make fmt) puts it back without touching the encrypted content, and yamllint checks the result.

Each cluster holds only its own private key, as the Secret sops-age in flux-system. The workstation holds all four in ~/.config/sops/age/keys.txt, and so must the offline copies: losing the talos key locks every app secret, losing the machine-secrets key means the cluster's PKI cannot be re-rendered.

Install

SOPS ships as a single binary; age is packaged by every distribution.

curl -Lo sops https://github.com/getsops/sops/releases/download/v3.13.3/sops-v3.13.3.linux.amd64
sudo install -m 0755 sops /usr/local/bin/sops
sudo dnf install age        # or: apt-get install age

Put the private keys where SOPS looks for them. The file may hold several keys, one after another:

mkdir -p ~/.config/sops/age
cat keys.txt >> ~/.config/sops/age/keys.txt
chmod 600 ~/.config/sops/age/keys.txt

A new key pair is only needed for a new scope or a rotation: age-keygen -o keys.txt prints the public key, which goes into the matching .sops.yaml.

Everyday use

Create. Write the Secret in clear text inside the directory it belongs to, then encrypt it in place before staging it:

sops -e --in-place secret.yaml

Edit. sops edit decrypts into a temporary file, opens $EDITOR and re-encrypts on save. The clear text never touches the working tree:

sops edit secret.yaml

Read. Decrypt to stdout:

sops -d secret.yaml

Avoid sops -d --in-place inside the repository. It leaves a clear-text Secret one git add away from being pushed. Two guards catch that mistake: the pre-commit hooks run gitleaks and scripts/check-unencrypted-secrets.sh, which refuses any kind: Secret without a sops block, and CI repeats both.

If a file's MAC is broken, typically after a hand edit or a bad merge, sops -d --ignore-mac secret.yaml still decrypts it so the values can be recovered and the file re-encrypted cleanly.

Decryption on the cluster

Every Flux Kustomization that may contain secrets names the key to decrypt with:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: ntfy
  namespace: flux-system
spec:
  # …
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The sops-age Secret is the one object that is created by hand, once per cluster, from that cluster's private key:

kubectl -n flux-system create secret generic sops-age \
  --from-file=age.agekey=keys.txt

Secrets that many namespaces need (backup credentials, the SMTP account, registry pull secrets) are still SOPS files in Git. They are decrypted once into the secrets-hub namespace and fanned out from there by External Secrets.

Backup and recovery

The private keys exist in three places: on the clusters, on the workstation, and offline (printed, and on the cold-storage drives). The offline copies are what a rebuild starts from; Disaster recovery walks through re-seeding sops-age on a fresh cluster, and SOPS / age key rotation lists every copy that has to be updated when a key changes.

What gets encrypted​

Keys​

Install​

Everyday use​

Decryption on the cluster​

Backup and recovery​