Skip to content

zot

Where Flux fetches every app from, outside the cluster it delivers to.

zot is a single-binary OCI registry: the distribution spec and nothing else unless you switch extensions on. Here it holds the artifacts build-oci-apps publishes, one per app overlay (oci-talos-app-<name>, oci-edge-app-<name>), each a kustomize build of main, signed with the homelab cosign key and tagged :edge plus :sha-<commit>. Flux on both clusters pulls :edge, verifies the signature and applies it. Container images stay on Gitea’s registry and the upstream ones; zot carries only the rendered manifests, a few hundred kilobytes in total.

It runs on the edge cluster rather than at home on purpose. Flux on talos must be able to deliver apps while Gitea, OpenBao or Keycloak are down or being rebuilt, and those come back as artifacts themselves. A registry inside the talos cluster would be one of the things waiting to come back; the Hetzner node is not.

Alternatives considered

The artifacts lived on Gitea’s registry until November 2026, then on a private Codeberg account. On 2026-11-12 Codeberg shut its registry off for every user, because registry traffic was slowing down the whole site, with no date for its return. Nothing broke, since Flux keeps applying the last artifact it has, but nothing new reached either cluster for the rest of the day.

Cloud Hosted

ToolOpen SourceFree TierMonthly Cost
Codeberg packagesYesYesFree; 2000 requests per 10 minutes, cleanup rules only in the web UI, shut off on 2026-11-12
GitHub Container RegistryNoYesFree; ruled out, the homelab stays off GitHub
Scaleway Container RegistryNoNoCents for a few hundred KB; an account and a vendor more

Self Hosted

ToolOpen SourceFull FeaturesNotes
Gitea packagesYesYesAlready deployed, but inside the cluster it delivers to: a rebuild would wait on itself
DistributionYesYesThe reference registry; one htpasswd realm in which every user can push, and no retention
HarborYesYesDatabase, Redis and half a dozen services for a few hundred kilobytes

Installation

Architecture

  • Deployment: single zot deployment in the zot namespace on the edge cluster, Recreate strategy, because the data volume is ReadWriteOnce and zot’s metadata is a bolt file only one process may open
  • Image: ghcr.io/project-zot/zot, version and digest pinned, updated by Renovate. The full build, for the web UI, the search extension the UI runs on and cosign signature trust; CVE scanning, sync, scrubbing, linting and the other extensions stay off in config.json, so zot never downloads a vulnerability database or calls out
  • Configuration: config.json in the base, turned into a hash-suffixed ConfigMap so any change restarts the pod
  • Storage: zot-data, a 2Gi local-path volume holding the blobs and the metadata database
  • Networking: ClusterIP service on port 5000; an HTTPRoute on the public gateway’s own oci.web.kueber.eu listener, which terminates TLS at Envoy

Security

  • Authentication is HTTP Basic over TLS, one user per client, no anonymous access. flux-talos may read oci-talos-app-*, flux-edge may read oci-edge-app-*, ci may read, create and update both (detect pulls :edge to compare, build pushes, signs and moves the tag), and johnny may read both, for browsing in the UI. Nobody may delete; retention is the only thing that removes data. A failed login waits five seconds (failDelay), and the passwords are 32 to 40 random characters. The UI and its search API sit behind the same login and show each user only the repositories it may read
  • Why static credentials. Flux 2.10 can authenticate an OCIRepository with a Docker config Secret (secretRef), a TLS client certificate (certSecretRef) or the workload identity of AWS, Azure and GCP; there is no way yet to hand a generic registry its ServiceAccount token. zot can already verify such tokens (bearer.oidc), and Flux’s RFC-0014 on short-lived credentials is where the other half is tracked. A client certificate would also be a long-lived secret, but Envoy would have to pass TLS through to zot, and flux push in CI cannot present one. The integrity guarantee never came from the login anyway: Flux refuses any artifact without a valid cosign signature from the homelab key, so the password only keeps the SOPS ciphertext and the cluster layout private
  • Where the credentials live. The bcrypt hashes are in the SOPS Secret zot-htpasswd. Each Flux password is a SOPS regcred-zot-oci Secret in k8s/platform/configs/<cluster>/flux/, deliberately not an ExternalSecret, because OpenBao’s own artifact is pulled with it. The ci and johnny logins are in OpenBao, one secret each with username and password, at kv/homelab/apps/zot/ci and kv/homelab/apps/zot/johnny; CI reads only its own, through the gitea-runner-oci role
  • The signing key never leaves OpenBao. CI signs each artifact with the transit key flux-artifacts in OpenBao: cosign sends the digest and gets the signature back, and every signature is in OpenBao’s audit log. The private key cannot be exported, so there is no copy and no password to keep, and a CI job can only sign while its five-minute token lasts. The public half has three copies in Git: the cosign-public-key Secret of each cluster’s Flux in k8s/platform/configs/<cluster>/flux/, and k8s/apps/zot/base/cosign/flux-artifacts.pub for the UI. External Secrets cannot deliver them, because OpenBao’s own artifact is verified with them and the edge cluster has no External Secrets
  • Runs as 65532, runAsNonRoot: true, readOnlyRootFilesystem: true, all capabilities dropped, no ServiceAccount token
  • Ingress only from the public gateway’s Envoy pods; no egress beyond the network-policy baseline

Updates

Managed by Renovate; the digest pin moves with the tag. zot keeps its storage layout across minor releases, and the configuration keys are what changes between them, so the release notes are worth a glance before a bump.

Data Management

Garbage collection runs once a day. The retention policy keeps every :edge tag and every semver release tag, plus the ten most recently pushed sha-* tags per repository, which are the rollback targets. Untagged manifests go after 24 hours, and the cosign signature of a manifest goes with it; the signature of a kept manifest stays, because zot files legacy sha256-….sig tags under their subject instead of counting them as tags of their own. There are no backups: every artifact is a render of main that CI can publish again, so after a loss the registry is refilled by running build-oci-apps.

User Management

Four users in zot-htpasswd and no self-service; the UI only logs in the users that exist there. To rotate one, generate a password, put its bcrypt hash (htpasswd -nB <user>) in zot-htpasswd and the password where its client reads it: the cluster’s regcred-zot-oci for a Flux user, kv/homelab/apps/zot/<user> for ci and johnny. Push both in the same commit. A Secret change does not restart the pod; zot watches the htpasswd file and reloads it, and kubectl -n zot rollout restart deploy/zot settles it if a reload was missed.

Signing Key

To replace the key, make a new version of it in OpenBao; CI signs with the newest version from its next run on. Put that version’s public key next to the old one in all three copies and push right away, run build-oci-apps with rebuild_all once both clusters have the new Secret, then remove the old key. Flux accepts an artifact signed by any key in its Secret, and zot trusts every file in _cosign, so nothing stops on the way; an artifact signed before a cluster has the new key waits for the next interval. Rollback tags signed with the old key stop verifying when it goes. Last, empty the registry and fill it again: the repository list shows the first artifact each repository ever got, which still carries the old signature and would read Unknown for good. Suspend the zot Kustomization on edge, delete the Deployment and the zot-data claim, resume it, and run build-oci-apps with rebuild_all. Running apps are not affected, and Flux sources fail to fetch only until the refill reaches them.

bao write -f transit/keys/flux-artifacts/rotate
bao read -format=json transit/keys/flux-artifacts | jq -j '.data.keys[.data.latest_version | tostring].public_key'

An OpenBao restored from a raft snapshot has the key back. One that lost its data without a snapshot has the transit mount from its self-initialisation but no key: create it with bao write -f transit/keys/flux-artifacts type=ecdsa-p256 and follow the same steps, removing the old key at once, since nothing can sign with it any more.

Configuration Management

Everything is in config.json: storage and retention under storage, users and their rights under http.accessControl. Access control matches repository names with globs, so a new app needs no change here as long as it keeps the oci-<cluster>-app-<name> naming.

Administration

The web UI at oci.web.kueber.eu lists every repository, its tags with digests, push times and sizes, the layers of each manifest and its referrers, which is where a cosign signature shows up. The trust extension checks each signature against the public key Flux verifies with, mounted read-only from the zot-cosign-keys ConfigMap over _cosign in the data directory, so a signature from the homelab key reads as trusted and names that key as its signer; anything else stays Unknown. zot re-checks every signature at start and every two hours, and new ones as they arrive. The key file is one of the three copies of the signing key’s public half (see Signing Key). It does not open the gzipped layer itself, so the rendered YAML of an artifact is a flux pull away rather than a click. /livez, /readyz and /startupz answer without a login and back the probes. Anything else needs credentials; the registry can be inspected with the same tools CI uses:

flux pull artifact oci://oci.web.kueber.eu/oci-talos-app-voyage:edge --output /tmp/voyage --creds "johnny:<password>"
cosign verify --key k8s/apps/zot/base/cosign/flux-artifacts.pub oci.web.kueber.eu/oci-talos-app-voyage:edge

zot logs every retention and garbage-collection decision with the tag, digest and reason, which is where to look when a tag went missing.

Usage

Nobody uses zot by hand. A push to main that changes an overlay’s render makes build-oci-apps publish that overlay, and the OCIRepository of the app picks up the new :edge within five minutes. A rebuild of every app is gt workflow run build-oci-apps.yaml -R johnny/homelab --ref main -f rebuild_all=true; without the input, a run only publishes what changed or is missing, which after a lost volume is everything. Each artifact carries a title, a description naming its overlay, the vendor kueber.eu homelab, the public repository as its source and its docs page, which is what the UI lists for it. None of them changes between builds, on purpose: the repository list shows the first artifact a repository ever got, because zot ranks images by the creation time in their config and a Flux artifact’s config has none. The commit of each build is in its revision annotation and in the tag list.

Cluster-specific deviations from the above live in the per-cluster README — see k8s/apps/zot/edge/README.md.

Cluster Deployment

Depends on

zot — Edge cluster

Deviations from defaults

  • Delivered from Git, not from an artifact — k8s/clusters/edge/zot.yaml points at the flux-system GitRepository, like gitea-runner on talos: the registry cannot pull itself. build-oci-apps still renders this overlay into oci-edge-app-zot, which nothing reads.
  • Hostname inside the passthrough wildcard — oci.web.kueber.eu is covered by the *.web.kueber.eu DNS record, so no record had to be added. The public-kueber gateway gives it its own HTTPS listener (https-oci-web-kueber-eu, certificate from the all-inkl DNS-01 issuer) next to the web-tls passthrough to the home cluster; Envoy picks the exact SNI before the wildcard. A split-horizon DNS that answers *.web.kueber.eu with the home VIPs must leave this one name alone, or Flux on talos would ask its own gateway for the registry. Browsers reuse an open HTTP/2 connection to any other *.web.kueber.eu site for this name, since the home cluster’s wildcard certificate covers it too, and so land on the home gateway; that gateway answers 421 Misdirected Request for oci.web.kueber.eu (k8s/platform/configs/talos/envoy-gateway/http-route-misdirected-oci.yaml), and the browser retries on a connection of its own.
  • No backups — zot-data is a local-path volume without a k8up annotation. Everything in it is a render of main plus its signature: after a loss, run build-oci-apps (gt workflow run build-oci-apps.yaml -R johnny/homelab --ref main), whose detect step takes a missing :edge for a first build and republishes every overlay. Only the sha-* history is gone.

Kubernetes Metadata

  • Image: ghcr.io/project-zot/zot:v2.1.23@sha256:9d3de83ac2099e6f4843045b4c9b65b7e8baf40f92d28f54906dee9d4d390fd7
Rendered manifests (kustomize build)
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: zot
  name: zot
  namespace: zot
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: zot
  strategy:
    rollingUpdate:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: zot
    spec:
      automountServiceAccountToken: false
      containers:
      - args:
        - serve
        - /etc/zot/config/config.json
        env:
        - name: GOMEMLIMIT
          value: 400MiB
        image: ghcr.io/project-zot/zot:v2.1.23@sha256:9d3de83ac2099e6f4843045b4c9b65b7e8baf40f92d28f54906dee9d4d390fd7
        livenessProbe:
          httpGet:
            path: /livez
            port: http
          periodSeconds: 20
        name: zot
        ports:
        - containerPort: 5000
          name: http
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /readyz
            port: http
          periodSeconds: 10
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 10m
            memory: 128Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: true
        startupProbe:
          failureThreshold: 24
          httpGet:
            path: /startupz
            port: http
          periodSeconds: 5
        volumeMounts:
        - mountPath: /etc/zot/config
          name: config
          readOnly: true
        - mountPath: /etc/zot/auth
          name: auth
          readOnly: true
        - mountPath: /var/lib/registry
          name: data
        - mountPath: /var/lib/registry/_cosign
          name: cosign-keys
          readOnly: true
        - mountPath: /tmp
          name: tmp
      securityContext:
        fsGroup: 65532
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 65532
        runAsNonRoot: true
        runAsUser: 65532
        seccompProfile:
          type: RuntimeDefault
      volumes:
      - configMap:
          name: zot-config-kt9fkmbk9k
        name: config
      - name: auth
        secret:
          secretName: zot-htpasswd
      - name: data
        persistentVolumeClaim:
          claimName: zot-data
      - configMap:
          name: zot-cosign-keys-t9cd9ch8k9
        name: cosign-keys
      - emptyDir: {}
        name: tmp
Last updated on