Skip to main content

The image supply chain: from upstream registry to a running node

A container image in this homelab lives in two forms that must never disagree: a line of text in Git (image: foo/bar:1.2.3@sha256:…) and a pile of layer blobs on a node's disk. This page follows both — how the text changes (Renovate, CI, merge) and how the bytes move (containerd, Spegel, peer-to-peer) — and what happens when the upstream registry that started it all goes away.

The deploy loop itself (Flux pulling main, rendering, applying) is covered in GitOps flow; the merge rules are covered in Renovate policy. This page is about the image specifically: why every reference is a digest, and what a node actually does when it needs one.

ActorWhere it lives in docs
Renovate (scheduled on gitea-runner)apps/gitea-runner
Gitea (Git host and in-cluster OCI registry)apps/gitea
CI workflows (.gitea/workflows/)apps/gitea-runner
Flux (applies the new pod spec)foundation/flux
containerd on Talos (does the actual pull)foundation/talos
Spegel (in-cluster P2P registry mirror)platform/spegel

The chain, top-down​

Steps 1–4 move text. Steps 5–7 move bytes. The digest is the handshake between the two worlds: whatever containerd ends up pulling, from whatever source, must hash to exactly what the manifest in Git says.

Digest pinning: the manifest names exact bytes​

Every workload image in k8s/ is pinned to both a tag and a digest:

# k8s/apps/miniflux/base/deployment.yaml
image: miniflux/miniflux:<tag>-distroless@sha256:<digest>

The tag is for humans (what version is this?); the digest is for machines (which bytes are these?). Pinning comes from config:best-practices in renovate.json — Renovate adds the digest on first sight and keeps it current afterwards. The one deliberate exception is the flux manager (pinDigests: false in renovate.json), which covers Flux's own component manifests.

The consequences run through the whole chain:

  • A registry can't change what you run. A re-pushed tag upstream produces a PR, not a silently different pod after the next node reboot.
  • Every image change is a Git commit. "When did this binary change?" is answered by git log, not by registry forensics.
  • Any mirror is trustworthy by construction. containerd verifies the digest of whatever it pulls — which is exactly what makes a peer-to-peer mirror like Spegel safe to insert into the pull path. A peer can at worst fail to serve an image; it cannot serve a wrong one.

Renovate proposes, never deploys​

Renovate runs on the gitea-runner pods and watches everything under k8s/** via the kubernetes, flux and helm-values managers, plus regex managers for # renovate: annotations and the Talos/Kubernetes versions in topf.yaml. A new upstream digest becomes a one-line PR.

Digest-only bumps auto-merge on a branch in the overnight window (automergeSchedule: * 0-7 * * *); anything semver waits for a human. The full ruleset and its reasoning live on the Renovate policy page — the point here is only that nothing in the image supply chain moves without a commit.

CI validates text it never runs​

The PR triggers .gitea/workflows/check.yaml on the runners, which runs make check: flux-local test and flux-local build render every Flux Kustomization for both clusters (talos and edge), then kubeconform schema-checks the output, and every app overlay under k8s/apps/ is rendered with kustomize and checked the same way. No kind cluster, no image pull — a digest bump is validated purely as manifest text, which is all it is.

There is a second, less obvious lane. On every push to main, .gitea/workflows/build-oci-apps.yaml renders each app's talos/ and edge/ overlay under k8s/apps/<app>/ with kustomize, validates it, and — only if the rendered output actually differs from the current :edge artifact — pushes it as a signed OCI artifact to the in-cluster Gitea registry (oci://gitea.web.kueber.eu/johnny/oci-<cluster>-app-<name>), cosign-signing the digest. Apps deployed this way use a Flux OCIRepository with verify.provider: cosign, so the kustomize-controller refuses artifacts that don't carry a valid signature. Platform controllers and a few apps (including gitea-runner itself) still reconcile straight from the GitRepository.

So the cluster's own registry is a product of the supply chain as well as a link in it: the CI jobs themselves run inside digest-pinned build images hosted on that same Gitea registry. That circularity is fine day-to-day and a known constraint during disaster recovery, when Gitea isn't up yet and everything must come from upstream.

From merge to kubelet​

Flux notices the new commit (or the new OCI artifact), renders, and applies the changed pod spec — the mechanics are the GitOps flow, steps 4–6. From the supply chain's perspective only one thing matters: a pod spec now exists whose image: names a digest no node has, and the kubelet hands that reference to containerd.

The pull, on the node​

This is where Talos and Spegel come in.

containerd's registry config. Talos keeps containerd's per-registry host config under /etc/cri/conf.d/hosts. The Spegel Helm values set exactly that path:

# k8s/platform/spegel/talos/spegel-values.yaml
spegel:
  containerdRegistryConfigPath: /etc/cri/conf.d/hosts

Spegel runs as a DaemonSet (namespace spegel, pod security privileged — it needs host networking and the containerd socket) and registers itself as a mirror for external registries in that config. From then on, containerd tries the local Spegel endpoint first for any pull, and falls back to the original upstream registry if the mirror can't serve it.

The peer-to-peer part. Spegel is stateless and has no storage of its own. When asked for an image, it resolves — over its peer-to-peer protocol — which node in the cluster already has the blobs in its containerd content store, and streams them from there. One node pays the internet egress; every other node pulls over the LAN.

Keeping the layers around. By default containerd discards compressed layer blobs after unpacking them, which would leave Spegel nothing to serve. A Talos machine-config patch turns that off on every node:

# talos/talos/patch-spegel.yaml
machine:
  files:
    - path: /etc/cri/conf.d/20-customization.part
      op: create
      permissions: 0o644
      content: |
        [plugins."io.containerd.cri.v1.images"]
          discard_unpacked_layers = false

The patch is talos/talos/all/60-spegel.yaml, a CRICustomizationConfig document that topf applies to all seven production nodes. The edge cluster is a single node — nothing to peer with — so it runs neither Spegel nor the patch.

Verification. Whichever path the bytes took — peer or upstream — containerd hashes them against the digest from the pod spec. A corrupt or tampered blob fails the pull; it cannot fail open.

What the mirror buys​

  • Registry outage resilience. If docker.io or ghcr.io is down (or rate-limiting), any image that exists on at least one node keeps being schedulable everywhere. Rollouts of already-running versions, pod rescheduling after a node drain, and DaemonSet churn all keep working through an upstream outage. Only genuinely new digests are blocked.
  • Dedup across nodes. A cluster-wide DaemonSet update or a Talos node upgrade re-pulls the same images on many nodes. With Spegel the upstream sees one pull; the other five nodes fill from their neighbour. This is why the disaster recovery drill notes that platform recovery is image-pull-dominated only "until Spegel catches up its mirror".
  • Faster rollouts. The image-pull share of a rolling update drops to LAN speed for every node after the first — the reason GitOps flow can quote 30–90-second app rollouts.

What it deliberately does not buy​

  • It is a cache, not an archive. Spegel serves only what some node's containerd already holds. Garbage-collected images, images from before the last full cluster rebuild, and brand-new digests all require the upstream registry. If you need an image to survive independently of upstream, that's a registry-hosting question (Gitea can do it), not a Spegel question.
  • It doesn't protect first pulls during bootstrap. A freshly reinstalled cluster has empty content stores; the first reconciliation pulls everything from the internet.
  • It doesn't cover the edge cluster. Single node, no peers, direct upstream pulls only.

What can break, and where to look​

SymptomMost likely causeWhere to look first
Renovate stops opening image PRsgitea-runner down, or Renovate job failingapps/gitea-runner · runner pod logs
Digest-bump PR fails CIrendered manifests invalid — usually a shared base, not the digest itselfflux-local build output in the failed workflow run
App Kustomization stuck, source shows verification errorOCI artifact signature missing/invalid (cosign key or unsigned push)flux get sources oci -A · the build-oci-apps run
flux-system source shows a verification error, nothing new appliesHEAD commit unsigned or signed by a key not in flux-commit-signersflux get sources git -A; Flux → commit signature verification
Pods ImagePullBackOff on a new digest, cluster-wideupstream registry outage or rate limit — no peer has the new bytes yetkubectl describe pod events; try again later or pin previous digest
Pulls slow everywhere, upstream fineSpegel DaemonSet unhealthy — nodes silently fall back to upstreamkubectl -n spegel get ds · platform/spegel
Peers never serve images (all pulls hit upstream)discard_unpacked_layers patch missing after a node rebuildtalos/talos/all/60-spegel.yaml; topf apply --dry-run
CI containers fail to pull their build imagein-cluster Gitea registry unavailable or credentials rotatedapps/gitea · DOCKER_REGISTRY_GITEA_* secrets

Note the failure direction: Spegel breaking degrades to "every pull goes to the internet" — slower and outage-sensitive, but functional. The reverse failure (upstream down, Spegel healthy) degrades to "only known images run" — which is exactly the state a stable cluster spends most of its life in anyway.

See also​