BuildKit
One shared BuildKit daemon that builds and pushes every container image CI produces.
BuildKit is the build engine behind docker build and docker buildx: it turns a Dockerfile into a graph of steps, runs them in parallel where it can, caches every step by its inputs, and exports the result — to a registry, a tarball or a local directory. Docker embeds it, but it also runs standalone as buildkitd, reachable over gRPC, and buildx can drive such a daemon through its remote driver as if it were local.
That standalone daemon is what CI builds on. The Gitea runner runs every job as a plain Kubernetes pod without a Docker daemon, so a job cannot build an image itself; it hands the build to buildkitd instead. One daemon serves all of them.
Why one central daemon
The alternative designs each move the problem somewhere worse:
| Option | What it costs |
|---|---|
| Docker-in-Docker per runner (the old setup) | A privileged sidecar next to every runner, any job one breakout away from its node, and a cache that dies with each pod |
| BuildKit sidecar per job pod | Still privileged, now in the namespace where the runner creates pods; no cache shared between builds |
| Kaniko / Buildah in the job | Unprivileged, but a different build tool with its own Dockerfile quirks, no buildx, every workflow rewritten |
| Rootless BuildKit | Needs unprivileged user namespaces, which Talos turns off (user.max_user_namespaces = 0) |
A central daemon keeps the one component that really needs privileges in a namespace of its own, gives every build the same warm layer cache, and changes each workflow by two lines: driver: remote and an endpoint.
Installation
Architecture
- Deployment
buildkitd, one replica, rootfulmoby/buildkitpinned by tag and digest. It listens ontcp://0.0.0.0:1234for the job pods and on the default unix socket for its own probes (buildctl debug workers). - Service
buildkitdon port 1234; the runner hands its address to every job asBUILDKIT_HOST(tcp://buildkitd.buildkit.svc.cluster.local:1234). - Layer cache on its own volume under
/var/lib/buildkit, so a restart or an image update starts warm. BuildKit’s garbage collection bounds it in percentages of that volume: usage stays undermaxUsedSpace(80 %) with 10 % free, and never shrinks belowreservedSpace(20 %), so growing the claim grows the cache without a config change. Builds that want their cache on any builder still use the registry (cache-to: type=registry). - Host network mode for RUN steps (
networkMode = "host"). Here “host” means the daemon’s own pod: build steps share its network namespace, so they inherit the pod MTU and the pod’s network policies. A bridge would need its MTU tuned below the WireGuard overlay by hand, as the old dind sidecar did with--mtu=1000. max-parallelism = 4caps concurrent build steps across all clients, so eight simultaneous CI jobs queue for the node’s CPUs instead of fighting over them.Recreateupdates.buildkitdlocks its state directory and the volume isReadWriteOnce, so two daemons never overlap: on SIGTERM the old one stops taking connections and lets running builds finish within the 15-minute termination grace period, then the new one mounts the cache. Builds started in that window fail to connect and need a re-run.
Security
- Privileged, alone. Rootful BuildKit creates namespaces and mounts for every build step, which needs
privileged: true. Thebuildkitnamespace runs at Pod Securityprivilegedwith nothing else in it, which is what lets the CI namespace sit atbaseline. - No entitlements.
network.hostandsecurity.insecurestay disabled, so a Dockerfile cannot ask for a privileged build step or the host network. - The network policy is the access control. The gRPC port has no TLS and no client authentication; only pods labelled as Gitea Actions jobs in the
gitea-runnernamespace may connect. Anyone who can reach port 1234 can run arbitrary build steps, so widening that rule is a security decision, not a convenience. - Credentials never live in the daemon. Registry logins stay in the job (
~/.docker/config.json); buildx forwards them over the build session only for the duration of a push or a registry cache import. - Egress to the internet on 80/443 (base images, package managers inside RUN steps) and to the gateway, which carries pushes to the Gitea registry.
Updates
Renovate bumps the image tag and digest. The cache volume normally carries over to the new daemon; should a new version reject the old state, delete the claim — nothing in it needs keeping.
Data Management
None worth keeping. The cache is reproducible by definition, so the volume is a single replica and nothing is backed up. If the node holding that replica is gone for good, delete the claim: the next pod provisions an empty one, and the cost is one cold build per image.
Administration
- What is it doing?
kubectl -n buildkit exec deploy/buildkitd -- buildctl debug workersshows the worker;buildctl duthe cache size by record; the pod log shows each session. - Free the cache.
kubectl -n buildkit exec deploy/buildkitd -- buildctl prune --allempties it without a restart. Deleting the pod no longer does: the cache lives on thebuildkitd-cachevolume. To start from nothing, scale the Deployment to 0, delete the claim and scale back up. - The pod is stuck in
ContainerCreating. The volume’s single replica sits on a node that is down or still detaching it from the old pod. Wait for Longhorn, or delete the claim (see Data Management). - A build fails with “connection refused” or hangs at “connecting”. Check the pod is Ready and that the job pod carries
app.kubernetes.io/component: job— the ingress policy admits nothing else. - A RUN step times out downloading. Look at the egress policy first: build steps share the daemon’s network policy, which allows 80 and 443 only.
Usage
From a Gitea Actions workflow:
- uses: docker/setup-buildx-action@v4
with:
driver: remote
endpoint: ${{ env.BUILDKIT_HOST }}
- uses: docker/build-push-action@v7
with:
push: true
tags: gitea.web.kueber.eu/johnny/app:latestFrom a shell step, without the actions:
docker buildx create --name ci --driver remote "$BUILDKIT_HOST" --use
docker buildx build --push --metadata-file meta.json -t gitea.web.kueber.eu/johnny/app:latest .
jq -r '."containerimage.digest"' meta.json--output type=local,dest=out brings files back instead of an image — the draw.io export of this repository runs that way. --load does not work: there is no Docker daemon to load into.
Cluster-specific deviations from the above live in the per-cluster README — see k8s/apps/buildkit/talos/README.md.
Cluster Deployment
Depends on
BuildKit — Talos cluster
Deviations from defaults
- Flux source is the Git repository, not an OCI artifact (
k8s/clusters/talos/buildkit.yaml) — like gitea-runner, so the CI build path can be fixed or reverted without a working CI. - Tier 2, no control-plane nodes — CI image builds stop while it is down, nothing is lost; builds are CPU-heavy and belong on the workers.
- Cache on
longhorn-object-single-encrypted, 50Gi — one replica because the cache is disposable (a lost node costs a cold build, not data), encrypted because layers carry private source code;object-longhornholds this namespace’s LUKS key. - First-estimate resources — 500m/1Gi requested, 12Gi memory limit so a runaway build fails its builds instead of pressuring the node; re-measure after a few weeks of builds.
Kubernetes Metadata
- Image:
docker.io/moby/buildkit:v0.35.0@sha256:aa632b921ef07411d2cb77292cd34213d4f7b384001d6fceef1e1f25bf9752bb
Rendered manifests (kustomize build)
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: daemon
app.kubernetes.io/name: buildkit
name: buildkitd
namespace: buildkit
spec:
replicas: 1
revisionHistoryLimit: 2
selector:
matchLabels:
app.kubernetes.io/component: daemon
app.kubernetes.io/name: buildkit
strategy:
type: Recreate
template:
metadata:
labels:
app.kubernetes.io/component: daemon
app.kubernetes.io/name: buildkit
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
automountServiceAccountToken: false
containers:
- args:
- --config=/etc/buildkit/buildkitd.toml
image: docker.io/moby/buildkit:v0.35.0@sha256:aa632b921ef07411d2cb77292cd34213d4f7b384001d6fceef1e1f25bf9752bb
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- buildctl
- debug
- workers
initialDelaySeconds: 5
periodSeconds: 30
name: buildkitd
ports:
- containerPort: 1234
name: grpc
protocol: TCP
readinessProbe:
exec:
command:
- buildctl
- debug
- workers
initialDelaySeconds: 5
periodSeconds: 30
resources:
limits:
memory: 12Gi
requests:
cpu: 500m
memory: 1Gi
securityContext:
privileged: true
volumeMounts:
- mountPath: /etc/buildkit
name: config
readOnly: true
- mountPath: /var/lib/buildkit
name: state
enableServiceLinks: false
priorityClassName: tier-2
terminationGracePeriodSeconds: 900
volumes:
- configMap:
name: buildkitd-config-4c87m8482t
name: config
- name: state
persistentVolumeClaim:
claimName: buildkitd-cache