Skip to content
BuildKit

BuildKit

One shared BuildKit daemon that builds and pushes every container image CI produces.

BuildKit is the build engine behind docker build and docker buildx: it turns a Dockerfile into a graph of steps, runs them in parallel where it can, caches every step by its inputs, and exports the result — to a registry, a tarball or a local directory. Docker embeds it, but it also runs standalone as buildkitd, reachable over gRPC, and buildx can drive such a daemon through its remote driver as if it were local.

That standalone daemon is what CI builds on. The Gitea runner runs every job as a plain Kubernetes pod without a Docker daemon, so a job cannot build an image itself; it hands the build to buildkitd instead. One daemon serves all of them.

Why one central daemon

The alternative designs each move the problem somewhere worse:

OptionWhat it costs
Docker-in-Docker per runner (the old setup)A privileged sidecar next to every runner, any job one breakout away from its node, and a cache that dies with each pod
BuildKit sidecar per job podStill privileged, now in the namespace where the runner creates pods; no cache shared between builds
Kaniko / Buildah in the jobUnprivileged, but a different build tool with its own Dockerfile quirks, no buildx, every workflow rewritten
Rootless BuildKitNeeds unprivileged user namespaces, which Talos turns off (user.max_user_namespaces = 0)

A central daemon keeps the one component that really needs privileges in a namespace of its own, gives every build the same warm layer cache, and changes each workflow by two lines: driver: remote and an endpoint.

Installation

Architecture

  • Deployment buildkitd, one replica, rootful moby/buildkit pinned by tag and digest. It listens on tcp://0.0.0.0:1234 for the job pods and on the default unix socket for its own probes (buildctl debug workers).
  • Service buildkitd on port 1234; the runner hands its address to every job as BUILDKIT_HOST (tcp://buildkitd.buildkit.svc.cluster.local:1234).
  • Layer cache on its own volume under /var/lib/buildkit, so a restart or an image update starts warm. BuildKit’s garbage collection bounds it in percentages of that volume: usage stays under maxUsedSpace (80 %) with 10 % free, and never shrinks below reservedSpace (20 %), so growing the claim grows the cache without a config change. Builds that want their cache on any builder still use the registry (cache-to: type=registry).
  • Host network mode for RUN steps (networkMode = "host"). Here “host” means the daemon’s own pod: build steps share its network namespace, so they inherit the pod MTU and the pod’s network policies. A bridge would need its MTU tuned below the WireGuard overlay by hand, as the old dind sidecar did with --mtu=1000.
  • max-parallelism = 4 caps concurrent build steps across all clients, so eight simultaneous CI jobs queue for the node’s CPUs instead of fighting over them.
  • Recreate updates. buildkitd locks its state directory and the volume is ReadWriteOnce, so two daemons never overlap: on SIGTERM the old one stops taking connections and lets running builds finish within the 15-minute termination grace period, then the new one mounts the cache. Builds started in that window fail to connect and need a re-run.

Security

  • Privileged, alone. Rootful BuildKit creates namespaces and mounts for every build step, which needs privileged: true. The buildkit namespace runs at Pod Security privileged with nothing else in it, which is what lets the CI namespace sit at baseline.
  • No entitlements. network.host and security.insecure stay disabled, so a Dockerfile cannot ask for a privileged build step or the host network.
  • The network policy is the access control. The gRPC port has no TLS and no client authentication; only pods labelled as Gitea Actions jobs in the gitea-runner namespace may connect. Anyone who can reach port 1234 can run arbitrary build steps, so widening that rule is a security decision, not a convenience.
  • Credentials never live in the daemon. Registry logins stay in the job (~/.docker/config.json); buildx forwards them over the build session only for the duration of a push or a registry cache import.
  • Egress to the internet on 80/443 (base images, package managers inside RUN steps) and to the gateway, which carries pushes to the Gitea registry.

Updates

Renovate bumps the image tag and digest. The cache volume normally carries over to the new daemon; should a new version reject the old state, delete the claim — nothing in it needs keeping.

Data Management

None worth keeping. The cache is reproducible by definition, so the volume is a single replica and nothing is backed up. If the node holding that replica is gone for good, delete the claim: the next pod provisions an empty one, and the cost is one cold build per image.

Administration

  • What is it doing? kubectl -n buildkit exec deploy/buildkitd -- buildctl debug workers shows the worker; buildctl du the cache size by record; the pod log shows each session.
  • Free the cache. kubectl -n buildkit exec deploy/buildkitd -- buildctl prune --all empties it without a restart. Deleting the pod no longer does: the cache lives on the buildkitd-cache volume. To start from nothing, scale the Deployment to 0, delete the claim and scale back up.
  • The pod is stuck in ContainerCreating. The volume’s single replica sits on a node that is down or still detaching it from the old pod. Wait for Longhorn, or delete the claim (see Data Management).
  • A build fails with “connection refused” or hangs at “connecting”. Check the pod is Ready and that the job pod carries app.kubernetes.io/component: job — the ingress policy admits nothing else.
  • A RUN step times out downloading. Look at the egress policy first: build steps share the daemon’s network policy, which allows 80 and 443 only.

Usage

From a Gitea Actions workflow:

- uses: docker/setup-buildx-action@v4
  with:
    driver: remote
    endpoint: ${{ env.BUILDKIT_HOST }}
- uses: docker/build-push-action@v7
  with:
    push: true
    tags: gitea.web.kueber.eu/johnny/app:latest

From a shell step, without the actions:

docker buildx create --name ci --driver remote "$BUILDKIT_HOST" --use
docker buildx build --push --metadata-file meta.json -t gitea.web.kueber.eu/johnny/app:latest .
jq -r '."containerimage.digest"' meta.json

--output type=local,dest=out brings files back instead of an image — the draw.io export of this repository runs that way. --load does not work: there is no Docker daemon to load into.

Cluster-specific deviations from the above live in the per-cluster README — see k8s/apps/buildkit/talos/README.md.

Cluster Deployment

Depends on

BuildKit — Talos cluster

Deviations from defaults

  • Flux source is the Git repository, not an OCI artifact (k8s/clusters/talos/buildkit.yaml) — like gitea-runner, so the CI build path can be fixed or reverted without a working CI.
  • Tier 2, no control-plane nodes — CI image builds stop while it is down, nothing is lost; builds are CPU-heavy and belong on the workers.
  • Cache on longhorn-object-single-encrypted, 50Gi — one replica because the cache is disposable (a lost node costs a cold build, not data), encrypted because layers carry private source code; object-longhorn holds this namespace’s LUKS key.
  • First-estimate resources — 500m/1Gi requested, 12Gi memory limit so a runaway build fails its builds instead of pressuring the node; re-measure after a few weeks of builds.

Kubernetes Metadata

  • Image: docker.io/moby/buildkit:v0.35.0@sha256:aa632b921ef07411d2cb77292cd34213d4f7b384001d6fceef1e1f25bf9752bb
Rendered manifests (kustomize build)
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: daemon
    app.kubernetes.io/name: buildkit
  name: buildkitd
  namespace: buildkit
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app.kubernetes.io/component: daemon
      app.kubernetes.io/name: buildkit
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/component: daemon
        app.kubernetes.io/name: buildkit
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/control-plane
                operator: DoesNotExist
      automountServiceAccountToken: false
      containers:
      - args:
        - --config=/etc/buildkit/buildkitd.toml
        image: docker.io/moby/buildkit:v0.35.0@sha256:aa632b921ef07411d2cb77292cd34213d4f7b384001d6fceef1e1f25bf9752bb
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - buildctl
            - debug
            - workers
          initialDelaySeconds: 5
          periodSeconds: 30
        name: buildkitd
        ports:
        - containerPort: 1234
          name: grpc
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - buildctl
            - debug
            - workers
          initialDelaySeconds: 5
          periodSeconds: 30
        resources:
          limits:
            memory: 12Gi
          requests:
            cpu: 500m
            memory: 1Gi
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /etc/buildkit
          name: config
          readOnly: true
        - mountPath: /var/lib/buildkit
          name: state
      enableServiceLinks: false
      priorityClassName: tier-2
      terminationGracePeriodSeconds: 900
      volumes:
      - configMap:
          name: buildkitd-config-4c87m8482t
        name: config
      - name: state
        persistentVolumeClaim:
          claimName: buildkitd-cache
Last updated on