Infrastructure-as-code layers

Four tools in this repo can all claim the phrase "infrastructure as code": OpenTofu, Ansible, Talos machine config, and Crossplane. Each one could, in principle, do most of the others' jobs — Tofu has an Ansible-shaped provisioner, Ansible can call cloud APIs, Crossplane literally runs OpenTofu. This page is about why each layer owns exactly what it owns, and where the handoffs are, so that a new resource lands in the right place on the first try.

It crosses every section of the docs, so it lives in topics/. The actors:

Layer	Owns	Source of truth	Docs
OpenTofu (run by a human)	Everything outside Kubernetes with a long-lived, shared lifecycle	tofu/environment/*	foundation/tofu
Ansible	Mutable Linux hosts — the Proxmox hypervisors, the NAS, the workstation	ansible/	foundation/ansible
Talos machine config (topf)	The OS on every Kubernetes node	talos/	foundation/talos
Crossplane (OpenTofu run in-cluster)	External resources whose lifecycle belongs to one app namespace	k8s/	platform/crossplane · platform/crossplane-extras

And one non-actor that defines the biggest boundary of all: Flux. Everything inside Kubernetes is GitOps — that story is told on GitOps flow and isn't repeated here.

The stack, top-down

The unusual part of this picture is the top layer: Crossplane reaches back out of the cluster. The cluster is provisioned by OpenTofu, and then runs OpenTofu itself — the same tool appears at the bottom and the top of the stack, operated by a human at the bottom and by a controller at the top. The rest of this page is mostly about keeping those two honest with each other.

Layer 1 — OpenTofu: everything outside Kubernetes

OpenTofu owns infrastructure that exists before any cluster does, or that is shared across clusters and sites. One directory per environment under tofu/environment/:

Environment	Provider target	What it declares
tofu/	Hetzner Object Storage (minio provider)	The state buckets themselves — tofu-state-558a4a64 for the other environments, crossplane-talos-workspaces-995867de for Crossplane (see bootstrap)
netbird/	NetBird	Groups, users, service accounts — shared identity consumed by the other environments
production/	Proxmox	The seven Talos VMs (talos-cp-01/02/04, talos-worker-01…04 in talos.tf), NetBird setup keys for the hypervisors
edge/	Hetzner Cloud	control-plane-1 (server.tf), VPC, firewalls, primary IPs for the edge cluster
home/	UniFi	Home network configuration
demo/	Hetzner Cloud	A standalone demo VM configured via cloud-init + Podman quadlets (quadlets/)
johnny-projects/	Hetzner Object Storage	Buckets for side projects

Reusable patterns live in tofu/modules/ (currently netbird/network and netbird/dns_record). Each environment carries a tofu.sh wrapper that loads credentials from a local, uncommitted .env file before invoking tofu — plans and applies are run by a human, from a workstation, after reviewing the diff. There is no apply-on-merge pipeline for this layer, deliberately: the blast radius of a bad apply here is "the cluster's VMs", and the GitOps flow can't help you when the thing being changed is what the cluster runs on.

Where this layer stops: the moment a machine boots. For Talos nodes that's literal — production/talos.tf downloads the Talos nocloud image from factory.talos.dev and attaches it to each VM, and edge/server.tf boots the Hetzner-provided Talos ISO. Tofu knows that a node exists and what IP it has; it knows nothing about what the node runs.

Layer 2 — Ansible: the mutable hosts

Ansible owns hosts that are mutable Linux and not Kubernetes nodes. The inventory (ansible/inventory.yml) currently lists:

• the four Proxmox hypervisors (proxmox-1…4) — commissioned by playbooks/proxmox_commissioning.yaml (the igpu-passthrough role that hands the iGPU to the worker VMs) and maintained by playbooks/proxmox_update.yml (apt upgrades, root authorized_keys)
• the Synology NAS (maresa) and a Home Assistant box
• the local Fedora workstation, via playbooks/local_software.yaml (runs against localhost)

Note the layering inversion: Ansible sits below OpenTofu for the production site. Tofu talks to the Proxmox API to create VMs — but the Proxmox hosts themselves are mutable Debian boxes that something has to configure first (PCI passthrough, updates, SSH keys). That something is Ansible. So the commissioning order for a new hypervisor is Ansible first, Tofu second — the reverse of the edge site, where Hetzner provides the API-ready substrate and Ansible never appears at all.

Where this layer stops: at anything running Talos. Talos has no SSH and no shell, so there is nothing for Ansible to hold on to — and nothing for it to add. That's not a limitation, it's the design: every job Ansible does on a mutable host (packages, users, sshd config, drift correction) is a job Talos eliminated.

Layer 3 — Talos machine config: the node OS

Talos configuration lives in talos/, one directory per cluster (talos/talos/ for production, talos/edge/ for edge), each with a topf.yaml inventory, a SOPS-encrypted talsecret.sops.yaml, and strategic-merge patches in all/, control-plane/ and, where there are workers, worker/. topf renders them per node and topf apply pushes them.

This layer is machine IaC, not cluster IaC: it pins the Talos and Kubernetes versions, disables kube-proxy for Cilium, configures disk encryption, nameservers, and the Spegel registry mirror — everything that has to be true before a kubelet can register.

The seam with the layer below is a shared constant, not a shared mechanism: tofu/environment/production/talos.tf assigns talos-cp-01 the address 192.168.100.101/24, and talos/talos/topf.yaml declares its endpoint at https://192.168.100.101:6443 with nodes matching those addresses. Nothing enforces this agreement — no data source, no output wiring. If the two files drift apart, topf apply simply can't reach the node. Keeping the IP plan identical on both sides of the boundary is a repo convention.

Where this layer stops: at talosctl bootstrap. Once etcd is up and Flux is installed, everything that runs on the cluster arrives via GitOps — the Talos layer never installs workloads.

Layer 4 — Crossplane: OpenTofu, run by the cluster

Crossplane closes the loop: it lets the cluster provision external resources for its own workloads, declaratively, through Flux. The machinery (all under k8s/):

• crossplane-extras installs upbound/provider-opentofu plus the composition functions (function-go-templating, function-auto-ready) and a hardened DeploymentRuntimeConfig for the provider pod.
• k8s/platform/configs/talos/crossplane/ defines a namespaced XRD, TofuWorkspace (tofuworkspaces.platform.kueber.eu), and its Composition — a pipeline that renders a per-namespace OpenTofu ProviderConfig and Workspace.
• Every app opts in with one small manifest: the shared component k8s/components/workspace/base/ drops a TofuWorkspace named workspace into the app's namespace, and the k8s/components/workspace/talos/ overlay patches in the state bucket and labels the namespace so External Secrets delivers the S3 credentials (secrets.tofu-workspace: "true").

The clever part is what the composition reads. Each go-templating step declares an ExtraResources requirement for the k8up Schedules and HTTPRoutes labeled crossplane-workspace: "true", carrying the composite's own namespace, and go-templates an inline OpenTofu module from them. Until function-go-templating stops dropping that namespace for label selectors (0.12.5 still does, its ToResourceSelector returns early on the matchLabels path), every namespace's objects arrive and the templates filter by namespace themselves:

So an app never declares "create me a bucket" — it declares a backup Schedule and a route, and the bucket, the bucket-scoped credential, and the internal DNS record follow from that. The NetBird provider is part of every Workspace, with an empty record set for namespaces without an int.kueber.eu route: the record's lifetime is the route's, so moving an app to the public gateway makes its Workspace delete the record on the next run rather than leaving it behind with no provider to delete it. The full walkthrough of the backup half is on Backblaze B2 backup location.

Where this layer stops: at anything whose lifetime isn't tied to a namespace. The Workspace state lives under a workspace_key_prefix per namespace in crossplane-talos-workspaces-995867de. A VM, a VLAN, or a NetBird group could never live here — nothing in their lifecycle maps to a namespace.

Tearing a namespace down is deliberately not automatic. The rendered module guards every backup bucket with prevent_destroy, so a plain prune of an app's manifests can never take its backups along — but it also means the Workspace's tofu destroy is designed to fail, and until it succeeds the Workspace keeps its finalizer and the namespace stays Terminating. Two things make the unwind work when you mean it:

1. The composition renders a Usage (protection.crossplane.io) for the ProviderConfig and for every Secret the Workspace reads. Without them the namespace controller deletes those alongside the Workspace and the destroy can't even connect ("ProviderConfig not found" — how the Komga and Pocket ID namespaces got stuck in August 2026).
2. TofuWorkspace.spec.retainBuckets (default true) is the opt-out: set it to false in the app's workspace/ overlay and commit before removing the app, after emptying the bucket by hand — object stores refuse to delete non-empty buckets, and the cluster's Backblaze key can only mint keys and create buckets, not delete them. With the flag off, the destroy removes the bucket-scoped application key, the internal DNS record, and the (empty) bucket, the Usages release, and the namespace terminates on its own.

Which layer does a new resource go in?

The resource…	Layer	Because
outlives any single cluster (VM, network, firewall, identity)	OpenTofu	Human-reviewed plan; exists before/independent of the cluster
is a setting on a mutable non-Talos host	Ansible	It's host state; idempotent play, --check --diff for drift
must be true before the kubelet starts (kernel, disk, versions)	Talos config	It's node state; rendered and applied via topf
runs on the cluster	Flux (not IaC here)	GitOps — see GitOps flow
is external but belongs to exactly one app namespace	Crossplane	Lifecycle tracks the app; provisioned from the app's own intent

The near-misses are the instructive ones. Backup buckets sound like OpenTofu (they're external object storage — and their state bucket is), but they're per-app, so they're Crossplane. The Proxmox hypervisors sound like OpenTofu targets (Tofu talks to them constantly), but as hosts they're Ansible's. The Talos VMs sound like they should carry machine config from their creator, but Tofu only boots the installer — the config comes from talos/ over the Talos API.

The bootstrap chicken-and-egg

Crossplane runs OpenTofu, and OpenTofu needs a state backend — which is itself infrastructure. The regress terminates in tofu/environment/tofu/, the one environment whose only job is to create the two versioned buckets on Hetzner Object Storage:

• tofu-state-558a4a64 — the backend for the human-run environments (encrypted with OpenTofu's native pbkdf2/aes_gcm state encryption)
• crossplane-talos-workspaces-995867de — the backend for every in-cluster Workspace (same encryption, keyed from the tofu-encryption secret, one workspace_key_prefix per namespace)

That directory is the root of trust for both ends of the stack, and it is deliberately boring: two buckets, versioning on, applied by hand. In a disaster recovery scenario it is also the first Tofu environment to check, because everything else's state hangs off it.

Where the seams can tear

Each handoff is a place where two layers agree by convention rather than by wiring. The known failure modes:

Symptom	Seam	What actually happened
topf apply can't reach a node	Tofu ↔ Talos	VM address in production/talos.tf and node address in topf.yaml drifted apart
New GPU worker VM won't start on a fresh hypervisor	Ansible ↔ Tofu	proxmox_commissioning.yaml (iGPU passthrough) never ran on that host
Workspace pod stuck, env secret missing	Flux ↔ Crossplane	Namespace lacks the secrets.tofu-workspace: "true" label, so External Secrets never delivered tofu-workspace-s3-credentials
Composition ignores an app's Schedule or route	app ↔ Crossplane	The resource is missing the crossplane-workspace: "true" label the composition selects on
Every in-cluster Workspace errors on init	Tofu ↔ Crossplane	The crossplane-talos-workspaces-995867de bucket or its credentials changed out from under them — check tofu/environment/tofu/
A renamed backup bucket "already exists"	Crossplane ↔ provider	prevent_destroy kept the old bucket; state has to be reconciled manually (the composition keeps commented import blocks for exactly this)
Namespace stuck Terminating on a Workspace finalizer	Flux ↔ Crossplane	The Workspace's destroy has nothing to connect with (ProviderConfig or credential Secret already gone — the Usages guard against this) or hits prevent_destroy (set retainBuckets: false first, see above)
Out-of-band change to a VM or VLAN quietly persists	human ↔ Tofu	This layer has no reconciler — drift shows up only at the next tofu plan

That last row is the deepest difference between the layers: Crossplane and Flux reconcile (drift is corrected within minutes), Ansible detects (drift shows in --check --diff when you run it), and OpenTofu reports (drift appears at the next plan). The further down the stack, the more a human is the control loop.

Where to look next

• OpenTofu · Ansible · Talos — the per-layer pages this one deliberately doesn't duplicate
• Crossplane and crossplane-extras — the in-cluster control plane and its providers
• GitOps flow — the layer between Talos and Crossplane that this page skips over
• Backblaze B2 backup location — the TofuWorkspace composition traced end-to-end for one concrete resource type
• Disaster recovery drill — the layers replayed in order, from bare metal back to reconciled