Private connectivity with NetBird

Almost nothing in this homelab is reachable from the internet. The edge cluster terminates the few public entry points; everything else — Proxmox consoles, the TrueNAS UI, the Synology, internal apps, the smart-meter LAN — is reachable only over the NetBird WireGuard overlay. That overlay is not one component: it is declared in OpenTofu, joined by LXCs, Talos nodes, Docker containers, and pod sidecars, consumed by an in-cluster operator and by Crossplane-generated DNS records.

This page is the cross-layer view: where each piece lives, how a peer actually gets onto the mesh, and how "private by default" holds together. The mesh reference itself (peer/network/resource model, policy tables) lives on Fabric → NetBird — this page links there rather than repeating it.

Actor	Where it lives in docs
NetBird control plane (api.netbird.io, managed)	fabric/netbird
tofu/environment/netbird (identity, groups, tokens)	foundation/tofu
Per-site Tofu envs (edge, production, home)	foundation/hetzner · foundation/proxmox · fabric/unifi
Routing peers (LXCs, control-plane-1, home box)	fabric/netbird
Workload sidecars (edge Envoy, Gatus, Homepage)	platform/envoy-gateway · apps/gatus · apps/homepage
netbird-operator (talos cluster)	platform/netbird-operator
Standalone agent on the Synology	apps/netbird
Crossplane workspaces writing int.kueber.eu records	platform/crossplane · components/http-internal

The layers, top-down

Every arrow downward is a one-way dependency: the per-site environments consume the identity layer via data lookups, peers consume setup keys minted by the per-site environments, and workloads consume the connectivity the peers provide. A per-site tofu apply can never drift the shared identity layer — that separation is the design decision the rest of this page hangs off.

Layer 1 — the identity environment

tofu/environment/netbird is deliberately tiny: two cross-cutting groups (administrators, sidecars), one human user (auto-assigned to administrators, role owner), and four service users — tofu_env_{netbird,production,home,edge} — each with a 365-day API token. Its state lives in an encrypted S3 backend (Hetzner object storage, AES-GCM state encryption), like the other environments described on foundation/tofu.

The tokens are the trust boundary between environments: each per-site environment authenticates to api.netbird.io with its own service-user token, so the blast radius of a leaked token or a bad apply is one site. The cross-site policy set — who in administrators reaches which network — is documented on fabric/netbird.

Layer 2 — per-site networks from one module

Production and home don't hand-write their NetBird wiring; they instantiate tofu/modules/netbird/network, which for a given name and ip_ranges map creates:

• a netbird_network named after the site,
• two groups: <name> (resource access) and <name>_peers (routing peers),
• one netbird_network_resource per subnet in ip_ranges,
• an Admin <Name> Policy (bidirectional, protocol all) from administrators to the network group,
• a netbird_network_router over the <name>_peers group with metric = 9999 and masquerade = true.

That masquerade flag is why traffic arriving from the mesh carries the routing peer's source IP — the SNAT behavior that the PROXY-protocol-v2 topic exists to work around.

Concretely:

Site	Resources (subnets)	Declared in
production	default, management, storage, public, energy (192.168.178.0/24)	tofu/environment/production/netbird.tf
home	management, iot, private, guest	tofu/environment/home/netbird.tf
edge	edge Management Subnet (the Hetzner hcloud_network_subnet range)	tofu/environment/edge/netbird.tf (inline, not via the module — its admin policy is tcp-only)

The energy resource is the only route into the FritzBox smart-meter LAN from anywhere — see fabric/unifi for why that reachability is asymmetric on purpose.

Layer 3 — how a peer actually joins

Everything joins the mesh the same way: netbird up --setup-key …, where the setup key was minted by Tofu with auto_groups baked in. The key is the group membership, which is what makes joining declarative — no clicking peers into groups after the fact.

Setup key (Tofu name)	Type / lifetime	Auto-groups	Used by
routing-peers-edge	reusable, ephemeral	edge, edge_peers	edge routing peer(s)
edge-sidecar-envoy	reusable, ephemeral	edge, edge_sidecar_envoy	Envoy proxy sidecar on the edge cluster
proxmox{1,2,3}	one-off, 24 h, 1 use	production, production_peers	the three connector LXCs (used once at provisioning)
production-truenas	reusable, ephemeral	production, production_peers	TrueNAS
workload-talos-gatus	reusable, ephemeral	production, sidecars, workload-talos-gatus	Gatus sidecar
workload-talos-homepage	reusable, ephemeral	production, sidecars, workload-talos-homepage	Homepage sidecar
routing-peers-home	reusable, ephemeral	home, home_peers	home routing peer

ephemeral means NetBird garbage-collects the peer when it goes offline for a while — right for pods and containers that re-register on every restart, wrong for the LXCs, which use single-use keys and keep a stable peer identity instead. The per-workload groups (workload-talos-*) exist so future policies can target exactly one workload's sidecar rather than "all sidecars".

The delivery mechanism for each key differs by layer, and that's the interesting part:

• Production LXCs — the key is interpolated straight into a remote-exec provisioner: Tofu creates lxc-proxmox{1,2,3}-netbird (Debian 13, 1 vCPU, 512 MB, three NICs: untagged management on vmbr0, VLAN 104 storage, VLAN 105 public), installs the agent from pkgs.netbird.io, and runs netbird up with the one-off key. See foundation/proxmox.
• Kubernetes workloads — the key sits in a SOPS-encrypted Secret (NB_SETUP_KEY) next to the app's manifests, decrypted by Flux like everything else (operations/sops).
• Synology — injected via host environment into a Docker Compose service, never committed; the rootless agent registers as netbird-home (apps/netbird).
• Edge in-cluster client — the base Deployment (netbird-routing-peer, 2 replicas, rootless image, runAsUser: 65532, all capabilities dropped — userspace WireGuard needs none when the pod isn't routing) ships to the edge cluster as a cosign-verified OCI artifact reconciled by Flux, with the SOPS secret patched in by the edge overlay (k8s/apps/edge/netbird-client/).

Layer 4 — sidecars: the mesh inside a pod

Three workloads carry a NetBird agent as a native sidecar (an initContainer with restartPolicy: Always): the edge Envoy proxy fleet, and Gatus and Homepage on the talos cluster. The pattern is identical in all three:

initContainers:
  - name: sidecar-netbird
    image: netbirdio/netbird:0.76.1@sha256:…
    restartPolicy: Always
    env:
      - name: NB_SETUP_KEY
        valueFrom: { secretKeyRef: { name: <app>-netbird, key: NB_SETUP_KEY } }
    securityContext:
      capabilities:
        drop: ["ALL"]
        add:
          - NET_ADMIN
          # netbird >= 0.75 opens a raw ipv4 socket when bringing up
          # wt0; without NET_RAW the engine dies before the wg handshake.
          - NET_RAW
    startupProbe:
      exec: { command: ["sh", "-c", "netbird status | grep -q Connected"] }

The sidecar brings up a wt0 interface inside the pod's network namespace, so the app container transparently reaches mesh destinations — no proxy configuration in the app. The startupProbe gates the pod on netbird status reporting Connected, so the app never starts before the tunnel exists.

The edge Envoy variant (patched into the EnvoyProxy resource, k8s/platform/edge/configs/envoy-gateway/envoy-proxy-public.yaml) additionally mounts /dev/net/tun from the host and runs as root — it registers under the edge_sidecar_envoy group, whose Sidecar Envoy Access Prod Public policy grants it exactly one thing: TCP to the production public subnet resource. That single policy is the entire edge→production data path; how client IPs survive the masquerade on that hop is the subject of the PPv2 topic.

Layer 5 — in-cluster consumers of the NetBird API

Two things in the talos cluster talk to the NetBird API rather than joining as peers:

1. netbird-operator — HelmRelease netbird-operator (chart kubernetes-operator) in the netbird namespace, with ingress and router features enabled, cluster name talos, cluster DNS svc.cluster.local, and the API key from the SOPS-encrypted netbird-operator-secret. It manages peers and network access declaratively via CRDs — details and alternatives on platform/netbird-operator.
2. Crossplane tofu workspaces — a ClusterExternalSecret (netbird-token, sourced from the in-cluster secrets hub) materializes a NetBird API token into every namespace labeled secrets.netbird: "true" — a label the workspace component stamps onto app namespaces. The tofu-workspace Composition then uses it for DNS, below.

The DNS story — two private zones

Private connectivity is only useful if names resolve, and there are two zones, populated by two very different mechanisms:

Zone	Populated by	Contains
sys.kueber.eu	tofu/modules/netbird/dns_record calls in the production and home envs	Infrastructure hosts: proxmox{1,2,3}/truenas under *.production.…, unifi-home/synology/home-assistant under *.home.… (A records, TTL 60)
int.kueber.eu	The Crossplane tofu-workspace Composition, automatically	One A record per internal app hostname → 192.168.105.151

The second row is the fully-automated one: when an app's namespace contains an HTTPRoute with a hostname ending in int.kueber.eu, the Composition injects a netbird_dns_record block into that app's OpenTofu workspace (authenticated with the distributed netbird-token), pointing the name at 192.168.105.151 — the internal Envoy Gateway's LoadBalancer IP from Cilium's internal ip-pool. Declaring the route is declaring the DNS. The internal Gateway itself, and why it is reachable from LAN and mesh but never the internet, is components/http-internal.

Both zones are NetBird-managed: mesh peers resolve them, everything else gets NXDOMAIN. (On the home LAN, AdGuard Home covers resolution for non-mesh clients.)

What "no public exposure" adds up to

• The only public listeners in the entire lab are on the edge cluster; it forwards inward over the mesh via one narrowly-scoped sidecar policy.
• Internal apps attach to the internal Gateway and get an int.kueber.eu record automatically — public exposure would require deliberately attaching to the public Gateway instead.
• Infra UIs (Proxmox, TrueNAS, UniFi, Synology, Home Assistant) are reachable through site routing peers under Admin <site> policies — membership in administrators is the credential, not a VPN config file.
• The energy LAN has no inbound path at all except the mesh.

What can break, and where to look

Symptom	Most likely cause	Where to look first
New peer can't register	Setup key revoked, or a one-off key already consumed	Setup keys in the per-site Tofu env; fabric/netbird
Whole production site unreachable from mesh	All three lxc-proxmox*-netbird routers down	foundation/proxmox — the LXCs
Public apps 5xx, healthy in cluster	Edge Envoy sidecar lost its tunnel, or the sidecar policy changed	Sidecar startupProbe on edge; PPv2 topic
App logs show a mesh IP instead of the real client	A PPv2 hop is missing after the masquerade	PPv2 topic
Sidecar crash-loops before the WG handshake	NET_RAW capability missing (required since netbird 0.75)	The sidecar securityContext in the app's manifests
*.int.kueber.eu name doesn't resolve for a new app	Hostname doesn't match the zone suffix, or netbird-token absent in the namespace	platform/crossplane workspace; the secrets.netbird namespace label
tofu apply in any env returns 401	The env's 1-year service-user token expired	tofu/environment/netbird — re-mint, then update the consumer
Synology invisible on the mesh, fine on the LAN	The agent container stopped	apps/netbird

Why this is a topic, not a fabric page

Fabric → NetBird documents the mesh as a network: peers, resources, policies, DNS. This page documents NetBird as a pattern that repeats at every layer — an identity-owning Tofu environment, module-stamped site networks, setup keys as declarative group membership, sidecars as per-pod mesh access, and API tokens flowing through External Secrets into Crossplane. When connectivity breaks, the question isn't "is NetBird down" — it's which layer's peer, key, policy, or token stopped holding its end up.