Skip to main content

Proxmox VE

Proxmox VE is the hypervisor for the on-prem production site. Four physical nodes — proxmox1 to proxmox4, three Intel NUC 13 Pro and one NUC 14 (see hardware/talos) — host the Talos control-plane and worker VMs plus a small fleet of NetBird connector LXCs.

Driven by tofu/environment/production via the bpg/proxmox provider.

Why Proxmox​

  • Open-source, locally hosted hypervisor with a real REST API the IaC layer can talk to.
  • Mixed workloads in one box. KVM VMs for Talos nodes, LXC containers for the NetBird connectors. The connectors don't deserve a full VM and Proxmox handles both natively.
  • Planned maintenance without downtime for the cluster. The Talos VMs are not live-migrated — their disks are local and three workers pass an iGPU through — but with three control planes and four workers a host can be drained and patched while the cluster keeps reconciling.
  • GPU passthrough that actually works for the talos-worker-* nodes feeding Jellyfin / Immich / Tube Archivist.

Alternatives considered​

OptionWhy not
Bare-metal Talos directly on the boxesNo room for the LXC connectors and ad-hoc utility VMs
ESXi / VMwareClosed source; recent licensing changes ruled it out
XCP-ngSolid alternative; Proxmox wins on community + tooling familiarity
HarvesterHCI, but pulls in its own Kubernetes — would conflict with Talos

Cluster layout​

NodeHardwareTalos VMs
proxmox1NUC 13 Procontrol plane talos-cp-01 + worker talos-worker-01
proxmox2NUC 13 Procontrol plane talos-cp-02 + worker talos-worker-02
proxmox3NUC 13 Proworker talos-worker-03; CPUs 0–3 stay free as the failover slot for a control plane
proxmox4NUC 14control plane talos-cp-04 (pinned to four E-cores) + worker talos-worker-04 (the P-cores)

The VM shapes come straight from talos.tf: a control plane has 4 cores, 8 GiB RAM and a 150 GiB system disk; a worker has 10 cores (12 on talos-worker-04), 40 GiB RAM, the same system disk plus a 1 000 GiB second disk for Longhorn. Every VM is the SecureBoot generation — OVMF, vTPM, signed UKI image — and carries Proxmox's protection flag so a teardown is always a deliberate manual step.

Each node also runs:

  • One Talos worker VM — extra NICs into VLAN 104 (storage) and VLAN 105 (public). The iGPU is passed through on talos-worker-01/02/03; talos-worker-04 currently runs without it (gpu: false).
  • One NetBird connector LXC (lxc-proxmox{1,2,3,4}-netbird) — three IPs, one per VLAN it routes (mgmt / storage / public). See NetBird.

The four hosts form one Proxmox cluster (corosync + cluster filesystem), which gives a single management plane and cluster-wide configuration. The Talos VMs themselves stay pinned to their host: disks sit on local-lvm and three workers hold an iGPU via hostpci. Quorum needs a majority of the four votes.

Bridge layout per node​

The full VLAN reference and IP plan lives on the Fabric overview and the UniFi page.

Storage​

PoolBackingUsed by
local-lvmEach node's local NVMeTalos VM disks, LXC root volumes
truenas-iscsiiSCSI to the TrueNAS NASOptional shared volumes (not used in steady state)

Talos VM disks are intentionally on local NVMe, not on shared storage. The etcd quorum tolerates one node down; we don't want a NAS outage to take all three control-planes with it. Persistent application data lives on Longhorn inside the cluster, with bulk media on TrueNAS via NFS.

OpenTofu workflow​

Everything in tofu/environment/production is declarative. Plan / apply against the Proxmox API:

cd tofu/environment/production
tofu init                        # idempotent
tofu plan -out=plan
tofu apply plan

Each apply will:

  • create / reconcile VMs from cloud-init templates,
  • attach VLAN-tagged NICs as required,
  • update DNS records via the NetBird overlay's DNS module,
  • register peers (the connector LXCs) into the right NetBird groups via setup keys.

The provider talks to Proxmox over HTTPS using a token (tofu_provider). Token + state are SOPS-encrypted at rest.

Operational notes​

  • Rolling reboots during a Proxmox upgrade: drain the Talos nodes on that host (kubectl drain), shut their VMs down, reboot the host, start them again. Nothing is migrated and no Proxmox HA is configured; with three control planes and four workers the cluster keeps quorum and capacity while one host is down.
  • GPU passthrough is per-VM. After replacing a worker, re-confirm the IOMMU groups before re-passing-through (Proxmox numbering can shift on kernel upgrades).
  • Cluster-network split on UniFi means corosync rides on VLAN 100. If you change VLAN 100's gateway, plan an extra-careful apply window.
  • Backups of the Proxmox VE configuration itself (PVE storage cfg, replication, cluster join) are in scope for the operations layer, separate from in-cluster backups. See Operations → Backups.

Where to look next​