Storage tiers and placement — from physical disks to a PVC

Every byte of live data in the homelab sits in exactly one of three places: a replicated Longhorn volume on the Talos workers' NVMe, an NFS export on the TrueNAS, or nowhere (ephemeral emptyDir, gone with the pod). This page traces the full stack from physical disk to storageClassName, and explains how the placement decision is made per app.

It is about live storage only. Where the second and third copies of a byte go — Restic, Syncthing, cold drives — is covered in Three-tier backups, end-to-end; this page links there rather than repeating it.

The layers involved:

Layer	Where it lives in docs
Intel NUC hosts × 4 (three NUC 13 Pro with 2 TB NVMe each, one NUC 14)	hardware/talos
Proxmox VE (carves the NVMe into VM disks)	foundation/proxmox
Talos Linux (owns the disks inside the VMs)	foundation/talos
Longhorn (replicated block storage, five classes)	platform/longhorn
TrueNAS SCALE (bulk media over NFS)	hardware/nas
CNPG (databases with their own replication)	platform/cloudnative-pg
k8up (the backup layer above all of this)	platform/k8up · topics/backups-end-to-end

The stack, top-down

Tier 0 — the physical disks

Two independent pools of spinning-and-solid-state exist:

• Cluster NVMe. Each of the four NUC hosts carries local NVMe. Proxmox runs on it and hands each talos-worker VM a second, 1 000 GiB virtual disk. In talos/talos/worker/10-longhorn.yaml that disk is claimed as a Talos user volume (UserVolumeConfig) — diskSelector: disk.dev_path == '/dev/sdb' && !system_disk, minimum 500 GiB, grow: true — and mounted at /var/mnt/longhorn; Talos propagates /var/mnt into the kubelet's mount namespace, so no bind-mount patch is needed. Only the worker VMs get this; the control-plane VMs hold no Longhorn data.
• TrueNAS HDDs. The TrueNAS SCALE build serves its zima pool (Seagate Exos drives behind an LSI 9211-8i in IT mode) over NFS on the storage VLAN at 192.168.104.200. This is where bulk media lives — datasets measured in terabytes that would be absurd to replicate three times across 500 GiB worker disks.

The off-site Synology and the cold WD Elements drives are not live-storage tiers — no pod ever mounts them. They belong entirely to the backup story.

Tier 1 — Longhorn on the workers

Longhorn (HelmRelease in longhorn-system, data path /var/mnt/longhorn) turns the four worker disks into replicated block volumes. Two deliberate defaults shape everything below:

1. defaultClass: false — there is no default StorageClass in the cluster. Every PVC must name its class explicitly. That's a feature: the comment in the NATS component (k8s/components/nats/base/nats.yaml) records how an unqualified claim once drifted onto an emptyDir and lost all JetStream state on every pod reschedule. A Pending PVC is a visible failure; silently landing on the wrong tier is not.
2. defaultReplicaCount: 3 — unless a class says otherwise, a volume has replicas on three of the four workers, so any single node can die without data loss.

A daily Longhorn RecurringJob (02:00, retain 3) snapshots every volume in the default group — that's crash-consistency for rollbacks, not a backup; backups are k8up's job.

The five storage classes

All classes live in k8s/platform/configs/talos/longhorn/:

Class	Replicas	Encrypted	Reclaim	Binding	Meant for
longhorn-encrypted	3	LUKS	Delete	Immediate	App config and state — the default workhorse
longhorn-db-single-encrypted	1	LUKS	Retain	WaitForFirstConsumer	CNPG database volumes, NATS JetStream
longhorn-object-single-encrypted	1	LUKS	Retain	WaitForFirstConsumer	Disposable object data (gitea-runner caches)
longhorn-object-encrypted	2	LUKS	Retain	WaitForFirstConsumer	SILO object data that must survive a node loss
longhorn-rwx	3	no	Delete	Immediate	The rare ReadWriteMany case on block storage

The replica counts are not arbitrary — they encode where redundancy lives:

• longhorn-encrypted (3×) is for apps with no replication of their own: Vaultwarden, Gitea, Jellyfin config, Immich app data, the arr stack, monitoring's Victoria-Metrics volumes, Nextcloud, and ~30 more. The storage layer is the only thing keeping the data alive, so it keeps three copies.
• longhorn-db-single-encrypted (1×) exists because CNPG already replicates at the Postgres layer: the talos overlay (k8s/components/cnpg/talos/patch-cluster.yaml) runs two instances with required pod anti-affinity across nodes. Streaming replication replaces Longhorn's 3× copies — 3 Longhorn replicas × 2 Postgres instances would store every database six times. dataLocality: best-effort keeps the single replica on the same node as the pod using it.
• longhorn-object-encrypted (2×) carries SILO, the S3 store behind the minio component, and the manifest says why: "MinIO (unlike CNPG) has no application-level replication, so surviving a node loss is the storage layer's job." Disposable object data — the gitea-runner cache and artifact store — stays on the -single variant, where a lost volume is an inconvenience, not a loss.
• longhorn-rwx (3×, unencrypted) has exactly one consumer today: the shared GeoIP database PVC in envoy-gateway-system, written by a CronJob and read concurrently by every Envoy Gateway pod.

Encryption secrets

Every class except longhorn-rwx sets encrypted: "true" — LUKS keys are ordinary Kubernetes Secrets, SOPS-encrypted in Git (see operations/sops). Two schemes coexist:

• longhorn-encrypted stamps a per-PVC secret (${pvc.name}-longhorn) — that's why nearly every app directory carries a secrets-longhorn.yaml.
• The db/object classes use a fixed per-namespace secret (cnpg-longhorn / object-longhorn), so StatefulSet volumeClaimTemplates and CNPG instance scaling never wait on a human to pre-create a key per volume.

Tier 2 — NFS from the TrueNAS

Bulk media never touches Longhorn. Eight NFS exports under /mnt/zima/ are wired in as static PV + PVC pairs in k8s/resources/storage/talos/{archive,audio,downloads,gaming,images,reading,stash,videos} — ReadWriteMany, nfsvers=4.1, persistentVolumeReclaimPolicy: Retain, capacities up to 40 Ti.

Apps don't share one cluster-wide claim. Each app's kustomization pulls the resource directory through a namePrefix (jellyfin's storage/ overlay renders the pair as jellyfin-truenas-nfs-videos), so every consumer owns an independent PV/PVC binding onto the same export. Deleting one app's claim can never yank the volume out from under another — and Retain means even that deletes no data on the NAS.

Who mounts what:

Export	Consumers (from the deployment patches in the apps' talos/ overlays)
videos (40 Ti)	Jellyfin, sonarr, radarr, Tube Archivist
images	Immich — originals under upload/, via subPath
audio	Navidrome, Audiobookshelf, lidarr
reading	Stump, Tachiyomi
downloads	Transmission plus the arr stack's import path
gaming	RomM
archive, stash	Stash, FileFlows, Filebrowser (mounts all eight)

The recurring pattern — visible in Jellyfin and Immich alike — is the split volume: config and databases on longhorn-encrypted (small, hot, replicated, backed up), media on NFS (huge, warm, mirrored off-site by Syncthing instead of k8up — see backups-end-to-end).

Tier 3 — deliberately ephemeral

Some state is cheapest to lose. The Valkey component (k8s/components/valkey/base/valkey.yaml) mounts both /conf and /data as emptyDir — it serves as cache/session store, and every consuming app can rebuild that from scratch faster than a restore would run. n8n similarly keeps a scratch volume on emptyDir next to its persistent n8n-data claim.

On the edge cluster there is no Longhorn at all: the single-node Talos edge cluster runs the local-path provisioner (k8s/platform/local-path-provisioner/edge/), and the stateful edge workloads — openbeehive's PVC and the single-instance CNPG clusters of both edge apps — use it: node-local, unreplicated, acceptable there by design.

Choosing a tier for a new app

The decision tree the existing placements encode:

1. Can the data be regenerated or re-fetched? → emptyDir (or longhorn-object-single-encrypted if it must merely survive a pod restart, like runner caches).
2. Does the app replicate its own data? Postgres via the CNPG component → longhorn-db-single-encrypted; anything else that self-replicates follows the same logic: one Longhorn replica, redundancy at the app layer, anti-affinity across nodes.
3. Is it bulk media, terabytes, shared between apps? → a static truenas-nfs-* claim from k8s/resources/storage/talos/, pulled in with a namePrefix.
4. Everything else — config, small databases, uploads that are the app → longhorn-encrypted, plus a ${pvc.name}-longhorn secret, plus the k8up.io/backup: "true" annotation so the warm backup tier picks it up.

Never omit storageClassName — with no default class the PVC will sit Pending forever, which is the intended failure mode.

What can break, and where to look

Symptom	Most likely cause	Where to look first
PVC stuck Pending	Missing storageClassName (no default class), or WaitForFirstConsumer with an unschedulable pod	kubectl describe pvc; the class table above
Volume attaches but mount fails	The LUKS secret (<pvc>-longhorn, cnpg-longhorn, object-longhorn) missing in the namespace	the app's secrets-longhorn.yaml; operations/sops
Longhorn volume Degraded	A worker (or its /dev/sdb user volume) is down; replicas rebuilding	Longhorn UI in longhorn-system; platform/longhorn
Node disk pressure on workers	3× replication amplifying a fast-growing volume	Longhorn UI node view; consider whether the volume belongs on NFS
Media apps hang in D state / probes time out	TrueNAS or the storage VLAN path is down — NFS hard mounts block	hardware/nas; the NFS server at 192.168.104.200
Database volume full	WAL retention — the CNPG base component caps max_slot_wal_keep_size for exactly this	platform/cloudnative-pg · operations/postgres-restore
Data still on disk after deleting a PVC	Not a bug: Retain on the db/object classes and on every NFS PV	this page — reclaim column above

See also

• Topics → Backups end-to-end — where the second and third copies of these bytes go
• Topics → Disaster recovery — rebuilding the tiers from nothing
• Platform → Longhorn — the controller itself, chart config, alternatives considered
• Platform → CloudNative-PG — why databases get single-replica volumes
• Hardware → TrueNAS build · Hardware → Talos cluster — the metal under it all
• Foundation → Talos — where the /var/mnt/longhorn user volume is declared