Policy and runtime security — defense in depth from admission to syscall

This page describes the cluster's security posture as one system rather than four products. Kyverno decides what is allowed into the cluster, Policy Reporter shows what would have been rejected, Tetragon watches what workloads actually do at the kernel level, and Cilium constrains who may talk to whom on the wire.

Each layer catches a different class of problem, and — just as importantly — each layer has a deliberately chosen enforcement level. Very little here is "block everything": most of the cluster runs in audit mode with a small, sharp set of enforced rules. The actors:

Actor	Where it lives in docs
Kyverno (admission webhooks, policy engine)	platform/kyverno
Kyverno policies + exceptions (k8s/platform/configs/talos/kyverno/)	this page
Policy Reporter (compliance dashboard)	apps/policy-reporter
Tetragon (eBPF runtime observability)	platform/tetragon
Cilium (CNI, enforces NetworkPolicy)	platform/cilium
app-network-policy component (default-deny baseline)	components/app-network-policy
Flux (delivers all of the above, and is itself policed)	foundation/flux

The layers, top-down

A compromise has to get past all four: a malicious image must pass admission, its violations would surface in Policy Reporter, its behavior is visible to Tetragon, and its lateral movement is stopped by default-deny networking.

Layer 1 — Kyverno at admission

Kyverno runs as a highly-available admission controller (3 replicas, webhook timeout 5s, plus 2 background-controller and 2 cleanup-controller replicas — k8s/platform/kyverno/talos/kyverno-values.yaml). Its Flux Kustomization depends on Cilium, and the policy layer below it (infra-configs) waits for both Kyverno and Tetragon to be healthy before applying — ordering is deliberate, the enforcer exists before the policies do.

The policies themselves live in k8s/platform/configs/talos/kyverno/ and split into two groups with very different enforcement levels.

Pod Security Standards — audit, not enforce

The kyverno-policies HelmRelease (chart 3.8.2) installs the upstream Pod Security Standards policy set as ClusterPolicy resources:

# k8s/platform/configs/talos/kyverno/helm-release.yaml (values excerpt)
policyKind: ClusterPolicy
podSecurityStandard: restricted
podSecuritySeverity: medium
failurePolicy: Ignore
validationFailureAction: Audit
validationAllowExistingViolations: true

That gives the cluster the full restricted profile — disallow-privileged-containers, disallow-host-path, disallow-host-namespaces, disallow-host-ports, disallow-capabilities (+ -strict), disallow-privilege-escalation, require-run-as-nonroot, restrict-seccomp (+ -strict), restrict-volume-types, disallow-selinux — but in Audit mode. A pod that runs as root is admitted and recorded, not rejected. Two settings underline the "observe first" stance: failurePolicy: Ignore means a wedged Kyverno never blocks the API server, and validationAllowExistingViolations: true means pre-existing workloads aren't suddenly illegal after a policy bump.

Audit mode is only useful if the reports aren't drowning in known noise. That's what the twelve PolicyException resources in configs/kyverno/exceptions/ are for — one per PSS policy, each pre-declaring the system workloads that legitimately need the capability. exception-privileged-containers.yaml is representative: Cilium and Tetragon in kube-system, Longhorn's instance-manager/longhorn-manager/CSI pods, and the Docker-in-Docker gitea-runner StatefulSet are allowed to run privileged; everything else that tries shows up as a genuine finding. PolicyException resources are only honored from the kyverno namespace (features.policyExceptions.namespace: kyverno in the Helm values), so an app can't quietly exempt itself.

Image verification — the enforced part

The sharp end of admission is in configs/kyverno/cosign/, and here the action is Enforce:

• verify-flux-images — every ghcr.io/fluxcd/* / docker.io/fluxcd/* controller image must carry a keyless Cosign signature from the https://github.com/fluxcd/* GitHub Actions identity, checked against the public Rekor transparency log. Unsigned or tampered Flux controllers cannot start. This guards the machine that applies everything else — if Flux is compromised, every other control is moot. (This is a vendored copy of the upstream community policy; a GitRepository pulling kyverno/policies directly exists in the directory but is commented out of the kustomization in favor of the pinned local copy.)
• verify-vaultwarden-image — ghcr.io/dani-garcia/vaultwarden* images must present a valid Sigstore bundle (verifyDigest: true, required: true) whose keyless identity matches the upstream release workflow (.github/workflows/release.yml@refs/tags/...), and that bundle must carry a SLSA Provenance v1 attestation for a GitHub-workflow build of dani-garcia/vaultwarden — not merely any statement the identity ever signed. The password manager is the highest-value app in the cluster, so it gets supply-chain verification at admission — see apps/vaultwarden.

The provenance check started life as a separate, namespaced Audit-mode Policy next to the Vaultwarden manifests, but that draft was never deployed. It was verified with the Kyverno CLI against the digest actually running and folded into the Enforce rule above on 2026-08-29; k8s/platform/configs/talos/kyverno/cosign/vaultwarden.yaml is now the only Vaultwarden-related policy in the repository, and nothing Kyverno-shaped lives under k8s/apps/vaultwarden/.

The pattern to notice: broad rules audit, narrow rules enforce. PSS across every namespace in Audit; image signature verification in Enforce only where the publisher signs reliably — six of the nine cosign policies today, the rest in Audit until their signatures are stable (the full table is on platform/kyverno).

Layer 2 — Policy Reporter for visibility

Audit-mode policies produce PolicyReport/ClusterPolicyReport resources, which are write-only noise without a consumer. Policy Reporter (UI + Kyverno plugin + metrics enabled) is that consumer: a stateless dashboard that reads the reports live from the API and answers "which workloads violate which policy" per namespace, with pass/fail/warn filtering.

Its own deployment demonstrates the layers it reports on:

• Delivered as an OCI artifact from Gitea whose signature is itself Cosign-verified by Flux (verify.provider: cosign on the OCIRepository in k8s/clusters/talos/policy-reporter.yaml) — the compliance dashboard can't be swapped out by an unsigned artifact.
• Reachable only through the internal gateway: its UI pod carries the ingress: internal label and pulls in the app-network-policy component, so Layer 4 applies to it like any other app. No public exposure.
• Metrics are scraped by the monitoring stack, so a spike in fail results can alert without anyone watching the UI.

Kyverno's background scan (every 23h, 2 workers) re-evaluates existing resources against current policies, so the reports also catch drift: a policy added today flags a workload deployed last year.

Layer 3 — Tetragon at runtime

Admission control ends the moment the pod starts. Tetragon (chart 1.7.0, DaemonSet in kube-system, from the same Helm repo as Cilium) picks up from there with eBPF programs below the container runtime — process execution, file access, and network connections, annotated with pod/namespace metadata.

Two cluster-specific choices in k8s/platform/tetragon/talos/tetragon-values.yaml are worth knowing:

• Talos needs a tracefs mount. extraHostPathMounts adds /sys/kernel/tracing explicitly — Talos exposes tracefs at a non-standard path, and without the mount the agent can't attach its eBPF programs.
• The event firehose is deliberately muted. The default export ships every process_exec/process_exit cluster-wide — measured at roughly 22M rows/day into VictoriaLogs with nothing consuming them. The exportAllowList therefore restricts exports to PROCESS_KPROBE, PROCESS_TRACEPOINT, PROCESS_LSM, and PROCESS_UPROBE — i.e. only events produced by explicit TracingPolicy resources.

That second point is an honest statement of where this layer stands today: no TracingPolicy resources are currently deployed (the configs/tetragon/ landing zone exists in the platform tree but its resource list is empty). Tetragon is fully installed, metrics-scraped (annotation-based, ports 2112/2113 for agent and operator), and Talos-capable — the enforcement rules themselves are the open TODO. When they land, they land as GitOps-managed CRDs in k8s/platform/configs/talos/tetragon/, flowing through the same Renovate-PR-to-running-container loop as everything else, and their events will start appearing in the export automatically because the allowlist already selects for them.

Tetragon itself is one of the workloads the Kyverno exceptions pre-declare: it runs privileged with hostPath mounts because eBPF loading requires it — Layer 1 knows about and vouches for Layer 3.

SELinux — permissive, and why volumes carry a fixed label

Below Tetragon sits one more kernel layer that is easy to forget: Talos boots with selinux=1 in permissive mode. Nothing is denied, but every access the policy would deny is written to the kernel audit stream (talosctl logs auditd). Longhorn volumes are formatted without labels, so their files are unlabeled_t, and every write from a pod_t process — a database checkpoint, a log-store flush, an object-store upload — produces one AVC record. On write-heavy nodes that reached thousands of records per second and saturated the kernel's audit rate limit (audit: rate limit exceeded in dmesg, millions of lost records) while nothing was actually blocked. The Longhorn StorageClasses in configs/longhorn/ therefore mount every filesystem volume with context=system_u:object_r:ephemeral_t:s0, the label the Talos policy grants pods, which silences the denials and keeps the volumes usable should enforcing mode ever be switched on. StorageClass mount options are copied into the PV at provision time, so volumes provisioned before the change need spec.mountOptions patched by hand, and each takes effect on its next mount.

Layer 4 — Cilium network policy

The last layer assumes everything above failed: the workload is running and hostile. Cilium enforces standard NetworkPolicy (and CiliumNetworkPolicy) in eBPF, and the app-network-policy component gives every participating app namespace a default-deny baseline:

• block-external-traffic — ingress from outside the namespace is dropped; intra-namespace traffic stays open. This is the lateral-movement stopper.
• allow-metrics-scraping — vmagent from the monitoring namespace may scrape any pod.
• allow-homepage-ingress — pods that opt in with the homepage: active label accept API calls from Homepage.
• allow-envoy-gateway-ingress-internal / -public (talos overlay) — only pods labeled ingress: internal, ingress: public, or ingress: all accept traffic from the corresponding Envoy Gateway, and only from the Envoy proxy pods owned by that specific gateway.

The mechanism is label-driven both ways: the gateway side matches gateway.envoyproxy.io/owning-gateway-name, the app side opts in via the ingress pod label. An app that forgets the label is unreachable through the gateway — fail closed, not open. Because Cilium policy identity follows labels rather than IPs, the rules survive pod churn and rollouts unchanged (see step 6 of the GitOps flow). When a connection is denied, Hubble's policy_denied verdicts are the diagnostic breadcrumb — the Cilium page covers the tooling.

What can break, and where to look

Symptom	Most likely cause	Where to look first
Flux controller pods won't start after node reboot / upgrade	verify-flux-images is Enforce and signature verification failed (Rekor/sigstore unreachable, or genuinely unsigned image)	kubectl -n kyverno logs of the admission controller; the policy's PolicyReport
Vaultwarden pod rejected at admission	Sigstore bundle check in verify-vaultwarden-image failed (10s webhook timeout, upstream signing change)	Kyverno admission logs; compare against the upstream release workflow identity
Policy Reporter shows a wall of fail for a system component	New PSS policy version flags a workload no PolicyException covers	configs/kyverno/exceptions/ — extend the matching exception
Policy Reporter UI is empty	Kyverno background scan not running, or reports not generated yet (23h interval)	platform/kyverno; kubectl get clusterpolicyreports
App unreachable through the gateway, pods healthy	Missing/wrong ingress: pod label for the envoy-gateway NetworkPolicies	components/app-network-policy; Hubble policy_denied
Pod-to-pod call across namespaces times out	block-external-traffic default deny — no allow rule in the target namespace	The target app's network-policies/ overlay; Hubble
Tetragon pods CrashLoopBackOff on Talos	tracefs hostPath mount missing/changed after a Talos upgrade	extraHostPathMounts in tetragon-values.yaml; foundation/talos
VictoriaLogs ingest volume explodes	Tetragon exportAllowList loosened (or removed) without a consumer for the events	tetragon-values.yaml; apps/monitoring
audit: rate limit exceeded floods a node's dmesg	SELinux permissive AVCs from a volume mounted without the context= option (PV provisioned before the option, or patched but not yet remounted)	talosctl logs auditd for the comm=; the PV's spec.mountOptions; restart the pod
infra-configs Kustomization stuck False	A ClusterPolicy/PolicyException fails validation against the Kyverno CRDs	flux logs --kind=Kustomization; recent commits to configs/kyverno/

Why audit-first is the point, not a cop-out

A homelab that enforces the full restricted PSS profile on day one spends its evenings writing exceptions under pressure, with workloads down. The posture here inverts that: everything is visible immediately (Audit + Policy Reporter + Tetragon metrics), and enforcement is added where the blast radius of a miss is highest — the supply chain of the GitOps engine, the password manager, and the platform controllers whose upstreams sign every release. Each layer can graduate independently: a PSS policy can flip to Enforce once its report is clean, the Vaultwarden SLSA provenance check has already made that move, and Tetragon's TracingPolicy slot is wired and waiting. The GitOps loop that delivers those graduations is the same one that delivers everything else.