The life of a request — from DNS lookup to pod

This page traces one HTTPS request through the whole stack: a browser somewhere on the internet loads https://immich.web.kueber.eu, and roughly 100 ms later a pod on the production cluster answers. Between those two moments the request touches public DNS, a Hetzner cloud firewall, two Envoy proxies, a WireGuard mesh, a GeoIP filter, and an eBPF datapath.

It crosses every section of the docs, so it lives in topics/. The actors involved:

Actor	Where it lives in docs
Public DNS zone (kueber.eu at All-Inkl)	platform/cert-manager-webhook-all-inkl
ddclient (keeps home-WAN records fresh)	apps/ddclient
AdGuard Home (resolver for internal names)	apps/adguard-home
Edge cluster (Hetzner, the public front door)	foundation/cluster-edge · foundation/hetzner
NetBird (WireGuard mesh between edge, production, home)	fabric/netbird
Envoy Gateway (both clusters' Gateway API implementation)	platform/envoy-gateway
cert-manager (wildcard certs via DNS-01)	platform/cert-manager
Cilium (LB IPs, eBPF service routing, network policy)	platform/cilium
The app's HTTPRoute + NetworkPolicy	components/http-internal · components/app-network-policy

The path, top-down

Two clusters, one mesh hop, exactly one TLS termination — and the response walks the same path backwards.

Step 1 — DNS

kueber.eu is hosted at All-Inkl (the same provider whose API cert-manager drives for DNS-01 challenges). Public app hostnames live under *.web.kueber.eu and resolve to the edge cluster's Hetzner floating IP — 116.203.167.124, the address the edge Cilium public LB pool pins to the Envoy service (k8s/platform/edge/configs/cilium/ip-pool-public.yaml). The floating IP survives instance replacement, so the DNS record never has to chase a rebuilt node.

Two DNS actors are near this path but not on it for a public app request:

• ddclient runs on the Maresa Synology and keeps the handful of records that must track the rotating home WAN IP (off-cluster admin paths, the NetBird fallback endpoint). App traffic doesn't touch those — the edge IP is static.
• AdGuard Home is the resolver for clients on the home LAN. It matters for the internal path (below), not for a request from the internet.

Step 2 — The edge front door

The edge cluster is a single Talos node on Hetzner. The allow-http cloud firewall admits 80/443 from anywhere, allow-ssh admits 22 (for git) — see Hetzner → cloud firewalls.

The packet lands on the merged public Envoy (EnvoyProxy public sets mergeGateways: true, 2 replicas). Gateway public-kueber (k8s/platform/edge/configs/envoy-gateway/gateway-public-kueber.yaml) has three listeners:

Listener	Port	Mode	Route
http-redirect	80	HTTP	HTTPRoute http-to-https-public → 301 to https
web-tls	443	TLS Passthrough	TLSRoute web-tls (SNI *.web.kueber.eu)
git	22	TCP	TCPRoute git

The important word is Passthrough: for *.web.kueber.eu the edge never terminates TLS, holds no certificate, and sees no cleartext. It matches the SNI and forwards the still-encrypted stream. (The edge does terminate TLS, but only for the edge-hosted apps on their own Gateways — public-seitenraum and public-openbeehive — which never cross to production.)

Both the TLSRoute and the TCPRoute point at the same backend: a headless Service prod-cluster with manually-defined Endpoints at 192.168.105.150 — the production cluster's public gateway IP.

Step 3 — Across the NetBird mesh

192.168.105.150 is a home-LAN address; from a Hetzner datacenter it is only reachable through the NetBird overlay. Each edge Envoy pod carries a NetBird sidecar (an always-restarting init container in the EnvoyProxy pod patch) that joins the mesh in the edge_sidecar_envoy group. The Sidecar Envoy Access Prod Public policy lets exactly that group reach the production public subnet 192.168.105.0/24, via the routing-peer LXCs on the Proxmox nodes (lxc-proxmox{1,2,3}-netbird).

Those routing peers masquerade — so at this hop the real client's IP would be lost to SNAT. The edge Envoy therefore prepends a PROXY protocol v2 header to every forwarded connection (BackendTrafficPolicy send-ppv2, targeting both the TLSRoute and the TCPRoute). The full mechanics of that — why one PPv2 hop suffices for HTTPS but SSH needs it end-to-end, and how to debug a missing hop — are covered in Real client IPs across the chain; this page just notes the hop exists.

Step 4 — Production terminates TLS

On the production side, the cluster-talos public Envoy Service holds 192.168.105.150 from the Cilium public LB pool, and Cilium's CiliumL2AnnouncementPolicy ARP-announces it on ens20 (the VLAN-105 NIC of the worker VMs, see Proxmox) from non-control-plane nodes. The WireGuard packet exits the routing LXC, crosses VLAN 105, and hits Envoy.

Gateway public (k8s/platform/talos/configs/envoy-gateway/gateway-public.yaml) does the real work:

• PPv2 required. ClientTrafficPolicy public sets proxyProtocol.optional: false — every connection must carry the prefix, so a client can't dodge the IP accounting by dialing the LB directly.
• TLS terminates here. The https listener holds the wildcard cert wildcard-public-web-kueber-eu-tls, issued by the allinkl-issuer ClusterIssuer via DNS-01 — the private key never leaves the trusted zone.
• XFF hygiene. The same policy strips any client-supplied X-Forwarded-For in earlyRequestHeaders before original-IP detection, then Envoy writes the PPv2-carried client IP into a fresh XFF for the backend. Security response headers (HSTS, X-Content-Type-Options, …) are stamped on the way out.
• GeoIP allowlist. SecurityPolicy public-geo-allow on the https listener denies by default and allows country: DE, resolved from the PPv2 client IP against a MaxMind DB — plus a carve-out for the edge cluster's own pod CIDR, because the edge pulls OCI artifacts through this very gateway and hairpins with a private source IP.

Step 5 — Route dispatch to the app

Now it's plain Gateway API. The app owns one HTTPRoute in its own namespace, attached across namespaces to the shared Gateway:

# k8s/apps/talos/immich/http-route.yaml
spec:
  parentRefs:
    - name: public
      namespace: envoy-gateway-system
  hostnames:
    - "immich.web.kueber.eu"
  rules:
    - matches: [{ path: { type: PathPrefix, value: / } }]
      backendRefs:
        - name: immich-server-service
          port: 2283

Envoy matches :authority, picks the route, and proxies a cleartext HTTP request — real client IP in X-Forwarded-For — to the Service.

Step 6 — Cilium delivers the last hop

There is no kube-proxy on this cluster. Cilium (kubeProxyReplacement: true) resolves immich-server-service:2283 to a pod endpoint in eBPF on the node the Envoy pod runs on. Native routing (10.100.0.0/16, no VXLAN) carries the packet; if the target pod is on another node, the inter-node hop is WireGuard-encrypted by Cilium itself.

Before the packet enters the pod, network policy gets a say: apps that pull in the app-network-policy component run default-deny, plus an allow-envoy-gateway-ingress-public rule that admits exactly the pods labeled gateway.envoyproxy.io/owning-gateway-name: public in envoy-gateway-system. An app that is only on the internal Gateway gets the internal variant instead — the public proxy can't reach it even if a route were misconfigured.

The pod answers, and the response retraces the chain: Envoy → WireGuard → edge Envoy → client. Nothing on the return path re-terminates anything.

The internal variant

A request from the home LAN to homepage.int.kueber.eu skips the entire left half of the diagram:

1. DNS — the client's resolver is AdGuard Home (handed out by UniFi DHCP). Internal hostnames are answered locally via DNS rewrites and never appear in public DNS — see http-internal.
2. Straight to the internal Gateway — the internal Envoy Service holds 192.168.105.151 from the Cilium internal LB pool, L2-announced the same way. Gateway internal terminates *.int.kueber.eu and *.talos.int.kueber.eu with their own wildcard certs.
3. No PPv2, no GeoIP — the internal ClientTrafficPolicy sets TLS floors and security headers, but there is no proxy chain to preserve IPs across and no country filter. The client IP arriving at Envoy is the real one.
4. Same steps 5–6 — HTTPRoute dispatch (e.g. k8s/apps/talos/homepage/http-internal/) and the Cilium last hop are identical, with the internal NetworkPolicy variant.

Mesh members get the same treatment from anywhere, since NetBird routes them into 192.168.105.0/24. The handful of services on the Maresa Synology itself (*.maresa.int.kueber.eu) short-circuit even earlier — AdGuard rewrites them to the local Traefik, and the cluster is never involved.

The SSH variant

git clone git@gitea.web.kueber.eu follows the public path but stays L4 the whole way: edge TCPRoute git (port 22) → PPv2 over the mesh → production TCPRoute gitea-ssh (k8s/apps/talos/gitea/tcp-route.yaml) → Gitea's built-in SSH server on 2222. Because nothing ever terminates TLS or HTTP, PPv2 must be re-emitted on the last hop too (BackendTrafficPolicy send-ppv2-gitea). That asymmetry — and every way it can go wrong — is the subject of the PPv2 page.

What can break, and where to look

Symptom	Most likely cause	Where to look first
Public hostname doesn't resolve	DNS record drift at All-Inkl	dig immich.web.kueber.eu — expect the edge floating IP
Resolves, but connection times out	Edge node down, or allow-http firewall label missing	foundation/hetzner; Gatus history
ERR_CONNECTION_RESET on 443	A bare (non-PPv2) connection hit the strict production listener	PPv2 page → common pitfalls
5xx from edge, app healthy in production	Mesh hop wedged — envoy sidecar peer or routing LXCs	netbird status in the edge envoy pod; fabric/netbird
403 for a legitimate user	GeoIP allowlist (DE-only) or a route-level SecurityPolicy that replaced it	security-policy-public-geo.yaml and its warning about route-level overrides
TLS errors after cert renewal	Wildcard secret not yet propagated to the Gateway	platform/cert-manager
App logs show a proxy IP instead of the client	A missing PPv2/XFF hop	the hop-by-hop table on the PPv2 page
Internal name doesn't resolve at home	AdGuard down or rewrite missing	apps/adguard-home; UniFi DHCP fallback note
Route matches but backend unreachable	Default-deny NetworkPolicy without the gateway allow-rule	components/app-network-policy; Hubble policy_denied

Why this is a topic, not a component page

Each hop already has an owner — the Gateways belong to Envoy Gateway, the mesh to NetBird, the LB IPs to Cilium. What no single page shows is the order: which hop sees which source IP, where TLS actually terminates, and which layer to blame when a request dies halfway. When something breaks, the diagnosis is "where along this path did it stop?" — the same framing as the GitOps flow, applied to the data plane instead of the control plane.