Identity end-to-end: LLDAP and Keycloak

Two services in the cluster carry the word "identity", and until August 2026 there were three. This page draws the whole picture once: where user accounts actually live, how every app is wired to the single OIDC provider, and every hop of a login — from the first anonymous browser request to the authenticated session cookie.

It crosses apps, platform, and gateway config, so it lives in topics/. The actors involved:

Actor	Role in the identity stack	Where it lives in docs
LLDAP (account.web.kueber.eu)	The user directory — accounts and groups	apps/lldap
Keycloak (keycloak.web.kueber.eu)	OIDC/SAML broker, federates users from LLDAP	apps/keycloak
Envoy Gateway (SecurityPolicy OIDC at the edge)	Enforces login in front of apps that can't	platform/envoy-gateway
CNPG (Postgres for Keycloak state)	Realm, client, and session storage	platform/cloudnative-pg
SOPS (client secrets committed encrypted)	Per-app OIDC credentials	operations/sops
Network policies (default-deny around both)	Who may even open a connection	components/app-network-policy

The shape of the stack

The division of labour, in one sentence each:

• LLDAP stores the humans. A lightweight LDAP server, one Recreate Deployment in the lldap namespace, SQLite on a Longhorn PVC. It exposes LDAP on service port 389 (container 3890) and its web UI on 17170, published as account.web.kueber.eu. It does not speak OIDC at all.
• Keycloak turns directory entries into OIDC. The heavyweight broker — realm kueber.eu, user federation from LLDAP (read-only, configured in the admin console, not in manifests), two replicas whose Infinispan caches replicate over JGroups (KC_CACHE: ispn, DNS_PING against the keycloak-headless Service).
• Pocket ID used to be the passkey-first second provider — no passwords, WebAuthn only, with its own LDAP_* sync against the directory. It was decommissioned in August 2026 once its last clients were re-registered in Keycloak; the page stays as a record of what it was and why it went.

Keycloak keeps its state in a dedicated CNPG Postgres cluster; LLDAP stays on SQLite. Both are delivered by the same GitOps loop as everything else — each has its own Flux Kustomization + OCIRepository pair under k8s/clusters/talos/.

Who authenticates against what

Every verified client relationship in the repo — all against the kueber.eu realm, the difference being where the OIDC handshake happens:

App	Provider	Where the OIDC config lives
FileBrowser Quantum	Keycloak (kueber.eu realm, client filebrowser)	Envoy Gateway SecurityPolicy oidc-filebrowser — auth happens before traffic reaches the app
ironscan	Keycloak (kueber.eu realm, client ironscan)	Envoy Gateway SecurityPolicy oidc-ironscan, same pattern
voyage	Keycloak (kueber.eu realm, client voyage)	In-app: VOYAGE_OIDC_KEYCLOAK_ISSUER env in the Deployment patch, secret in SOPS
kindred	Keycloak	In-app: KINDRED_OIDC_KEYCLOAK_* keys in the SOPS secret
cairn	Keycloak (kueber.eu realm, client cairn)	In-app: CAIRN_OIDC_PROVIDERS / CAIRN_OIDC_KEYCLOAK_* in the config ConfigMap, client secret in SOPS; links to an existing account on a verified matching email
Stump	Keycloak (kueber.eu realm, client stump)	In-app: STUMP_OIDC_* env in the stump-oidc ConfigMap, client secret in SOPS; callback /api/v2/auth/oidc/callback
Gitea	Keycloak	In-app: OIDC section of app.ini, mounted from a SOPS secret
Immich	Keycloak	In-app: configured in the admin UI, not in manifests
Memos	Keycloak (kueber.eu realm, client memos)	In-app: deployment-managed memos-idp-keycloak.json from the SOPS memos-idp Secret under /etc/secrets; callback /auth/callback
Gatus	Keycloak	In-app: issuer-url: https://keycloak.web.kueber.eu/realms/kueber.eu in the config ConfigMap, client ID/secret from SOPS
Vikunja	Keycloak	In-app: VIKUNJA_AUTH_OPENID_PROVIDERS_KEYCLOAK_* keys in the vikunja-openid SOPS secret
cadence	Keycloak	In-app: OIDC_KEYCLOAK_ISSUER_URL / _AUDIENCE / _CLIENT_ID keys in the SOPS secret
RomM	Keycloak	In-app: OIDC_ENABLED / OIDC_PROVIDER / OIDC_CLIENT_* keys in the SOPS secret

A few apps are OIDC consumers whose issuer only exists inside an encrypted value, so the repo doesn't show the URL — but they were all moved to Keycloak in the August 2026 migration: Grafana (GF_AUTH_GENERIC_OAUTH_*, with PKCE), Outline (OIDC_AUTH_URI and friends), Nextcloud (Social Login app), and Miniflux (OAUTH2_*). And one is deliberately not wired up: Jellyfin keeps its own user store — the SSO plugin exists but isn't worth it for a handful of accounts.

The accounting app is the in-between case worth knowing about: a gateway SecurityPolicy (oidc-accounting, issuer https://keycloak.web.kueber.eu/realms/kueber.eu) exists in its directory but is commented out of the kustomization — the app currently does its own OIDC via OIDC_ISSUER / OIDC_CLIENT_* keys in its SOPS secret instead.

The Kubernetes API logs in the same way

Since August 2026 kubectl is one more Keycloak client. Both clusters configure the kube-apiserver with a structured KubeAuthenticationConfig (the Talos patch control-plane/30-oidc.yaml), not with the older --oidc-* flags:

Setting	Value
Issuer	https://keycloak.web.kueber.eu/realms/kueber.eu
Audience	kubernetes
Username	claim preferred_username, prefixed oidc:
Groups	claim groups, prefixed oidc:
Admission rule	the token must carry the group /homelab/k8s/talos (on the edge cluster /homelab/k8s/edge)

The validation rule rejects any token without the cluster's group before RBAC is even consulted, so a realm user who is not in that group cannot authenticate at all. For those who are, a single ClusterRoleBinding maps the group oidc:/homelab/k8s/talos to cluster-admin. There is no finer-grained role today; the group is the permission.

On the workstation, kubectl obtains the token through an OIDC exec credential plugin that opens the browser against Keycloak, so cluster access carries the same second factor as every other login.

Keycloak runs on the cluster it authenticates for, which makes a break-glass path mandatory: the client certificate from topf kubeconfig does not depend on Keycloak and keeps working when the realm is unreachable. Anonymous access stays enabled for exactly three paths, /livez, /readyz and /healthz, because the health probes rely on them.

Two login flows, hop by hop

Flow A — OIDC enforced at the gateway (filebrowser, ironscan)

The app itself never sees an unauthenticated request. Envoy does the whole OAuth2 dance:

Two hard-won details are encoded in comments in k8s/apps/filebrowser-quantum/talos/security-policy.yaml and are worth repeating here:

1. The hairpin problem. In-cluster DNS for keycloak.web.kueber.eu points back at the gateway, so Envoy can't just "call the issuer" — the Backend resource with wellKnownCACertificates: System and explicit SNI is what makes the token exchange leave through the front door and validate TLS properly, with no manifest edits when Let's Encrypt rotates chains.
2. Discovery is skipped on purpose. authorizationEndpoint and tokenEndpoint are declared explicitly because the Envoy Gateway controller's OIDC discovery client (through v1.8.3) builds an empty root CA pool from a name-only CA entry when the Backend uses system trust — failing translation and 500ing the whole route.

The gateway pattern gates every path on the route. FileBrowser Quantum carves anonymous share links (/public, plus the legacy /share redirect) out into a second HTTPRoute the policy deliberately does not target; ironscan accepts that even /healthz is gated externally, since kubelet probes hit the pod directly.

Flow B — in-app OIDC (Gatus against Keycloak)

Here the app is a regular OIDC client and the gateway just proxies:

The same shape holds for Vikunja, cadence, RomM, voyage, kindred, cairn, Stump, Gitea, and Immich — only the redirect path and the env-var spelling differ per app. The practical difference from Flow A: here the app holds the client secret and issues its own session, so a Keycloak outage breaks new logins but not sessions already established.

Keycloak's split personality: two hostnames

Keycloak is the only app here with two HTTPRoutes on purpose:

• keycloak.web.kueber.eu (public gateway) exposes only /realms/, /resources/, and /.well-known/ — the endpoints OIDC clients and browsers need. KC_HOSTNAME points here.
• keycloak.int.kueber.eu (internal-only route, via the http-internal resource) carries the admin console. KC_HOSTNAME_ADMIN points here, so admin URLs are simply never generated on the public host.

Hitting the admin console on the public hostname therefore 404s by route-match, not by authorization — the path prefixes just aren't routed. Both routes rewrite X-Frame-Options / Content-Security-Policy headers so the account and admin consoles can frame themselves.

Behind the routes, KC_PROXY_HEADERS: xforwarded plus KC_PROXY_TRUSTED_ADDRESSES: 10.100.0.0/16 make Keycloak trust forwarded client IPs only from the Envoy CIDR — this is what keeps real addresses in the audit log, the same story as Topics → Real client IPs across the chain.

Where the secrets live

Every credential in this stack is committed encrypted and decrypted in-cluster by Flux (SOPS):

• LLDAP: LLDAP_JWT_SECRET, LLDAP_KEY_SEED, LLDAP_LDAP_BASE_DN, LLDAP_LDAP_USER_PASS in the lldap Secret.
• Keycloak: KEYCLOAK_ADMIN / KEYCLOAK_ADMIN_PASSWORD in keycloak-env; DB credentials come from the CNPG-generated Secret.
• Per app: one small secret each — filebrowser-oidc and ironscan-oidc hold just client-secret; app-level clients hold issuer + ID + secret together. Rotating a client secret means changing it in the provider and in the SOPS secret — they are two copies of the same value, and nothing reconciles them for you.

What can break, and where to look

Symptom	Most likely cause	Where to look first
Every Keycloak-gated app fails at once	Keycloak down, or its CNPG cluster unhealthy	apps/keycloak · platform/cloudnative-pg
Gateway-OIDC route returns 500 before any login page	EG OIDC discovery / Backend CA translation bug — endpoints must stay explicit	security-policy.yaml comments in the filebrowser / ironscan overlays
Login works but the app rejects the callback	Client secret rotated in the provider but not in the SOPS secret (or vice versa)	the app's secret-oidc.yaml / provider client config
Password change doesn't take effect in Keycloak logins	LLDAP federation sync — accounts live in LLDAP, Keycloak only brokers	LLDAP web UI at account.web.kueber.eu, then Keycloak federation settings
Keycloak sessions drop on every deploy	JGroups cluster not forming — DNS_PING needs the keycloak-headless Service	keycloak-headless endpoints; KC_CACHE_STACK env
Admin console unreachable "from the internet"	By design — it only exists on keycloak.int.kueber.eu	the http-admin overlay in k8s/apps/keycloak/talos/
An LDAP consumer can't reach LLDAP	Default-deny: the lldap namespace only admits the public gateway, vmagent from the monitoring namespace, and same-namespace pods — no manifest currently allows cross-namespace port 389	components/app-network-policy
OIDC works, but audit logs show the gateway's IP	Forwarded-header trust chain	Topics → Real client IPs

Why one OIDC provider, and why it took a while

For a year and a half the cluster ran two: Keycloak for the directory-backed accounts, and Pocket ID for the apps where a passwordless, passkey-only flow was the whole point. Pocket ID is a lovely piece of software for a homelab — one small binary, WebAuthn and nothing else, almost no operational weight — and that is exactly why it lingered. But two providers meant two client registries, two places to rotate a secret, two audit trails, and a per-app decision about which column an app belonged in. Keycloak brings mature OIDC and SAML, realm isolation, LDAP federation, and role-gated authentication flows, so in August 2026 every remaining Pocket ID client was re-registered in the kueber.eu realm and Pocket ID was decommissioned. Keycloak is now the single point of failure for every login, which is why its state rides the standard backup and disaster-recovery paths like every other CNPG app — and why the gateway-enforced apps (Flow A) are the ones to watch, since they have no session of their own to fall back on.