Identity end-to-end: LLDAP, Keycloak, and Pocket ID

Three services in the cluster carry the word "identity", and it is easy to lose track of who does what. This page draws the whole picture once: where user accounts actually live, which of the two OIDC providers each app trusts, and every hop of a login — from the first anonymous browser request to the authenticated session cookie.

It crosses apps, platform, and gateway config, so it lives in topics/. The actors involved:

Actor	Role in the identity stack	Where it lives in docs
LLDAP (account.web.kueber.eu)	The user directory — accounts and groups	apps/lldap
Keycloak (keycloak.web.kueber.eu)	OIDC/SAML broker, federates users from LLDAP	apps/keycloak
Pocket ID (pocketid.web.kueber.eu)	Passkey-first OIDC provider	apps/pocketid
Envoy Gateway (SecurityPolicy OIDC at the edge)	Enforces login in front of apps that can't	platform/envoy-gateway
CNPG (Postgres for Keycloak and Pocket ID state)	Realm, client, session, passkey storage	platform/cloudnative-pg
SOPS (client secrets committed encrypted)	Per-app OIDC credentials	operations/sops
Network policies (default-deny around all three)	Who may even open a connection	components/app-network-policy

The shape of the stack

The division of labour, in one sentence each:

• LLDAP stores the humans. A lightweight LDAP server, one Recreate Deployment in the lldap namespace, SQLite on a Longhorn PVC. It exposes LDAP on service port 389 (container 3890) and its web UI on 17170, published as account.web.kueber.eu. It does not speak OIDC at all.
• Keycloak turns directory entries into OIDC. The heavyweight broker — realm kueber.eu, user federation from LLDAP (read-only, configured in the admin console, not in manifests), two replicas whose Infinispan caches replicate over JGroups (KC_CACHE: ispn, DNS_PING against the keycloak-headless Service).
• Pocket ID is the passkey-first alternative. No passwords, WebAuthn only. It carries a full LDAP_* configuration block in its SOPS secret (LDAP_URL, LDAP_BIND_DN, user/group search filters, attribute mappings up to LDAP_ADMIN_GROUP_NAME) pointing sync at the directory — the values, including LDAP_ENABLED itself, are encrypted, so the wiring is in Git but its on/off state is not readable from the repo.

Both providers keep their state in dedicated CNPG Postgres clusters; LLDAP alone stays on SQLite. All three are delivered by the same GitOps loop as everything else — each has its own Flux Kustomization + OCIRepository pair under k8s/clusters/talos/.

Who authenticates against what

Every verified client relationship in the repo, by provider:

App	Provider	Where the OIDC config lives
filebrowser	Keycloak (kueber.eu realm, client filebrowser)	Envoy Gateway SecurityPolicy oidc-filebrowser — auth happens before traffic reaches the app
ironscan	Keycloak (kueber.eu realm, client ironscan)	Envoy Gateway SecurityPolicy oidc-ironscan, same pattern
voyage	Keycloak (kueber.eu realm, client voyage)	In-app: VOYAGE_OIDC_KEYCLOAK_ISSUER env in the Deployment patch, secret in SOPS
kindred	Keycloak	In-app: KINDRED_OIDC_KEYCLOAK_* keys in the SOPS secret
Gitea	Keycloak	In-app: OIDC section of app.ini, mounted from a SOPS secret
Immich	Keycloak	In-app: configured in the admin UI, not in manifests
Gatus	Pocket ID	In-app: issuer-url: https://pocketid.web.kueber.eu in the config ConfigMap, client ID/secret from SOPS
Vikunja	Pocket ID	In-app: VIKUNJA_AUTH_OPENID_PROVIDERS_POCKETID_* keys in the vikunja-openid SOPS secret
cadence	Pocket ID	In-app: OIDC_POCKETID_ISSUER_URL / _AUDIENCE / _CLIENT_ID keys in the SOPS secret
RomM	Pocket ID	In-app: OIDC_ENABLED / OIDC_PROVIDER / OIDC_CLIENT_* keys in the SOPS secret

A few apps are OIDC consumers whose issuer only exists inside an encrypted value, so the repo doesn't say which provider they point at: Grafana (GF_AUTH_GENERIC_OAUTH_*, with PKCE), Outline (OIDC_AUTH_URI and friends), Nextcloud (Social Login app), and Miniflux (OAUTH2_*). And one is deliberately not wired up: Jellyfin keeps its own user store — the SSO plugin exists but isn't worth it for a handful of accounts.

The accounting app is the in-between case worth knowing about: a gateway SecurityPolicy (oidc-accounting, issuer https://pocketid.web.kueber.eu) exists in its directory but is commented out of the kustomization — the app currently does its own OIDC via OIDC_ISSUER / OIDC_CLIENT_* keys in its SOPS secret instead.

Two login flows, hop by hop

Flow A — OIDC enforced at the gateway (filebrowser, ironscan)

The app itself never sees an unauthenticated request. Envoy does the whole OAuth2 dance:

Two hard-won details are encoded in comments in k8s/apps/talos/filebrowser/security-policy.yaml and are worth repeating here:

1. The hairpin problem. In-cluster DNS for keycloak.web.kueber.eu points back at the gateway, so Envoy can't just "call the issuer" — the Backend resource with wellKnownCACertificates: System and explicit SNI is what makes the token exchange leave through the front door and validate TLS properly, with no manifest edits when Let's Encrypt rotates chains.
2. Discovery is skipped on purpose. authorizationEndpoint and tokenEndpoint are declared explicitly because the Envoy Gateway controller's OIDC discovery client (through v1.8.3) builds an empty root CA pool from a name-only CA entry when the Backend uses system trust — failing translation and 500ing the whole route.

The gateway pattern gates every path on the route. filebrowser carves anonymous share links (/share, /static, /api/public) out into a second HTTPRoute the policy deliberately does not target; ironscan accepts that even /healthz is gated externally, since kubelet probes hit the pod directly.

Flow B — in-app OIDC (Gatus against Pocket ID)

Here the app is a regular OIDC client and the gateway just proxies:

The same shape holds for Vikunja, cadence, RomM (all Pocket ID) and for voyage, kindred, Gitea, Immich (all Keycloak) — only the redirect path and the env-var spelling differ per app.

Keycloak's split personality: two hostnames

Keycloak is the only app here with two HTTPRoutes on purpose:

• keycloak.web.kueber.eu (public gateway) exposes only /realms/, /resources/, and /.well-known/ — the endpoints OIDC clients and browsers need. KC_HOSTNAME points here.
• keycloak.int.kueber.eu (internal-only route, via the http-internal resource) carries the admin console. KC_HOSTNAME_ADMIN points here, so admin URLs are simply never generated on the public host.

Hitting the admin console on the public hostname therefore 404s by route-match, not by authorization — the path prefixes just aren't routed. Both routes rewrite X-Frame-Options / Content-Security-Policy headers so the account and admin consoles can frame themselves.

Behind the routes, KC_PROXY_HEADERS: xforwarded plus KC_PROXY_TRUSTED_ADDRESSES: 10.100.0.0/16 make Keycloak trust forwarded client IPs only from the Envoy CIDR — this is what keeps real addresses in the audit log, the same story as Topics → Real client IPs across the chain.

Where the secrets live

Every credential in this stack is committed encrypted and decrypted in-cluster by Flux (SOPS):

• LLDAP: LLDAP_JWT_SECRET, LLDAP_KEY_SEED, LLDAP_LDAP_BASE_DN, LLDAP_LDAP_USER_PASS in the lldap Secret.
• Keycloak: KEYCLOAK_ADMIN / KEYCLOAK_ADMIN_PASSWORD in keycloak-env; DB credentials come from the CNPG-generated Secret.
• Pocket ID: ENCRYPTION_KEY, the whole LDAP_* block, and SMTP relay credentials mapped from the shared allinkl-smtp-credentials Secret (the email component) for account-event mails.
• Per app: one small secret each — filebrowser-oidc and ironscan-oidc hold just client-secret; app-level clients hold issuer + ID + secret together. Rotating a client secret means changing it in the provider and in the SOPS secret — they are two copies of the same value, and nothing reconciles them for you.

What can break, and where to look

Symptom	Most likely cause	Where to look first
Every Keycloak-gated app fails at once	Keycloak down, or its CNPG cluster unhealthy	apps/keycloak · platform/cloudnative-pg
Gateway-OIDC route returns 500 before any login page	EG OIDC discovery / Backend CA translation bug — endpoints must stay explicit	security-policy.yaml comments in the filebrowser / ironscan overlays
Login works but the app rejects the callback	Client secret rotated in the provider but not in the SOPS secret (or vice versa)	the app's secret-oidc.yaml / provider client config
Password change doesn't take effect in Keycloak logins	LLDAP federation sync — accounts live in LLDAP, Keycloak only brokers	LLDAP web UI at account.web.kueber.eu, then Keycloak federation settings
Keycloak sessions drop on every deploy	JGroups cluster not forming — DNS_PING needs the keycloak-headless Service	keycloak-headless endpoints; KC_CACHE_STACK env
Admin console unreachable "from the internet"	By design — it only exists on keycloak.int.kueber.eu	the http-admin overlay in k8s/apps/talos/keycloak/
An LDAP consumer can't reach LLDAP	Default-deny: the lldap namespace only admits the gateway, homepage, Prometheus, and same-namespace pods — no manifest currently allows cross-namespace port 389	components/app-network-policy
Passkey login fails for one user	Pocket ID passkey registration — WebAuthn is per-device	apps/pocketid admin UI
OIDC works, but audit logs show the gateway's IP	Forwarded-header trust chain	Topics → Real client IPs

Why two OIDC providers at all

It looks redundant, and deliberately so — the Keycloak page lists Pocket ID as an "alternative considered" that runs alongside rather than replacing it. Keycloak brings mature OIDC and SAML, realm isolation, and LDAP federation, so it fronts the directory-backed accounts. Pocket ID brings passkey-only login with almost no operational weight, so it serves the apps where a passwordless flow is the whole point. Both are one Deployment plus one CNPG database; neither is load-bearing for the other. Losing Pocket ID locks you out of the Pocket ID column above — and nothing else. Losing Keycloak is the bigger event, which is why its state rides the standard backup and disaster-recovery paths like every other CNPG app.