Certificate lifecycle: every cert in the homelab, from Let's Encrypt to the kubelet
Every TLS handshake in the homelab terminates against a certificate that some machine issued, some machine renews, and — in one case — some machine has to explicitly approve. This page maps all of them. There are three distinct certificate systems running side by side, and they never share a CA:
- Public TLS — Let's Encrypt certificates, issued by cert-manager via a DNS-01 webhook for the ALL-INKL registrar, consumed by Envoy Gateway listeners.
- Kubelet serving certificates — issued by the Kubernetes cluster CA through the CSR API, auto-approved by kubelet-serving-cert-approver.
- Talos' own PKI — the OS and control-plane trust roots (
talosctlmTLS, etcd, the cluster CA itself), generated once at cluster creation and stored intalsecret.sops.yaml(see Talos).
It crosses every section of the docs, so it lives in topics/. The actors:
| Actor | Where it lives in docs |
|---|---|
| cert-manager (issues + renews) | platform/cert-manager |
| allinkl-webhook (solves DNS-01 challenges at ALL-INKL) | platform/cert-manager-webhook-all-inkl |
| Envoy Gateway (terminates TLS) | platform/envoy-gateway |
| kubelet-serving-cert-approver (approves node serving CSRs) | platform/kubelet-serving-cert-approver |
| metrics-server (the main consumer of kubelet serving certs) | platform/metrics-server |
| Talos (owns the cluster CA and node identity) | foundation/talos |
| SOPS + age (keeps the KAS credentials and Talos PKI in Git) | operations/sops |
| Flux (deploys all of the above) | foundation/flux |
The three planes, side by side
The rest of this page walks each column.
Plane 1 — public TLS via cert-manager
The issuer
Both clusters run a single ClusterIssuer named allinkl-issuer (k8s/platform/configs/<cluster>/cert-manager/cluster-issuer.yaml). It points at the Let's Encrypt production endpoint with the tlsserver ACME profile, and stores the ACME account key in the Secret allinkl-webhook-account-key. Every solver is DNS-01 — there is no HTTP-01 anywhere, because the certificates are mostly wildcards and several hostnames resolve to private IPs, neither of which HTTP-01 can handle.
The DNS-01 solver is not a built-in: ALL-INKL (the registrar, KAS API a.k.a. kasserver) has no upstream cert-manager integration, so a custom webhook fills the gap.
The all-inkl webhook
cert-manager-webhook-all-inkl deploys as HelmRelease allinkl-webhook (chart allinkl-webhook from oci://ghcr.io/johnnycube), running in the cert-manager namespace. It registers under the API group acme.johanneskueber.com with solver name allinkl. When cert-manager needs to prove domain ownership, it calls the webhook; the webhook logs into the KAS API and creates the _acme-challenge TXT record, Let's Encrypt validates it, and the webhook cleans it up.
The KAS credentials live in SOPS-encrypted Secrets (see operations/sops), and RBAC is deliberately tight: a Role (all-inkl:credentials-reader) grants the allinkl-webhook ServiceAccount get — not list, not watch — on exactly the named credential Secrets.
The two clusters differ in one interesting way:
- Production (talos) has one solver and one credential Secret,
allinkl-credentials. - Edge has two KAS accounts. A
dnsZonesselector routes challenges foropenbeehive.org,opencairn.org,bucketlist-tracker.com,knpsck.net,sproutatlas.com, andtri-gains.comto the credentials inallinkl-credentials-w021ae5b; everything else (johanneskueber.com,kueber.eu,seitenraum.com, …) falls through toallinkl-credentials-w0097b9d. Same issuer name, same webhook — the selector picks the account.
From annotation to Envoy listener
No Certificate resources are written by hand. cert-manager runs with enableGatewayAPI: true (set via ControllerConfiguration in the cert-manager values), which enables the gateway-shim: it watches Gateway resources for the cert-manager.io/cluster-issuer annotation and creates one Certificate per TLS listener, targeting the Secret named in that listener's certificateRefs.
Every Gateway carries the same annotation block:
metadata:
annotations:
cert-manager.io/cluster-issuer: allinkl-issuer
cert-manager.io/private-key-algorithm: ECDSA
cert-manager.io/private-key-size: "384"
cert-manager.io/private-key-encoding: PKCS8
cert-manager.io/private-key-rotation-policy: Always
cert-manager.io/usages: "digital signature,key agreement,server auth"
So every public cert is ECDSA P-384, and rotation-policy: Always means each renewal generates a fresh private key rather than re-signing the old one.
The resulting listener → Secret map (all Secrets in envoy-gateway-system):
| Cluster | Gateway | Hostname | Secret |
|---|---|---|---|
| talos | public | *.web.kueber.eu | wildcard-public-web-kueber-eu-tls |
| talos | internal | *.int.kueber.eu | wildcard-internal-int-kueber-eu-tls |
| talos | internal | *.talos.int.kueber.eu | wildcard-internal-talos-int-kueber-eu-tls |
| edge | internal | *.edge.int.kueber.eu | wildcard-internal-edge-int-kueber-eu-tls |
| edge | public-seitenraum | *.seitenraum.com | seitenraum-wildcard-com-tls |
| edge | public-seitenraum | seitenraum.com | seitenraum-com-tls |
| edge | public-openbeehive | app.openbeehive.org | app-openbeehive-org-tls |
One deliberate absence: the edge Gateway public-kueber has a *.web.kueber.eu listener in TLS Passthrough mode — no certificate at all. The edge cluster forwards those handshakes untouched to the production cluster, which terminates them with its own wildcard-public-web-kueber-eu-tls. Why the traffic is relayed that way (and what the PROXY protocol has to do with it) is its own topic: Envoy Gateway PROXY protocol v2.
Renewal
Renewal is a non-event by design. cert-manager tracks expiry on every Certificate and re-runs the whole ACME order well before the certificate runs out. The issuers request Let's Encrypt's tlsserver profile, which has issued 45-day certificates since May 2026; cert-manager renews at two thirds of the lifetime, so on day 30 with 15 days to spare — new key (because rotation-policy: Always), new order, new TXT record via the webhook, new signed cert written into the same Secret. Envoy Gateway watches the Secret and hot-swaps the served certificate; no pod restarts, no Gateway edits, no human. If renewal breaks, Gatus notices first — it validates the certificate chain on every public URL it probes.
In-cluster consumers of cert-manager
Public listeners are not the only customers. KubeVela runs its admission webhook with admissionWebhooks.certManager.enabled: true, so cert-manager issues and rotates that webhook's serving certificate instead of relying on chart-generated self-signed certs. cert-manager's own webhook and cainjector bootstrap their trust internally.
Plane 2 — kubelet serving certificates
Kubernetes auto-approves kubelet client certificates but, for good security reasons, refuses to auto-approve kubelet serving certificates. Without intervention every kubelet serves a self-signed cert on :10250, and anything that scrapes it — metrics-server above all — must pass --kubelet-insecure-tls.
The homelab closes this loop in two halves:
- Talos side. Both clusters apply the
all/40-kubelet.yaml.tplpatch, which setsrotate-server-certificates: truein the kubelet's extra args. The kubelet then requests its serving cert through the Kubernetes CSR API (signerkubernetes.io/kubelet-serving) instead of self-signing — and re-requests it periodically as it approaches expiry. - Cluster side. kubelet-serving-cert-approver watches for exactly those CSRs, validates that the requesting node is asking for names and IPs it legitimately owns, and approves them. The cluster CA signs; the kubelet picks up the cert.
Deployment is unusual for this repo: no Helm chart, no local manifests. A Flux GitRepository points straight at the upstream repo pinned to a release tag, filtered down to a single file (/deploy/standalone-install.yaml), and a Flux Kustomization named kubelet-serving-cert-approver-standalone applies it into the kubelet-serving-cert-approver namespace (Pod Security enforced at restricted). Renovate bumps the tag like any other dependency — see Renovate policy.
The payoff is one flag that isn't there: metrics-server runs without --kubelet-insecure-tls, scraping every kubelet over properly verified TLS.
Approval is the interesting failure mode here: if the approver is down, nodes still join and run workloads (client certs are unaffected), but a kubelet rotating its serving cert will sit with a Pending CSR — and metrics for that node go stale.
Plane 3 — Talos' internal PKI
Underneath both planes sits the PKI that Talos generates when the cluster is born. talsecret.sops.yaml — one per cluster, SOPS-encrypted in Git — holds the cluster CA, the machine tokens, and the rest of the trust material. topf renders it into every node's machine config (see Talos for the full flow).
This PKI never touches cert-manager:
talosctlis mTLS-only. The Talos API on every node authenticates the admin by client certificate; there is no password, no SSH.- The Kubernetes cluster CA that signs kubelet serving certs in Plane 2 comes from here.
- etcd peer and client certs are issued from their own CA in the same secret bundle.
- Kubelet client certs rotate automatically against the cluster CA with no approver needed — only the serving side needs Plane 2.
Rotation is the blunt instrument of the three planes: per the Talos recovery table, a cluster CA rotation means regenerating secrets and rolling the cluster node by node — and a leaked talsecret is a wipe-and-reinstall, not a rotation. That's acceptable precisely because this PKI's private material only ever exists inside SOPS and on the nodes themselves.
What can break, and where to look
| Symptom | Most likely cause | Where to look first |
|---|---|---|
| Browser warns on a public URL, cert expired | Renewal failed silently for ~15 days | kubectl get certificate -A; then the CertificateRequest/Order/Challenge chain |
Challenge stuck in pending | allinkl-webhook can't write the TXT record (KAS creds wrong/rotated) | allinkl-webhook logs in cert-manager ns; secrets rotation |
| Challenge pending for one zone only (edge) | Wrong KAS account matched — dnsZones selector on allinkl-issuer | k8s/platform/configs/edge/cert-manager/cluster-issuer.yaml |
| New Gateway listener never gets a cert | Annotation missing, or gateway-shim disabled | Gateway annotations; enableGatewayAPI in the cert-manager values |
| Envoy serves the default/stale cert | Secret name in certificateRefs doesn't match what cert-manager wrote | Secrets in envoy-gateway-system vs. the listener's certificateRefs |
kubectl top node empty / metrics-server TLS errors | Kubelet serving CSR unapproved | kubectl get csr; kubelet-serving-cert-approver logs |
talosctl refused: certificate error | Stale talosconfig client cert | Regenerate via topf talosconfig > talosconfig; see Talos |
| KubeVela webhook rejects everything after upgrade | Webhook serving cert / CA bundle mismatch | cert-manager-issued webhook cert in the KubeVela namespace |
Why this is a topic, not three product pages
Each component's page (cert-manager, cert-manager-webhook-all-inkl, kubelet-serving-cert-approver, Talos) documents its own installation and defaults. What none of them shows is the boundary lines: which CA signs what, where trust starts, and which of the three planes a given TLS error actually belongs to. When a handshake fails, the first diagnostic question is always "which column of the diagram is this?" — and that question only has an answer on a page that draws all three.