Incident checklist
For the moment something is down and the question is where to start. The page does three things: narrows the fault to a layer, gives the first move for every alert the cluster raises, and points to the detailed failure tables that the topic pages already carry.
The first five minutes
Work from the outside in. Each step either clears a layer or names it.
1. Is it everything public, or one app? Open Gatus. All public endpoints red at once is almost never thirty broken apps; it is the path in front of them: the edge cluster, the NetBird link between edge and production, or the public gateway. One red endpoint is that app.
2. Is the mesh up? Internal hostnames resolve and route only through NetBird. If nothing under *.int.kueber.eu answers, check that the workstation is connected before suspecting the cluster.
3. Is the cluster healthy?
kubectl get nodes
flux get kustomizations | grep -v True
kubectl get pods -A | grep -v -e Running -e Completed
A NotReady node explains most of what follows it. A failing Kustomization names the app and usually the resource. The pod list shows what is crash-looping or pending right now.
4. Is storage healthy?
kubectl get volumes.longhorn.io -n longhorn-system \
-o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness | grep -v healthy
Pods stuck in ContainerCreating or Pending on a healthy node are a volume problem until proven otherwise.
5. What changed? git log --since=yesterday --oneline in the repository. Renovate merges digest updates overnight; a break that started in the early morning often has a commit with its name on it, and reverting that commit is a legitimate first fix.
When there is no alert at all
Alerting, notification and status page all run on the production cluster. If the whole cluster is down, nothing reports it: the signal is silence, and a Gatus page that does not load. This is a known gap, described in Observability & alerting. When everything is quiet and something still feels wrong, start at step 3.
Alert by alert
| Alert | What it means | First action |
|---|---|---|
FluxReconciliationFailure | A Flux object has been not-ready for 30 minutes | flux get all -A | grep -v True, read the message; for a HelmRelease that has given up retrying, flux reconcile helmrelease <name> -n <namespace> --reset |
FluxResourceSuspended | Something has been suspended for six hours | Somebody paused it and forgot. flux resume it, or remove the suspension from Git if it was committed |
KubePodCrashLooping | A container keeps dying on start | kubectl logs --previous on the pod. After a restore or an upgrade, most often a database that is not there yet |
KubeContainerOOMKilled | A container hit its memory limit | Check whether it is a leak or a limit that was always too tight; raise the limit in the app's overlay |
KubeContainerRestartStorm | More than three restarts in an hour | Same as crash-looping, earlier. Look at the probe timings as well as the logs |
KubeDeploymentReplicasUnavailable | A rollout is stuck or a pod cannot schedule | kubectl describe the newest pod: image pull, pending volume or failing readiness probe |
KubeStatefulSetReplicasMismatch | Same, for a StatefulSet | Same; StatefulSets wait for each ordinal, so the first unhealthy pod blocks the rest |
LonghornVolumeDegraded | A volume is running with fewer replicas than configured | Expected for up to an hour after a node reboot. Persisting longer: look for a node or disk that is not schedulable |
LonghornVolumeFaulted | A volume has no healthy replica | Stop the workload. If a replica can be salvaged in the Longhorn UI, do that; otherwise restore the volume |
LonghornNodeNotReady | Longhorn cannot use a node | Is the Kubernetes node up? If yes, check the Longhorn manager pod on it and the node's disk status |
LonghornDiskUsageHigh | A data disk is over 85% | Find the largest volumes; old snapshots and orphaned replicas are the usual reclaimable space |
K8upScheduleLastJobFailed | The last job of a backup schedule failed | kubectl get jobs -n <namespace> and the failed pod's log. Repository locked by a killed job is the common cause |
K8upJobFailed | A single backup, check or prune job failed | Same |
K8upPreBackupFailed | The database dump before a backup failed | That night's backup has no database in it. Check the pre-backup pod's log; a DNS failure resolving cnpg-rw is the known recurring cause, see backups-cnpg |
K8upBackupNotRunning | A schedule exists but no job has run | Is the k8up operator running? kubectl get pods -n k8up-system |
CertManagerCertExpiringSoon | A certificate has under 14 days left | Renewal should have happened at day 30 of 45. Follow the chain Certificate → CertificateRequest → Order → Challenge, see Certificate lifecycle |
CertManagerCertNotReady | A certificate has been not-ready for an hour | Same chain; a stuck challenge usually means the DNS webhook cannot write the record |
CrossplaneManagedResourceNotSynced | A workspace cannot apply its OpenTofu | kubectl describe workspace in the app's namespace; provider credentials or an API outage at Hetzner, Backblaze or NetBird |
CrossplaneManagedResourceNotReady | A workspace has not become ready | Same |
Getting in when the usual way is down
- Keycloak is down, so
kubectlcannot log in. The certificate-based kubeconfig does not depend on it:topf kubeconfig > kubeconfigfromtalos/talos. - The Kubernetes API is down.
talosctltalks to the nodes directly:talosctl --talosconfig talosconfig -n <node ip> dmesg,services,etcd members. - A node is unresponsive. Its VM can be inspected and reset from the Proxmox host that carries it; see Node maintenance for when a reset is the right call.
- The mesh is down. Everything above needs a route into the production networks. Without NetBird that means being on site.
Where the detailed tables are
Each subsystem's page ends with a "what can break, and where to look" table. When the layer is known, go there:
| Layer | Page |
|---|---|
| A request does not arrive | Life of a request |
| Mesh, routing, internal DNS | NetBird connectivity |
| Certificates | Certificate lifecycle |
| Logins | Identity end-to-end |
| Deployments not rolling out | GitOps flow, Image supply chain |
| Volumes | Storage tiers |
| Admission rejections, network policy | Policy & runtime security |
| Missing or wrong alerts | Observability & alerting |
| GPU, transcoding | Hardware acceleration |
| Edge versus production placement | Multi-cluster topology |
When the incident is over and the cause was not on any of those tables, add the row.