OpenBao
A secret store that applications read from at runtime — instead of a secret that lives in a manifest.
OpenBao is the Linux Foundation’s fork of HashiCorp Vault, kept under the MPL after Vault moved to the BSL: the same API, CLI, storage format and plugin surface, with its own additions (static-seal auto-unseal, declarative self-initialisation, OCI plugin distribution). It stores key/value secrets encrypted at rest, hands them out to authenticated clients under ACL policies, and writes an audit line for every request. Here it is the backend of a second External Secrets store, so an app can read a credential that was created or rotated in OpenBao without the value ever passing through Git. Its transit engine also holds the key CI signs the app artifacts with: the key signs inside OpenBao and never leaves it.
Alternatives considered
Self Hosted
| Tool | Open Source | Full Features | Notes |
|---|---|---|---|
| HashiCorp Vault | BSL | Yes | The original; the licence is the reason the fork exists, otherwise identical for this use |
| Infisical | Yes | Partly | Developer-facing UI and sync agents; needs Postgres and Redis, no raft, no plugin surface |
| Bitwarden Secrets Manager | Partly | Partly | Would ride on Vaultwarden’s data model, but Vaultwarden does not implement it |
| SOPS alone | Yes | Partly | What every secret here uses today: perfect for values that change with a commit, blind to values that change at runtime |
SOPS stays the source for everything that is configuration. OpenBao covers what SOPS cannot: a credential generated or rotated by a running system, read by another one, with an audit trail and a policy in between.
Installation
Architecture
- Deployment: Single
openbaoDeployment in theopenbaonamespace, one replica (integrated raft storage on one RWO volume),Recreatestrategy, container port 8200 with/v1/sys/healthprobes — readiness wants 200 (initialised, unsealed, active), liveness accepts sealed and uninitialised - Image:
quay.io/openbao/openbao, a release version pinned by digest; the image entrypoint turnsserverintobao server -config=/openbao/config, where the configuration is mounted from a hash-suffixed ConfigMap - Storage: raft on a 2Gi
longhorn-encryptedPersistentVolumeClaim at/openbao/data;disable_mlockas the raft backend requires - Seal:
seal "static"with a 32-byte key from a SOPS-encrypted Secret in the environment — the server unseals itself on every start, there are no Shamir shares and no recovery keys - Initialisation:
initializeblocks in the configuration run once on the first start: thekvv2 andtransitmounts, ACL policies, the kubernetes and oidc auth methods with their roles. The transient root token is revoked afterwards. The stdout audit device is anauditstanza of the same file, applied on every start (the API refuses to enable audit devices) - OIDC configuration: a CronJob writes
auth/oidc/configevery 15 minutes from the workspace outputs, because that write needs Keycloak up and self-initialisation runs only once. The server starts with nothing but its seal key and picks up login, and any re-minted client secret, on a later run - Networking: ClusterIP service on 8200 with metrics annotations, HTTPRoute on the internal gateway at
openbao.int.kueber.eu; TLS ends at the gateway, the listener trustsX-Forwarded-Forfrom the pod network. Login is the server’s own oidc auth method against Keycloak, no gateway policy in front - External Secrets: a
ClusterSecretStorenamedopenbao(Vault provider, kv v2 atkv/, kubernetes auth with a dedicated ServiceAccount and audienceopenbao) - Resources: 50m CPU and 256Mi requested, 256Mi limit,
tier-1
Security
- Runs as uid 100 (the image’s
openbaouser) in gid 1000,runAsNonRoot: true, read-only root filesystem with anemptyDirat/tmp, all capabilities dropped,RuntimeDefaultseccomp. The service-account token is mounted on purpose: the kubernetes auth method reviews client tokens with it (system:auth-delegator) - Login through Keycloak (oidc auth method, authorization code flow): only users carrying the
openbao.userrole (mapped to thedevelopersgroup) pass the client’s flow, and every one of them gets theadminpolicy — the Keycloak gate is the allow-list - Six machine identities through the kubernetes auth method, each bound to one ServiceAccount:
external-secrets(readkv/data/*, listkv/metadata/*, 1h tokens),backup(the raft snapshot endpoint, 15m tokens) andoidc-config(updateauth/oidc/config, 5m tokens) in theopenbaonamespace,gitea-runner-ociingitea-runner(readkv/data/homelab/apps/zot/ci, read the transit keyflux-artifactsand sign with it, 5m tokens), which CI jobs use to publish and sign artifacts for zot,gitea-runner-imagesingitea-runner(read the transit keycontainer-imagesand sign with it, 5m tokens), which image builds sign every first-party container image with, andprefect-flow-runinprefect(readkv/data/anthropic/console/prefectandkv/data/telegram/bots/prefectbot, 5m tokens), which flow runs read their Claude API key and Telegram bot credentials with - Break-glass: the
operatorrole of the same auth method gives theadminpolicy to a token minted for the ServiceAccountopenbao-operator, so a restore does not depend on Keycloak. No pod runs as that account. Minting a token takes cluster admin, who can read the seal key anyway, and the External Secrets controller’s ClusterRole allows it too - Ingress only from the internal gateway’s Envoy pods, the External Secrets controller, CI jobs on the runner’s
oci-publishandimage-publishlabels and Prefect flow-run pods; egress of its own only to the API server (TokenReview), the Keycloak discovery and token exchange ride the app-network-policy baseline’s gateway allowance - Every request is audited to stdout (secrets HMACed), collected with the cluster logs; a server whose only audit device fails stops serving, and stdout cannot fail
- The raft volume is Longhorn-encrypted with a key in the namespace; the seal key, the restic password and the workspace encryption key are SOPS-encrypted, the Keycloak client secret arrives in
workspace-outputs - Kyverno verifies the image’s keyless cosign signature against the
openbao/openbaorelease-images workflow on a version tag (Audit while the first quay.io policy proves itself)
Updates
Renovate bumps the version and moves the digest of all three image references (server, pre-backup pod and OIDC config job) in one PR, auto-merged overnight. OpenBao keeps the storage format backwards compatible within a major; the raft volume carries the data across the rollout, the static seal reopens it.
Data Management
- Storage: raft on the encrypted PVC — the only state
- Backups: a k8up
PreBackupPodlogs in asopenbao-backupand streams/v1/sys/storage/raft/snapshotto restic as one.snapfile per run; the PVC itself is not restic-annotated because a live copy of the bbolt file is not consistent. Schedules to Hetzner and Backblaze, bucket suffix-openbao-4c1f0e27 - Restore:
bao operator raft snapshot restore -forcewith the snapshot, the same seal key and a break-glass token; the per-cluster README has the commands - Secrets written to
kv/are versioned by the mount (kv v2); deleting a version is soft until destroyed - The transit keys
flux-artifactsandcontainer-imagesexist only in the raft data and its snapshots: they are not exportable, and self-initialisation does not create them, since a fresh server would make different ones. Without a snapshot, they are replaced as described under zot → Signing Key and Gitea Runner → Administration
User Management
Keycloak decides who may log in (client openbao, realm kueber.eu, created by the app’s workspace with spec.oidc, gated on the developers group). The oidc role admin maps a login to the admin policy with 1h tokens (12h maximum); there are no local users and no root token. Machine clients authenticate as Kubernetes ServiceAccounts, and so does an operator when Keycloak is not available, through the break-glass role.
Configuration Management
One HCL file, config/openbao.hcl: listener, storage, seal, telemetry, the audit device and the initialize blocks. The ConfigMap is hash-suffixed, so an edit restarts the pod — but initialize ran once; later changes to mounts, policies, roles or auth config are made as an operator with bao (or the UI) and are then part of the raft data, not of Git. The seal key enters through the environment (BAO_UNSEAL_KEY) and is referenced from the configuration with env://. The Keycloak client id and secret stay out of the server’s pod: they are mounted only in the CronJob that writes auth/oidc/config.
Usage
Open openbao.int.kueber.eu from the LAN or NetBird, choose OIDC and sign in with Keycloak; from a workstation BAO_ADDR=https://openbao.int.kueber.eu bao login -method=oidc opens the same flow and leaves a token for the CLI. Secrets live under kv/: bao kv put kv/<app>/<name> key=value, bao kv get kv/<app>/<name>. An application reads them through External Secrets:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: example
spec:
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: example
dataFrom:
- extract:
key: example/app # kv/data/example/appThe controller refreshes on its interval, so a value rotated in OpenBao reaches the Secret without a deploy. bao kv metadata get shows the versions, bao kv rollback brings one back.
Cluster-specific deviations from the above live in the per-cluster README — see k8s/apps/openbao/talos/README.md.
Cluster Deployment
App URLs
OpenBao — Talos cluster
Deviations from defaults
- Internal gateway only (
openbao.int.kueber.eu) — a secret store has no business on the internet; LAN and NetBird reach the internal VIP, the Crossplane workspace writes theint.kueber.eurecord. Every in-cluster caller (External Secrets, the pre-backup pod) uses the ClusterIP, plain HTTP, inside the namespace network policy. - Static-seal auto-unseal, no Shamir shares —
config/openbao.hclcarriesseal "static"with the key fromsecret-openbao-unseal.yaml(BAO_UNSEAL_KEY, 32 random bytes base64, SOPS-encrypted; plaintext only in the session scratchpad that minted it). Whoever holds the age key and the repository can unseal, which is the trust boundary every other secret here already has. The seal key also encrypts raft snapshots, so a restore needs this Secret as it was at snapshot time. Rotation = new value and newcurrent_key_id, old pair moved toprevious_*, one restart. - Declarative self-initialisation, no root token — the
initializeblocks inconfig/openbao.hclrun once against empty storage: thekv(v2) andtransitmounts, theadmin/external-secrets/backup/oidc-config/gitea-runner-oci/gitea-runner-images/prefect-flow-runpolicies, the kubernetes auth method with its seven roles, and the oidc auth method with itsadminrole. The transient root token is revoked at the end; nobody ever holds one. A failed block makes the server exit and refuse to start until the volume is wiped, so nothing in those blocks may depend on anything outside the pod, and a change to them is tested on a fresh PVC, not on the live one. Changing the config later restarts the pod (hash-suffixed ConfigMap) but does not re-runinitialize: later setup is an operator’sbaosession. The stdout audit device is anauditstanza in the same file, not a request: 2.7 refuses to enable audit devices through the API, and a config-declared device is re-applied on every start. - Login through Keycloak, in the app — the oidc auth method against the client the TofuWorkspace mints (
workspace/patch-oidc.yaml, gated onopenbao.userfor thedevelopersgroup). Anyone who passes Keycloak’s gate lands on theadminpolicy — the gate is the allow-list, there is no second tier. UI callback/ui/vault/auth/oidc/oidc/callback, CLI callbackhttp://localhost:8250/oidc/callback. No gatewaySecurityPolicy: the CLI and the API need the route without a cookie session in front. auth/oidc/configbelongs to a CronJob, not to the self-init — writing it makes the server contact Keycloak’s discovery URL, and the self-init runs once and stops the server on a failure, so the server would otherwise depend on Keycloak and Keycloak’s workspace to come up at all.cronjob-oidc-config.yamlwrites it every 15 minutes fromworkspace-outputsinstead: the server starts with nothing but its seal key, login appears on the first run after Keycloak and the workspace are ready, and a re-minted client secret is picked up without anyone touching it. The job logs in asopenbao-oidc-config, whose policy can update that one path and nothing else, and it writes on every run because the server never returns the stored secret to compare against. Noworkspace-outputsyet is a clean exit. A failed write fails the Job and leaves the stored config alone;ttlSecondsAfterFinishedthen ages the Job out, becauseKubeJobFailedfires for as long as a failed Job exists. Its pods carry their ownapp.kubernetes.io/name, since the Service and the network policies select onopenbao.- Break-glass admin through the kubernetes auth method — role
operatormaps the ServiceAccountopenbao-operator(audienceopenbao) to theadminpolicy for an hour, so a restore or a broken login does not depend on Keycloak. No pod runs as the account, and minting a token for it takes cluster admin, who can read the seal key anyway. The External Secrets controller’s ClusterRole can also mint tokens for any ServiceAccount, so a compromised controller reachesadminhere rather than stopping atkvreads. The commands are under Restore. - CI reads the registry password and signs here — role
gitea-runner-ocimaps the ServiceAccount of that name ingitea-runner(audienceopenbao) to a policy that readskv/homelab/apps/zot/ci, reads the transit keyflux-artifactsand signs with it, for 5 minutes, and the network policy admits only jobs on the runner’soci-publishlabel. CI logs in to zot with theusernameandpasswordthere. Each zot login is a secret of its own underkv/homelab/apps/zot/, because a kv policy cannot narrow to one key: CI cannot readkv/homelab/apps/zot/johnnynext to it. The signing key (ecdsa-p256, not exportable, no plaintext backup, not deletable) was created by an operator, not byinitialize, which would make a new key on every fresh server; it lives in the raft data and its snapshots only. Losing both means a new key, see zot → Signing Key. - Image builds sign here too — role
gitea-runner-imagesmaps the ServiceAccount of that name ingitea-runner(audienceopenbao) to a policy that reads the transit keycontainer-imagesand signs with it, for 5 minutes, and nothing else; the network policy admits jobs on the runner’simage-publishlabel. Every first-party image build signs its pushed digest this way, and Kyverno’sverify-gitea-johnny-imageschecks that signature. The key was created by an operator for the same reason asflux-artifactsand lives only in the raft data and its snapshots; replacing it is described under Gitea Runner → Administration. - Prefect flow runs read their credentials here — role
prefect-flow-runmaps the ServiceAccount of that name inprefect(audienceopenbao) to a policy that readskv/anthropic/console/prefect(Claude API key) andkv/telegram/bots/prefectbot(Telegram bot) for 5 minutes. The flow logs in itself, without an ExternalSecret, because the values are only needed inside the run; the work pool template projects the token into every flow-run pod, and the network policy admits flow-run pods only. - Cannot ship its own fix through CI —
build-oci-appsreads its registry password and signs here, so with this server down nothing can be published, a fix to this server included. What is already published still comes through, because Flux pulls with a SOPS secret. To deliver a fix meanwhile, pointk8s/clusters/talos/openbao.yamlat Git the waygitea-runner.yamldoes (sourceReftheGitRepositoryflux-system,path: ./k8s/apps/openbao/talos), then move it back. - External Secrets store
openbao—cluster-secret-store.yamlis the secondClusterSecretStorenext tosecrets-hub. It logs in through the kubernetes auth method as theexternal-secretsServiceAccount of this namespace (tokens minted by the operator with audienceopenbao, verified in the TokenReview); theexternal-secretspolicy readskv/data/*and listskv/metadata/*. The server reviews tokens with its own projected token, henceopenbao-auth-delegator(rbac.yaml) and the API-server egress rule. - Backups are raft snapshots, not the volume —
backups/pre-backup-openbao.yamllogs in asopenbao-backup(rolebackup, policy = the snapshot endpoint) and streams/v1/sys/storage/raft/snapshotto restic as one.snapfile; the PVC carries no restic annotation because a live copy of the bbolt file is not a consistent backup. Backup jobs run as uid 1000 (nothing on disk to own). Restic bucket suffix-openbao-4c1f0e27; window 03:03 Hetzner / 05:06 Backblaze — the first slots on the offset grid, the 6-minute grid of the first 40 apps being full. - Image signature verified by Kyverno, in Audit —
k8s/platform/configs/talos/kyverno/cosign/openbao.yamlchecks the keyless signature ofquay.io/openbao/openbao(identityrelease-images.yml@refs/tags/v*). First quay.io image under a cosign policy here, so it starts in Audit; flip to Deny once the policy report shows the bundle lookup passing. - Metrics open on the pod, closed at the edge —
unauthenticated_metrics_accesson the listener andprometheus.io/path: /v1/sys/metrics?format=prometheuson the Service; the only peers that reach port 8200 are the internal gateway, External Secrets, vmagent,oci-publishandimage-publishCI jobs, Prefect flow-run pods and this namespace’s own pods (the pre-backup pod and the OIDC config job). - tier-1, 256Mi request = limit — every ExternalSecret backed by this store stops refreshing while it is down. 2Gi raft volume on
longhorn-encrypted(secret-openbao-data-longhorn.yaml), expandable.
Bootstrap
- Watch the first boot:
kubectl -n openbao logs deploy/openbaomust show the self-initialisation requests succeeding andcore: post-unseal setup complete;bao statusin the pod reportsInitialized true,Sealed false, seal typestatic. - Wait for the first
openbao-oidc-configrun after the workspace has writtenworkspace-outputs, or skip the wait withkubectl -n openbao create job --from=cronjob/openbao-oidc-config oidc-now. Then openhttps://openbao.int.kueber.eu, method OIDC, role left empty (adminis the default), Sign in with OIDC provider — Keycloak decides. From a workstation:BAO_ADDR=https://openbao.int.kueber.eu bao login -method=oidc. - Put the first secret under
kv/and reference it from anExternalSecretwithsecretStoreRef: {kind: ClusterSecretStore, name: openbao}; the store’sReadycondition confirms the kubernetes-auth login. - Trigger a backup once (
kubectl -n openbao create -faBackupreferencingschedule-hetzner’s backend, or wait for 03:03) and check thatkubectl -n openbao get snapshots.k8up.iolists/openbao-openbao-snapshot.snap(done 2026-11-03: snapshot 71960d45, 29 KB).
Restore
restic dump latest /openbao-openbao-snapshot.snap > openbao.snap # k8up names the stream <namespace>-<container>; hetzner or backblaze repository
kubectl -n openbao cp openbao.snap "$(kubectl -n openbao get pod -l app.kubernetes.io/name=openbao -o name | cut -d/ -f2):/tmp/openbao.snap"
JWT="$(kubectl -n openbao create token openbao-operator --audience openbao --duration 15m)"
TOKEN="$(kubectl -n openbao exec deploy/openbao -- bao write -field=token auth/kubernetes/login role=operator jwt="$JWT")"
kubectl -n openbao exec deploy/openbao -- env BAO_TOKEN="$TOKEN" bao operator raft snapshot restore -force /tmp/openbao.snap-force because the running server was initialised separately; the snapshot is readable only with the same openbao-unseal key. Keycloak is not involved: a fresh server has the operator role from its self-init, and everything else, the oidc config included, comes back with the snapshot. The token does not survive the restore, so log in again for any further work.
Kubernetes Metadata
- Image:
quay.io/openbao/openbao:2.7.2@sha256:5046911075e137e154ce49ab051502f4ec7cf367d484ae4a4dce665d9b90561f
Rendered manifests (kustomize build)
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kustomize.toolkit.fluxcd.io/force: enabled
labels:
app.kubernetes.io/name: openbao
name: openbao
namespace: openbao
spec:
replicas: 1
revisionHistoryLimit: 2
selector:
matchLabels:
app.kubernetes.io/name: openbao
strategy:
rollingUpdate:
type: Recreate
template:
metadata:
labels:
app.kubernetes.io/name: openbao
spec:
automountServiceAccountToken: true
containers:
- args:
- server
env:
- name: BAO_UNSEAL_KEY
valueFrom:
secretKeyRef:
key: key
name: openbao-unseal
- name: SKIP_CHOWN
value: '1'
- name: BAO_ADDR
value: http://127.0.0.1:8200
image: quay.io/openbao/openbao:2.7.2@sha256:5046911075e137e154ce49ab051502f4ec7cf367d484ae4a4dce665d9b90561f
livenessProbe:
failureThreshold: 5
httpGet:
path: /v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204
port: web
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 3
name: openbao
ports:
- containerPort: 8200
name: web
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /v1/sys/health?standbyok=true
port: web
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
resources:
limits:
memory: 256Mi
requests:
cpu: 50m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
volumeMounts:
- mountPath: /openbao/config
name: config
readOnly: true
- mountPath: /openbao/data
name: data
- mountPath: /tmp
name: tmp
priorityClassName: tier-1
securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 100
seccompProfile:
type: RuntimeDefault
serviceAccountName: openbao
volumes:
- configMap:
name: openbao-config-gmbtffd5h2
name: config
- name: data
persistentVolumeClaim:
claimName: openbao-data
- emptyDir: {}
name: tmp