Skip to content
OpenBao

OpenBao

A secret store that applications read from at runtime — instead of a secret that lives in a manifest.

OpenBao is the Linux Foundation’s fork of HashiCorp Vault, kept under the MPL after Vault moved to the BSL: the same API, CLI, storage format and plugin surface, with its own additions (static-seal auto-unseal, declarative self-initialisation, OCI plugin distribution). It stores key/value secrets encrypted at rest, hands them out to authenticated clients under ACL policies, and writes an audit line for every request. Here it is the backend of a second External Secrets store, so an app can read a credential that was created or rotated in OpenBao without the value ever passing through Git. Its transit engine also holds the key CI signs the app artifacts with: the key signs inside OpenBao and never leaves it.

Alternatives considered

Self Hosted

ToolOpen SourceFull FeaturesNotes
HashiCorp VaultBSLYesThe original; the licence is the reason the fork exists, otherwise identical for this use
InfisicalYesPartlyDeveloper-facing UI and sync agents; needs Postgres and Redis, no raft, no plugin surface
Bitwarden Secrets ManagerPartlyPartlyWould ride on Vaultwarden’s data model, but Vaultwarden does not implement it
SOPS aloneYesPartlyWhat every secret here uses today: perfect for values that change with a commit, blind to values that change at runtime

SOPS stays the source for everything that is configuration. OpenBao covers what SOPS cannot: a credential generated or rotated by a running system, read by another one, with an audit trail and a policy in between.

Installation

Architecture

  • Deployment: Single openbao Deployment in the openbao namespace, one replica (integrated raft storage on one RWO volume), Recreate strategy, container port 8200 with /v1/sys/health probes — readiness wants 200 (initialised, unsealed, active), liveness accepts sealed and uninitialised
  • Image: quay.io/openbao/openbao, a release version pinned by digest; the image entrypoint turns server into bao server -config=/openbao/config, where the configuration is mounted from a hash-suffixed ConfigMap
  • Storage: raft on a 2Gi longhorn-encrypted PersistentVolumeClaim at /openbao/data; disable_mlock as the raft backend requires
  • Seal: seal "static" with a 32-byte key from a SOPS-encrypted Secret in the environment — the server unseals itself on every start, there are no Shamir shares and no recovery keys
  • Initialisation: initialize blocks in the configuration run once on the first start: the kv v2 and transit mounts, ACL policies, the kubernetes and oidc auth methods with their roles. The transient root token is revoked afterwards. The stdout audit device is an audit stanza of the same file, applied on every start (the API refuses to enable audit devices)
  • OIDC configuration: a CronJob writes auth/oidc/config every 15 minutes from the workspace outputs, because that write needs Keycloak up and self-initialisation runs only once. The server starts with nothing but its seal key and picks up login, and any re-minted client secret, on a later run
  • Networking: ClusterIP service on 8200 with metrics annotations, HTTPRoute on the internal gateway at openbao.int.kueber.eu; TLS ends at the gateway, the listener trusts X-Forwarded-For from the pod network. Login is the server’s own oidc auth method against Keycloak, no gateway policy in front
  • External Secrets: a ClusterSecretStore named openbao (Vault provider, kv v2 at kv/, kubernetes auth with a dedicated ServiceAccount and audience openbao)
  • Resources: 50m CPU and 256Mi requested, 256Mi limit, tier-1

Security

  • Runs as uid 100 (the image’s openbao user) in gid 1000, runAsNonRoot: true, read-only root filesystem with an emptyDir at /tmp, all capabilities dropped, RuntimeDefault seccomp. The service-account token is mounted on purpose: the kubernetes auth method reviews client tokens with it (system:auth-delegator)
  • Login through Keycloak (oidc auth method, authorization code flow): only users carrying the openbao.user role (mapped to the developers group) pass the client’s flow, and every one of them gets the admin policy — the Keycloak gate is the allow-list
  • Six machine identities through the kubernetes auth method, each bound to one ServiceAccount: external-secrets (read kv/data/*, list kv/metadata/*, 1h tokens), backup (the raft snapshot endpoint, 15m tokens) and oidc-config (update auth/oidc/config, 5m tokens) in the openbao namespace, gitea-runner-oci in gitea-runner (read kv/data/homelab/apps/zot/ci, read the transit key flux-artifacts and sign with it, 5m tokens), which CI jobs use to publish and sign artifacts for zot, gitea-runner-images in gitea-runner (read the transit key container-images and sign with it, 5m tokens), which image builds sign every first-party container image with, and prefect-flow-run in prefect (read kv/data/anthropic/console/prefect and kv/data/telegram/bots/prefectbot, 5m tokens), which flow runs read their Claude API key and Telegram bot credentials with
  • Break-glass: the operator role of the same auth method gives the admin policy to a token minted for the ServiceAccount openbao-operator, so a restore does not depend on Keycloak. No pod runs as that account. Minting a token takes cluster admin, who can read the seal key anyway, and the External Secrets controller’s ClusterRole allows it too
  • Ingress only from the internal gateway’s Envoy pods, the External Secrets controller, CI jobs on the runner’s oci-publish and image-publish labels and Prefect flow-run pods; egress of its own only to the API server (TokenReview), the Keycloak discovery and token exchange ride the app-network-policy baseline’s gateway allowance
  • Every request is audited to stdout (secrets HMACed), collected with the cluster logs; a server whose only audit device fails stops serving, and stdout cannot fail
  • The raft volume is Longhorn-encrypted with a key in the namespace; the seal key, the restic password and the workspace encryption key are SOPS-encrypted, the Keycloak client secret arrives in workspace-outputs
  • Kyverno verifies the image’s keyless cosign signature against the openbao/openbao release-images workflow on a version tag (Audit while the first quay.io policy proves itself)

Updates

Renovate bumps the version and moves the digest of all three image references (server, pre-backup pod and OIDC config job) in one PR, auto-merged overnight. OpenBao keeps the storage format backwards compatible within a major; the raft volume carries the data across the rollout, the static seal reopens it.

Data Management

  • Storage: raft on the encrypted PVC — the only state
  • Backups: a k8up PreBackupPod logs in as openbao-backup and streams /v1/sys/storage/raft/snapshot to restic as one .snap file per run; the PVC itself is not restic-annotated because a live copy of the bbolt file is not consistent. Schedules to Hetzner and Backblaze, bucket suffix -openbao-4c1f0e27
  • Restore: bao operator raft snapshot restore -force with the snapshot, the same seal key and a break-glass token; the per-cluster README has the commands
  • Secrets written to kv/ are versioned by the mount (kv v2); deleting a version is soft until destroyed
  • The transit keys flux-artifacts and container-images exist only in the raft data and its snapshots: they are not exportable, and self-initialisation does not create them, since a fresh server would make different ones. Without a snapshot, they are replaced as described under zot → Signing Key and Gitea Runner → Administration

User Management

Keycloak decides who may log in (client openbao, realm kueber.eu, created by the app’s workspace with spec.oidc, gated on the developers group). The oidc role admin maps a login to the admin policy with 1h tokens (12h maximum); there are no local users and no root token. Machine clients authenticate as Kubernetes ServiceAccounts, and so does an operator when Keycloak is not available, through the break-glass role.

Configuration Management

One HCL file, config/openbao.hcl: listener, storage, seal, telemetry, the audit device and the initialize blocks. The ConfigMap is hash-suffixed, so an edit restarts the pod — but initialize ran once; later changes to mounts, policies, roles or auth config are made as an operator with bao (or the UI) and are then part of the raft data, not of Git. The seal key enters through the environment (BAO_UNSEAL_KEY) and is referenced from the configuration with env://. The Keycloak client id and secret stay out of the server’s pod: they are mounted only in the CronJob that writes auth/oidc/config.

Usage

Open openbao.int.kueber.eu from the LAN or NetBird, choose OIDC and sign in with Keycloak; from a workstation BAO_ADDR=https://openbao.int.kueber.eu bao login -method=oidc opens the same flow and leaves a token for the CLI. Secrets live under kv/: bao kv put kv/<app>/<name> key=value, bao kv get kv/<app>/<name>. An application reads them through External Secrets:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: example
spec:
  secretStoreRef:
    kind: ClusterSecretStore
    name: openbao
  target:
    name: example
  dataFrom:
    - extract:
        key: example/app   # kv/data/example/app

The controller refreshes on its interval, so a value rotated in OpenBao reaches the Secret without a deploy. bao kv metadata get shows the versions, bao kv rollback brings one back.

Cluster-specific deviations from the above live in the per-cluster README — see k8s/apps/openbao/talos/README.md.

Cluster Deployment

OpenBao — Talos cluster

Deviations from defaults

  • Internal gateway only (openbao.int.kueber.eu) — a secret store has no business on the internet; LAN and NetBird reach the internal VIP, the Crossplane workspace writes the int.kueber.eu record. Every in-cluster caller (External Secrets, the pre-backup pod) uses the ClusterIP, plain HTTP, inside the namespace network policy.
  • Static-seal auto-unseal, no Shamir shares — config/openbao.hcl carries seal "static" with the key from secret-openbao-unseal.yaml (BAO_UNSEAL_KEY, 32 random bytes base64, SOPS-encrypted; plaintext only in the session scratchpad that minted it). Whoever holds the age key and the repository can unseal, which is the trust boundary every other secret here already has. The seal key also encrypts raft snapshots, so a restore needs this Secret as it was at snapshot time. Rotation = new value and new current_key_id, old pair moved to previous_*, one restart.
  • Declarative self-initialisation, no root token — the initialize blocks in config/openbao.hcl run once against empty storage: the kv (v2) and transit mounts, the admin / external-secrets / backup / oidc-config / gitea-runner-oci / gitea-runner-images / prefect-flow-run policies, the kubernetes auth method with its seven roles, and the oidc auth method with its admin role. The transient root token is revoked at the end; nobody ever holds one. A failed block makes the server exit and refuse to start until the volume is wiped, so nothing in those blocks may depend on anything outside the pod, and a change to them is tested on a fresh PVC, not on the live one. Changing the config later restarts the pod (hash-suffixed ConfigMap) but does not re-run initialize: later setup is an operator’s bao session. The stdout audit device is an audit stanza in the same file, not a request: 2.7 refuses to enable audit devices through the API, and a config-declared device is re-applied on every start.
  • Login through Keycloak, in the app — the oidc auth method against the client the TofuWorkspace mints (workspace/patch-oidc.yaml, gated on openbao.user for the developers group). Anyone who passes Keycloak’s gate lands on the admin policy — the gate is the allow-list, there is no second tier. UI callback /ui/vault/auth/oidc/oidc/callback, CLI callback http://localhost:8250/oidc/callback. No gateway SecurityPolicy: the CLI and the API need the route without a cookie session in front.
  • auth/oidc/config belongs to a CronJob, not to the self-init — writing it makes the server contact Keycloak’s discovery URL, and the self-init runs once and stops the server on a failure, so the server would otherwise depend on Keycloak and Keycloak’s workspace to come up at all. cronjob-oidc-config.yaml writes it every 15 minutes from workspace-outputs instead: the server starts with nothing but its seal key, login appears on the first run after Keycloak and the workspace are ready, and a re-minted client secret is picked up without anyone touching it. The job logs in as openbao-oidc-config, whose policy can update that one path and nothing else, and it writes on every run because the server never returns the stored secret to compare against. No workspace-outputs yet is a clean exit. A failed write fails the Job and leaves the stored config alone; ttlSecondsAfterFinished then ages the Job out, because KubeJobFailed fires for as long as a failed Job exists. Its pods carry their own app.kubernetes.io/name, since the Service and the network policies select on openbao.
  • Break-glass admin through the kubernetes auth method — role operator maps the ServiceAccount openbao-operator (audience openbao) to the admin policy for an hour, so a restore or a broken login does not depend on Keycloak. No pod runs as the account, and minting a token for it takes cluster admin, who can read the seal key anyway. The External Secrets controller’s ClusterRole can also mint tokens for any ServiceAccount, so a compromised controller reaches admin here rather than stopping at kv reads. The commands are under Restore.
  • CI reads the registry password and signs here — role gitea-runner-oci maps the ServiceAccount of that name in gitea-runner (audience openbao) to a policy that reads kv/homelab/apps/zot/ci, reads the transit key flux-artifacts and signs with it, for 5 minutes, and the network policy admits only jobs on the runner’s oci-publish label. CI logs in to zot with the username and password there. Each zot login is a secret of its own under kv/homelab/apps/zot/, because a kv policy cannot narrow to one key: CI cannot read kv/homelab/apps/zot/johnny next to it. The signing key (ecdsa-p256, not exportable, no plaintext backup, not deletable) was created by an operator, not by initialize, which would make a new key on every fresh server; it lives in the raft data and its snapshots only. Losing both means a new key, see zot → Signing Key.
  • Image builds sign here too — role gitea-runner-images maps the ServiceAccount of that name in gitea-runner (audience openbao) to a policy that reads the transit key container-images and signs with it, for 5 minutes, and nothing else; the network policy admits jobs on the runner’s image-publish label. Every first-party image build signs its pushed digest this way, and Kyverno’s verify-gitea-johnny-images checks that signature. The key was created by an operator for the same reason as flux-artifacts and lives only in the raft data and its snapshots; replacing it is described under Gitea Runner → Administration.
  • Prefect flow runs read their credentials here — role prefect-flow-run maps the ServiceAccount of that name in prefect (audience openbao) to a policy that reads kv/anthropic/console/prefect (Claude API key) and kv/telegram/bots/prefectbot (Telegram bot) for 5 minutes. The flow logs in itself, without an ExternalSecret, because the values are only needed inside the run; the work pool template projects the token into every flow-run pod, and the network policy admits flow-run pods only.
  • Cannot ship its own fix through CI — build-oci-apps reads its registry password and signs here, so with this server down nothing can be published, a fix to this server included. What is already published still comes through, because Flux pulls with a SOPS secret. To deliver a fix meanwhile, point k8s/clusters/talos/openbao.yaml at Git the way gitea-runner.yaml does (sourceRef the GitRepository flux-system, path: ./k8s/apps/openbao/talos), then move it back.
  • External Secrets store openbao — cluster-secret-store.yaml is the second ClusterSecretStore next to secrets-hub. It logs in through the kubernetes auth method as the external-secrets ServiceAccount of this namespace (tokens minted by the operator with audience openbao, verified in the TokenReview); the external-secrets policy reads kv/data/* and lists kv/metadata/*. The server reviews tokens with its own projected token, hence openbao-auth-delegator (rbac.yaml) and the API-server egress rule.
  • Backups are raft snapshots, not the volume — backups/pre-backup-openbao.yaml logs in as openbao-backup (role backup, policy = the snapshot endpoint) and streams /v1/sys/storage/raft/snapshot to restic as one .snap file; the PVC carries no restic annotation because a live copy of the bbolt file is not a consistent backup. Backup jobs run as uid 1000 (nothing on disk to own). Restic bucket suffix -openbao-4c1f0e27; window 03:03 Hetzner / 05:06 Backblaze — the first slots on the offset grid, the 6-minute grid of the first 40 apps being full.
  • Image signature verified by Kyverno, in Audit — k8s/platform/configs/talos/kyverno/cosign/openbao.yaml checks the keyless signature of quay.io/openbao/openbao (identity release-images.yml@refs/tags/v*). First quay.io image under a cosign policy here, so it starts in Audit; flip to Deny once the policy report shows the bundle lookup passing.
  • Metrics open on the pod, closed at the edge — unauthenticated_metrics_access on the listener and prometheus.io/path: /v1/sys/metrics?format=prometheus on the Service; the only peers that reach port 8200 are the internal gateway, External Secrets, vmagent, oci-publish and image-publish CI jobs, Prefect flow-run pods and this namespace’s own pods (the pre-backup pod and the OIDC config job).
  • tier-1, 256Mi request = limit — every ExternalSecret backed by this store stops refreshing while it is down. 2Gi raft volume on longhorn-encrypted (secret-openbao-data-longhorn.yaml), expandable.

Bootstrap

  1. Watch the first boot: kubectl -n openbao logs deploy/openbao must show the self-initialisation requests succeeding and core: post-unseal setup complete; bao status in the pod reports Initialized true, Sealed false, seal type static.
  2. Wait for the first openbao-oidc-config run after the workspace has written workspace-outputs, or skip the wait with kubectl -n openbao create job --from=cronjob/openbao-oidc-config oidc-now. Then open https://openbao.int.kueber.eu, method OIDC, role left empty (admin is the default), Sign in with OIDC provider — Keycloak decides. From a workstation: BAO_ADDR=https://openbao.int.kueber.eu bao login -method=oidc.
  3. Put the first secret under kv/ and reference it from an ExternalSecret with secretStoreRef: {kind: ClusterSecretStore, name: openbao}; the store’s Ready condition confirms the kubernetes-auth login.
  4. Trigger a backup once (kubectl -n openbao create -f a Backup referencing schedule-hetzner’s backend, or wait for 03:03) and check that kubectl -n openbao get snapshots.k8up.io lists /openbao-openbao-snapshot.snap (done 2026-11-03: snapshot 71960d45, 29 KB).

Restore

restic dump latest /openbao-openbao-snapshot.snap > openbao.snap   # k8up names the stream <namespace>-<container>; hetzner or backblaze repository
kubectl -n openbao cp openbao.snap "$(kubectl -n openbao get pod -l app.kubernetes.io/name=openbao -o name | cut -d/ -f2):/tmp/openbao.snap"
JWT="$(kubectl -n openbao create token openbao-operator --audience openbao --duration 15m)"
TOKEN="$(kubectl -n openbao exec deploy/openbao -- bao write -field=token auth/kubernetes/login role=operator jwt="$JWT")"
kubectl -n openbao exec deploy/openbao -- env BAO_TOKEN="$TOKEN" bao operator raft snapshot restore -force /tmp/openbao.snap

-force because the running server was initialised separately; the snapshot is readable only with the same openbao-unseal key. Keycloak is not involved: a fresh server has the operator role from its self-init, and everything else, the oidc config included, comes back with the snapshot. The token does not survive the restore, so log in again for any further work.

Kubernetes Metadata

  • Image: quay.io/openbao/openbao:2.7.2@sha256:5046911075e137e154ce49ab051502f4ec7cf367d484ae4a4dce665d9b90561f
Rendered manifests (kustomize build)
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/force: enabled
  labels:
    app.kubernetes.io/name: openbao
  name: openbao
  namespace: openbao
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: openbao
  strategy:
    rollingUpdate:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: openbao
    spec:
      automountServiceAccountToken: true
      containers:
      - args:
        - server
        env:
        - name: BAO_UNSEAL_KEY
          valueFrom:
            secretKeyRef:
              key: key
              name: openbao-unseal
        - name: SKIP_CHOWN
          value: '1'
        - name: BAO_ADDR
          value: http://127.0.0.1:8200
        image: quay.io/openbao/openbao:2.7.2@sha256:5046911075e137e154ce49ab051502f4ec7cf367d484ae4a4dce665d9b90561f
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204
            port: web
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        name: openbao
        ports:
        - containerPort: 8200
          name: web
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /v1/sys/health?standbyok=true
            port: web
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        resources:
          limits:
            memory: 256Mi
          requests:
            cpu: 50m
            memory: 256Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: true
        volumeMounts:
        - mountPath: /openbao/config
          name: config
          readOnly: true
        - mountPath: /openbao/data
          name: data
        - mountPath: /tmp
          name: tmp
      priorityClassName: tier-1
      securityContext:
        fsGroup: 1000
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 100
        seccompProfile:
          type: RuntimeDefault
      serviceAccountName: openbao
      volumes:
      - configMap:
          name: openbao-config-gmbtffd5h2
        name: config
      - name: data
        persistentVolumeClaim:
          claimName: openbao-data
      - emptyDir: {}
        name: tmp
Last updated on