OpenBao rootless on Podman
Part 2 of the SSH series that starts with Why SSH certificates.
At the end of this post OpenBao runs on the home server as a rootless container. It has two SSH CAs (users and hosts), roles for each kind of certificate, and a login through Forgejo in the browser. Its configuration is one script in Git, and a nightly Raft snapshot goes into the backup.
Where it runs, and who can reach it
Bao runs on my home server, home1. It is reachable from the LAN and through my WireGuard tunnel, never from the internet. The home router does not forward port 443 to it.
Two DNS records make the same name work in both places:
- On the LAN, the local DNS (Pi-hole) answers
bao.example.netwith the LAN address,192.168.50.5. - The public record of
bao.example.netpoints to the WireGuard address of the home server,10.99.0.1. That address is private and only works inside the tunnel. Away from home, I start the tunnel first.
Traefik gets the certificate with the DNS challenge, so Let’s Encrypt never needs to connect to the host. TLS ends at Traefik.
Hosts do not need Bao to check a certificate. They only need the CA’s public key, which they have as a file. Bao must be reachable only to get a certificate: from my laptop, from the Ansible control node and from the CI runner. If Bao is down, logins with existing certificates keep working.
Step 1: the quadlet
# ~svc/.config/containers/systemd/openbao.container
[Unit]
Description=OpenBao
[Container]
ContainerName=openbao
# Minor version pinned, patch releases through auto-update
Image=quay.io/openbao/openbao:2.7
AutoUpdate=registry
Label=traefik.enable=true
Label=traefik.http.routers.openbao.rule=Host(`bao.example.net`)
Label=traefik.http.routers.openbao.entrypoints=https
Label=traefik.http.routers.openbao.tls.certresolver=default
Label=traefik.http.routers.openbao.middlewares=secureHeaders@file
Label=traefik.http.services.openbao.loadbalancer.server.port=8200
Exec=bao server -config=/openbao/config/openbao.hcl
# The image's user openbao (100:1000) is svc on the host: no chown needed
UserNS=keep-id:uid=100,gid=1000
Environment=SKIP_CHOWN=1
Volume=%h/.config/openbao/openbao.hcl:/openbao/config/openbao.hcl:ro,z
Volume=openbao-data.volume:/openbao/file:Z
ReadOnly=true
Tmpfs=/tmp
DropCapability=ALL
NoNewPrivileges=true
Network=proxy.network:ip=10.89.0.14
[Service]
Restart=always
[Install]
WantedBy=default.target
# openbao-data.volume
[Volume]
Some notes:
UserNS=keep-id:uid=100,gid=1000maps the image’sopenbaouser tosvcon the host. The files in the volume then belong tosvc, and the entrypoint does not need tochownanything.SKIP_CHOWN=1tells it not to try.ReadOnly=true,DropCapability=ALLandNoNewPrivileges=true: Bao needs to write only into its volume and/tmp.- Pin the minor version. I want to read the release notes before a minor upgrade of the thing that holds my CAs. Patch releases come through
podman auto-update.
Step 2: the server config
# ~svc/.config/openbao/openbao.hcl
ui = true
api_addr = "https://bao.example.net"
cluster_addr = "http://127.0.0.1:8201"
# Integrated storage: mlock is not wanted (memory-mapped database)
disable_mlock = true
storage "raft" {
path = "/openbao/file"
node_id = "home1"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true
}
# Every request, with secret values HMACed, to stdout: journald through Podman
audit "file" "stdout" {
options {
file_path = "stdout"
}
}
Integrated storage (Raft) is a single directory and needs no external database. A single node is a valid Raft cluster.
tls_disable looks scary, but the listener is reachable only on the Podman network, from Traefik. TLS ends there.
The audit device is declared in the config file, so it exists from the very first start. The log is in the journal with everything else.
Step 3: initialise
Start it, then initialise it once:
systemctl --user daemon-reload && systemctl --user start openbao
export BAO_ADDR=https://bao.example.net
bao operator init -key-shares=1 -key-threshold=1 -format=json > /dev/shm/init.json
init prints the unseal key and the initial root token. They go straight into pass, and the file is deleted:
jq -r '.unseal_keys_b64[0]' /dev/shm/init.json | pass insert -m infra/home1/openbao-unseal-key
jq -r '.root_token' /dev/shm/init.json | pass insert -m infra/home1/openbao-root-token
shred -u /dev/shm/init.json
One key share is enough for one person. Shamir splitting into several shares makes sense when several people must agree to unseal.
I use /dev/shm so that the file is in memory and never on disk.
Step 4: unseal, and why not automatically
Bao encrypts everything in its storage. After every start it is sealed: it has the data but not the key. Somebody must give it the unseal key.
I wanted auto-unseal with the home server’s TPM. The plan: encrypt a static seal key with systemd-creds --with-key=tpm2 and give it to the container with LoadCredentialEncrypted=. Bao would then unseal itself after a reboot, and a stolen disk would be useless without the board.
It did not work. The board has a TPM 1.2, and systemd-creds supports only TPM 2.0. So I unseal by hand, from my laptop, with the key from pass:
#!/bin/bash
# bao-unseal: run on the laptop, at home or through the tunnel
set -euo pipefail
addr=${BAO_ADDR:-https://bao.example.net}
sealed() {
local s
s=$(curl -fsS -m 15 "$addr/v1/sys/seal-status") || { echo "OpenBao not reachable at $addr" >&2; exit 1; }
python3 -c 'import json,sys; sys.exit(0 if json.loads(sys.argv[1])["sealed"] else 1)' "$s"
}
if ! sealed; then
echo "OpenBao is not sealed"
exit 0
fi
# the key goes to curl on stdin, never onto a command line
pass show infra/home1/openbao-unseal-key | head -1 \
| python3 -c 'import json,sys; print(json.dumps({"key": sys.stdin.read().strip()}))' \
| curl -fsS -m 15 -X PUT -d @- -o /dev/null "$addr/v1/sys/unseal"
if sealed; then
echo "still sealed" >&2
exit 1
fi
echo "OpenBao unsealed"
The key never appears in ps or in the shell history. A Prometheus blackbox probe watches /v1/sys/health, which answers 503 while Bao is sealed. An alert tells me after five minutes.
This is less work than it sounds. The home server reboots only when I reboot it. But remember that a podman auto-update of the image, or any change to the quadlet, restarts the container, and the restart seals it again.
Step 5: the configuration as a script
Everything inside Bao (mounts, roles, policies, logins) is set by one script in the repository. It is idempotent: I change the script and run it again. There is no click in the web UI that I would forget.
The script needs a token with full rights. The first run uses the root token from pass. Later runs use my userpass login, which the script creates:
BAO_TOKEN=$(pass show infra/home1/openbao-root-token) ./bao-config.sh # first run
bao login -no-print -method=userpass username=alice && ./bao-config.sh # later
Always use -no-print. Without it, bao login prints the token on the screen.
After the first run, revoke the root token. A new one can be made from the unseal key with bao operator generate-root when you need it.
The SSH CAs
set -euo pipefail
export BAO_ADDR=${BAO_ADDR:-https://bao.example.net}
mounted() { bao "$1" list -format=json | python3 -c 'import json,sys; sys.exit(0 if sys.argv[1] + "/" in json.load(sys.stdin) else 1)' "$2"; }
secrets() { mounted secrets "$2" || bao secrets enable -path="$2" "$1"; }
auth() { mounted auth "$2" || bao auth enable -path="$2" "$1"; }
# Two SSH CAs, keys generated inside Bao (ed25519)
for m in ssh-user ssh-host; do
secrets ssh "$m"
bao read "$m/config/ca" >/dev/null 2>&1 \
|| bao write "$m/config/ca" generate_signing_key=true key_type=ed25519 >/dev/null
done
generate_signing_key=true creates the key inside Bao. It never leaves it. bao read ssh-user/config/ca shows the public key, which is what the hosts need.
The || bao write part makes it idempotent. A second run would otherwise replace the CA key, and every certificate would be invalid.
The user roles
A small function writes a role. The JSON body is easier to read than a long list of key=value arguments:
# Key ID: who asked (token display name) and with which role; it shows in sshd's log
user_role() { # name principal ttl extensions [critical options as JSON]
local exts crit=${5:-'{}'}
exts=$(python3 -c 'import json,sys; print(json.dumps({e: "" for e in sys.argv[1].split(",") if e}))' "$4")
bao write "ssh-user/roles/$1" - >/dev/null <<JSON
{
"key_type": "ca",
"allow_user_certificates": true,
"allowed_users": "$2",
"default_user": "$2",
"ttl": "$3",
"max_ttl": "$3",
"allowed_extensions": "$4",
"default_extensions": $exts,
"allowed_critical_options": "$(python3 -c 'import json,sys; print(",".join(json.loads(sys.argv[1])))' "$crit")",
"default_critical_options": $crit,
"allowed_user_key_lengths": {"ed25519": [0], "sk-ed25519": [0]},
"key_id_format": "{{token_display_name}}-{{role_name}}"
}
JSON
}
user_role admin alice 16h permit-pty,permit-port-forwarding
user_role device alice 720h permit-pty,permit-port-forwarding
user_role ansible ansible 16h permit-pty
user_role deploy-websites deploy-websites 5m "" '{"force-command": "podman auto-update"}'
What each field does:
allowed_usersanddefault_user: the role signs exactly one principal. A caller cannot ask forroot.ttlequal tomax_ttl: a caller cannot ask for a longer validity.allowed_extensions: only these. No agent forwarding, no X11.allowed_user_key_lengths: only Ed25519 keys, also on a FIDO2 token (sk-ed25519).key_id_format: the key ID in the certificate says who asked, for exampleoidc-alice-admin. sshd logs it with every login. This is how I see in a host’s log which role signed the certificate that just logged in.
The host role
bao write ssh-host/roles/hosts - >/dev/null <<'JSON'
{
"key_type": "ca",
"allow_host_certificates": true,
"allowed_domains": "example.org,example.net",
"allow_subdomains": true,
"ttl": "8760h",
"max_ttl": "8760h",
"allowed_user_key_lengths": {"ed25519": [0]},
"key_id_format": "{{token_display_name}}-host"
}
JSON
Host certificates are valid for one year. The Ansible run renews them when less than 60 days are left (next post).
Policies
bao policy write admin - <<'HCL'
# Daily use (OIDC login): sign user and host certificates
path "ssh-user/sign/admin" { capabilities = ["update"] }
path "ssh-user/sign/device" { capabilities = ["update"] }
path "ssh-user/sign/ansible" { capabilities = ["update"] }
path "ssh-host/sign/hosts" { capabilities = ["update"] }
HCL
bao policy write operator - <<'HCL'
# Running Bao itself (userpass login): everything
path "*" { capabilities = ["create", "read", "update", "patch", "delete", "list", "sudo"] }
HCL
bao policy write backup - <<'HCL'
path "sys/storage/raft/snapshot" { capabilities = ["read"] }
HCL
The daily login can sign and do nothing else. It cannot read the configuration, change roles or create tokens.
Step 6: logins
OIDC through Forgejo, for every day
Forgejo is an OpenID Connect provider. In Forgejo, go to Settings, Applications, and create an OAuth2 application with two redirect URIs:
http://localhost:8250/oidc/callback
https://bao.example.net/ui/vault/auth/oidc/oidc/callback
The first one is for the CLI. bao login -method=oidc opens the browser and waits on port 8250 for the answer. The second one is for the web UI. Store the client ID and the secret in pass, then:
auth oidc oidc
python3 -c 'import json,sys; print(json.dumps({"oidc_discovery_url": "https://git.example.org", "oidc_client_id": sys.argv[1], "oidc_client_secret": sys.stdin.read().strip(), "default_role": "alice"}))' \
"$(pass show infra/home1/openbao-oidc-client-id | head -1)" < <(pass show infra/home1/openbao-oidc-client-secret | head -1) \
| bao write auth/oidc/config - >/dev/null
bao write auth/oidc/role/alice - >/dev/null <<'JSON'
{
"user_claim": "preferred_username",
"bound_claims": {"preferred_username": "alice"},
"oidc_scopes": "openid,profile",
"allowed_redirect_uris": "http://localhost:8250/oidc/callback,https://bao.example.net/ui/vault/auth/oidc/oidc/callback",
"token_policies": "admin",
"token_ttl": "16h",
"token_max_ttl": "16h"
}
JSON
bound_claims is important. Without it, every account on the Forgejo instance could log in to Bao and get the admin policy.
The secret goes to bao on stdin, inside the JSON, and never onto a command line.
userpass, for running Bao
auth userpass userpass
if ! bao read auth/userpass/users/alice >/dev/null 2>&1; then
pass show infra/home1/openbao-alice-password >/dev/null 2>&1 \
|| pass generate -n infra/home1/openbao-alice-password 32 >/dev/null
pass show infra/home1/openbao-alice-password | head -1 \
| python3 -c 'import json,sys; print(json.dumps({"password": sys.stdin.read().strip(), "token_policies": "admin,operator", "token_ttl": "1h", "token_max_ttl": "4h"}))' \
| bao write auth/userpass/users/alice - >/dev/null
fi
This login has the operator policy for changes to Bao, with a short token. It is also my way in when Forgejo is down, for example during the reinstall of the host it runs on.
AppRole, for the backup
auth approle approle
bao write auth/approle/role/backup token_policies=backup token_ttl=5m token_max_ttl=10m \
secret_id_ttl=0 token_num_uses=0 >/dev/null
if ! pass show infra/home1/openbao-backup-approle >/dev/null 2>&1; then
role_id=$(bao read -field=role_id auth/approle/role/backup/role-id)
bao write -f -format=json auth/approle/role/backup/secret-id \
| python3 -c 'import json,sys; print(json.dumps({"role_id": sys.argv[1], "secret_id": json.load(sys.stdin)["data"]["secret_id"]}))' "$role_id" \
| pass insert -m infra/home1/openbao-backup-approle >/dev/null
fi
Ansible writes this JSON to /root/.config/openbao/backup-approle.json on the home server (mode 0600). The backup script uses it to log in.
Step 7: backup and restore
Do not back up the live Raft directory. It is a database that is being written to. Take a snapshot through the API instead. That is one more step in the backup script from the monitoring and backup post:
# a backup_functions entry for home1
openbao_snapshot() {
local bao=(curl -fsS -m 60 --resolve bao.example.net:443:127.0.0.1) token
token=$("${bao[@]}" -d @/root/.config/openbao/backup-approle.json \
https://bao.example.net/v1/auth/approle/login \
| python3 -c 'import json,sys; print(json.load(sys.stdin)["auth"]["client_token"])') || return 1
"${bao[@]}" -H "X-Vault-Token: $token" -o "$DUMP_DIR/openbao/raft.snap.tmp" \
https://bao.example.net/v1/sys/storage/raft/snapshot \
&& mv "$DUMP_DIR/openbao/raft.snap.tmp" "$DUMP_DIR/openbao/raft.snap"
}
--resolve ...:127.0.0.1 sends the request to the local Traefik, with the right name for the certificate. The live volume is excluded from restic. While Bao is sealed this step fails, and the backup reports it.
The snapshot is encrypted with Bao’s keys. To restore it on any host with Podman:
- Start an empty Bao, initialise it with one share, and unseal it with the new key.
- Log in with the new root token, then run
bao operator raft snapshot restore -force raft.snap. - Bao is sealed again, now with the old key. Unseal it with the old key from pass.
- Everything is back: both CAs, the roles, the policies, the logins. Revoke the new root token.
Try this once, on a VM, before you need it.
Check
bao login -no-print -method=oidc # the browser opens, log in to Forgejo
bao token lookup -format=json | jq '.data | {display_name, policies, ttl}'
bao read -field=public_key ssh-user/config/ca
bao write -field=signed_key ssh-user/sign/admin public_key=@$HOME/.ssh/id_ed25519.pub \
| ssh-keygen -Lf /dev/stdin
journalctl --user -u openbao -n 3 -o cat | jq -c '{type, path: .request.path}'
The certificate shows principal alice, 16 hours, key ID oidc-alice-admin. The audit log shows the request.
Next: Hosts that trust the SSH CA.
Further reading
- OpenBao documentation: concepts, configuration, auth methods and secrets engines.
- OpenBao on GitHub: releases and the container images.
- Integrated storage (Raft): the storage backend, also for a single node.
- Seal and unseal: Shamir keys, auto unseal, and why the barrier exists.
- Signed SSH certificates: the SSH secrets engine, for user and host keys.
- SSH secrets engine API: every field of a role:
allowed_users,ttl,default_extensionsand more. - JWT/OIDC auth method: the browser login with Forgejo and, in post 4, the CI login.
- OIDC provider configuration: examples for many identity providers.
- Forgejo as an OAuth2 provider: the client registration on the Forgejo side.
- Userpass auth method: the fallback login.
- AppRole auth method: the login of the backup job.
- Policies: paths, capabilities and how Bao combines policies.
- File audit device: the log of every request.
- bao operator raft:
snapshot saveandsnapshot restore.