Linux Lunatix

Why SSH certificates

Stefan Marx
| 12 minutes

Part 1 of the SSH series. It builds on the hosts from the consolidation series, but you can read it on its own.

This post explains OpenSSH certificates. It has a demo that you can run on one machine in ten minutes, and it explains why I put my CA on OpenBao. The next four posts build the real thing.

The problems with plain keys

SSH with plain keys has two trust lists, and both are maintained by hand.

authorized_keys on the server says which keys may log in. With four hosts, a few accounts each, two laptops, a tablet and a CI job, that is a lot of lines. A new device means editing every host. A lost device means finding its key everywhere and removing it. A CI deploy key lives forever, unless somebody rotates it.

known_hosts on the client says which host keys to trust. The first connection asks you to confirm a fingerprint. Almost nobody checks it, which is why this is called “trust on first use”. After a reinstall, ssh shows a big warning, and you either check the new key over a console or you click it away. In the rehearsal post I showed how much work a careful check is.

Both lists work. But they grow, nobody removes old entries, and every change touches many places.

What a certificate is

An OpenSSH certificate is a public key plus some data, signed by a CA key:

A server that trusts the CA accepts any user certificate signed by it, if the certificate is still valid and lists a principal that may use the account. A client that trusts a host CA accepts any host that presents a certificate for its name.

These are not X.509 certificates. There is no chain and no CRL distribution. OpenSSH has its own simple format, and it has supported it since 2010 (OpenSSH 5.4).

What this changes

BeforeWith certificates
every host lists every user keyevery host lists one CA key and the allowed principals
a new device: edit every hosta new device: get a certificate
a lost device: remove its key everywherea lost device: its certificates expire in hours
a CI key that never expiresa certificate for five minutes, per job
known_hosts per host, TOFUone @cert-authority line on the client
reinstall: check the key over a consolereinstall: sign the new host key, done

The short validity is the most important part. Revocation is hard in any PKI. A certificate that expires after 16 hours needs no revocation in most cases.

Try it: a CA in ten minutes

You can try all of this on one machine, with ssh-keygen and a local sshd. Do it in a VM or a container if you do not want to touch your machine’s sshd.

A user CA

mkdir -p ~/ca-demo && cd ~/ca-demo
ssh-keygen -t ed25519 -f user_ca -C "demo user CA" -N ''

user_ca is the CA’s private key and user_ca.pub its public key. In real life the private key is the most sensitive file of the whole setup. Here it is a demo.

Sign a user key

ssh-keygen -t ed25519 -f alice -N ''
ssh-keygen -s user_ca -I "alice-laptop" -n alice -V +1h alice.pub
ssh-keygen -Lf alice-cert.pub
alice-cert.pub:
        Type: ssh-ed25519-cert-v01@openssh.com user certificate
        Public key: ED25519-CERT SHA256:Xq2...
        Signing CA: ED25519 SHA256:k9T... (using ssh-ed25519)
        Key ID: "alice-laptop"
        Serial: 0
        Valid: from 2026-11-01T17:10:00 to 2026-11-01T18:10:00
        Principals:
                alice
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

-I is the key ID, -n the principals and -V the validity. By default ssh-keygen adds all the extensions. A real CA should add only what you need.

ssh finds alice-cert.pub by itself when you use the key alice: it looks for <key>-cert.pub next to the key.

Trust the CA on the server

sudo install -m 0644 user_ca.pub /etc/ssh/user_ca.pub
sudo install -d /etc/ssh/principals
# which principals may log in as the account "demo"
echo alice | sudo tee /etc/ssh/principals/demo
sudo tee /etc/ssh/sshd_config.d/10-ca.conf <<'EOF'
TrustedUserCAKeys /etc/ssh/user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/principals/%u
EOF
sudo useradd -m demo
sudo sshd -t && sudo systemctl reload sshd
ssh -i ~/ca-demo/alice -o IdentitiesOnly=yes demo@localhost

You are in, and demo has no authorized_keys file. The server log shows who it was:

sshd[4242]: Accepted publickey for demo from ::1 port 50122 ssh2: ED25519-CERT SHA256:Xq2... ID alice-laptop (serial 0) CA ED25519 SHA256:k9T...

AuthorizedPrincipalsFile is the important line. Without it, sshd accepts a certificate for an account only if the account name itself is one of its principals. With it, you decide per account which principals may use it. One principal can then log in as several accounts, and a certificate for alice cannot log in as root or ansible.

A host CA

ssh-keygen -t ed25519 -f host_ca -C "demo host CA" -N ''
sudo ssh-keygen -s host_ca -I "$(hostname)" -h -n localhost,$(hostname -f) -V +52w \
    /etc/ssh/ssh_host_ed25519_key.pub
echo 'HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub' | sudo tee /etc/ssh/sshd_config.d/11-hostcert.conf
sudo systemctl reload sshd

-h makes it a host certificate. The principals are the names that clients use for the host.

On the client, one line in known_hosts replaces all the host entries:

echo "@cert-authority localhost,*.example.org $(cat host_ca.pub)" > ~/ca-demo/known_hosts
ssh -o UserKnownHostsFile=~/ca-demo/known_hosts -o StrictHostKeyChecking=yes \
    -i ~/ca-demo/alice demo@localhost

No question about a fingerprint. StrictHostKeyChecking=yes proves it: ssh would refuse a host it does not know.

Clean up

sudo rm /etc/ssh/sshd_config.d/10-ca.conf /etc/ssh/sshd_config.d/11-hostcert.conf \
        /etc/ssh/ssh_host_ed25519_key-cert.pub /etc/ssh/user_ca.pub
sudo rm -r /etc/ssh/principals
sudo systemctl reload sshd && sudo userdel -r demo

Options that the server enforces

Critical options restrict what a certificate can do:

There is a catch here. Whoever may ask the CA for a certificate with an option may also ask for another value. If the CA’s policy lets a CI job choose its own force-command, the job can choose bash. So my hosts enforce the important options themselves, in the principals file. Each line there can carry the same options as an authorized_keys line:

# /etc/ssh/principals/svc
alice
restrict,command="podman auto-update" deploy-websites

A certificate for alice logs in normally. A certificate for deploy-websites gets podman auto-update and nothing else, even if the CA signed it without the option. Post 3 of this series shows the Ansible role that writes these files.

More details worth knowing

The demo covers the basics. These details matter as soon as you run a CA for real.

Sign only the extensions you need

ssh-keygen adds all five extensions by default. -O clear removes them, and each further -O adds one back:

ssh-keygen -s user_ca -I "alice-laptop" -n alice -V +1h \
    -O clear -O permit-pty alice.pub

This certificate can open a shell, but it cannot forward ports, agents or X11. A certificate for a backup job needs no permit-pty either. The critical options work the same way:

ssh-keygen -s user_ca -I "backup-job" -n backup -V +10m \
    -O clear -O force-command="/usr/local/bin/backup.sh" \
    -O source-address=192.168.50.0/24 backup.pub

Validity and clocks

The server checks the validity against its own clock. If the client’s clock or the CA’s clock runs a little ahead, a fresh certificate can be “not yet valid” on the server. Let it start a few minutes in the past:

ssh-keygen -s user_ca -I "alice-laptop" -n alice -V -5m:+16h alice.pub

-V also takes absolute times, for example -V 20261101:20261201. Keep the clocks of all hosts in sync with NTP. Certificates depend on it.

Always set principals

-n is not optional in practice. In the certificate format, an empty principal list means “any principal”. With TrustedUserCAKeys, sshd refuses a user certificate without principals, but other programs may handle it differently. A CA that signs whatever it gets is a risk. Give every certificate its principals, and give your CA a list of principals it may sign.

Serial numbers and key IDs

-z sets a serial number:

ssh-keygen -s user_ca -I "alice-laptop" -z 1001 -n alice -V +16h alice.pub

The key ID and the serial appear in the server log for each login. They are also the handles for revocation. Choose key IDs that tell you who and what, for example alice-laptop or ci-blog-main. A CA such as OpenBao sets a random serial by itself.

Revocation, when you need it

Short certificates make revocation rare, but not impossible. The certificate of a tablet lives for weeks, and a host certificate for a year. sshd reads a revocation list from RevokedKeys. That can be a plain list of public keys, or a key revocation list (KRL) in OpenSSH’s own compact format. A KRL can revoke by serial, by key ID or by key:

cat > revoke.txt <<'EOF2'
serial: 1001
id: alice-tablet
EOF2
ssh-keygen -k -f revoked.krl -s user_ca.pub revoke.txt
# test a certificate against the KRL
ssh-keygen -Q -f revoked.krl alice-cert.pub

-u adds to an existing KRL instead of replacing it. On the server:

RevokedKeys /etc/ssh/revoked.krl

On the client, a line with @revoked in known_hosts refuses a host key, even with a valid host certificate:

@revoked * ssh-ed25519 AAAA...the-old-host-key...

Rotate the CA key

One day you will want a new CA key, because the old one leaked or because you want a new algorithm. TrustedUserCAKeys can list several keys, one per line. That makes the change smooth:

  1. Create the new CA key and add its public key to TrustedUserCAKeys on every host. Both keys are now trusted.
  2. Sign new certificates with the new key only.
  3. Wait until the last certificate from the old key has expired.
  4. Remove the old public key from every host.

On the client, two @cert-authority lines do the same for host certificates. With short certificates, step 3 takes one day.

A CA per account, without sshd_config

If you cannot change sshd_config, the CA can go into a user’s authorized_keys:

cert-authority,principals="alice" ssh-ed25519 AAAA...user-ca-public-key...

This trusts the CA only for this one account. It is useful on a shared host where you are not root. principals= has the same role as the principals file.

Host certificates and names

The client checks that the host certificate lists the name it used to connect. ssh vps1 with a certificate for vps1.example.org fails, unless your SSH config maps the short name to the full name with HostName or HostKeyAlias. Put every name that clients use into the host certificate’s principals. Post 3 shows how I handle short names.

See what happens

ssh -v shows which keys and certificates the client offers and what the server answers:

debug1: Offering public key: /home/alice/.ssh/id_ed25519 ED25519-CERT SHA256:... explicit
debug1: Server accepts key: /home/alice/.ssh/id_ed25519 ED25519-CERT SHA256:... explicit
debug1: Host 'vps1.example.org' is known and matches the ED25519-CERT host certificate.

On the server, journalctl -u sshd shows the key ID, the serial and the CA of each accepted certificate. It also shows the reason for a refusal, for example an expired certificate or a principal that is not allowed.

Where should the CA key live?

The demo kept the CA key in a file. That works for a homelab, but it has problems:

What I wanted from a CA:

Why OpenBao

I looked at four options.

OptionFor itAgainst it
ssh-keygen and a filesimple, nothing to runeverything in the list above
step-ca (Smallstep)built for this, OIDC, good SSH supportone more tool only for certificates
HashiCorp Vaultthe standard, SSH secrets engineBusiness Source License since 2023
OpenBaoVault’s open fork (MPL 2.0, Linux Foundation), same API and SSH engineyounger project

I chose OpenBao. Its SSH secrets engine does exactly what I need. It has OIDC and JWT login, policies, an audit log, and integrated storage with snapshots. It can also hold other secrets later, and step-ca cannot do that. Most Vault documentation and tools work with it unchanged: the CLI is bao instead of vault, and the API is the same. Fedora packages the openbao CLI, and the server runs from the official container image.

The design

Two CAs, one for users and one for hosts, so that a host key can never log in as a user. Each kind of certificate gets a role in Bao with its own rules:

RolePrincipalValidFor
adminalice16 hmy laptop, every day
devicealice30 daysa tablet that cannot talk to Bao
ansibleansible16 hthe Ansible key on the control node
deploy-websitesdeploy-websites5 minCI jobs; only podman auto-update
hosts (host CA)the host’s FQDN1 yearhost certificates, signed during the Ansible run

16 hours covers a working day. In the morning I run one command, log in through the browser, and have certificates for the day.

The plain keys go away, except for one break-glass key per host. That key is for the day when Bao is down and all certificates have expired.

The series

  1. This post: why and how it works.
  2. OpenBao rootless on Podman: the server, its configuration as a script, unseal, backup.
  3. Hosts that trust the SSH CA: the Ansible side, host certificates, the client.
  4. CI deploys without a deploy key: Forgejo Actions OIDC and five-minute certificates.
  5. Removing the plain SSH keys: break-glass, devices without Bao, and how not to lock yourself out.

Next: OpenBao rootless on Podman.

Further reading

Previous: Rehearse before you reinstall Next: OpenBao rootless on Podman

Related content

CI deploys without a deploy key
Removing the plain SSH keys
Hosts that trust the SSH CA