Removing the plain SSH keys
Part 5 of the SSH series that starts with Why SSH certificates.
Certificates work on every host now. The old plain keys are still in the authorized_keys files, and while they are there, the certificates change nothing about security. A stolen laptop key would still open every door. This post removes the plain keys. Only one break-glass key per account stays, for the day when Bao is gone.
The target
| Account | Certificate principals | Plain keys |
|---|---|---|
the admin account (fedora) | alice | break-glass |
svc (the apps) | alice (+ deploy-websites on the web host) | none |
ansible | ansible | break-glass |
svc gets no plain key at all. If I need it without Bao, I log in as the admin account and use sudo.
Step 1: the break-glass key
The break-glass key is a normal Ed25519 key. Its private part lives only in pass, never in ~/.ssh. Create it in memory and move it straight into pass:
d=$(mktemp -d -p /dev/shm)
ssh-keygen -q -t ed25519 -N '' -C 'break-glass@infra' -f "$d/key"
pass insert -m infra/ssh-break-glass < "$d/key"
cp "$d/key.pub" ~/ops/client/ssh/break-glass.pub
shred -u "$d/key" && rm -r "$d"
The public part goes into the inventory:
# group_vars/all.yml
ssh_break_glass_key: "ssh-ed25519 AAAA...break-glass-public-key... break-glass@infra"
To use it, a small script loads the key from pass into ssh-agent for 15 minutes. It never writes it to disk:
#!/bin/bash
# ssh-break-glass: only when Bao is gone and the certificates have expired
#
# ssh-break-glass vps1 # any ssh arguments
# ssh-break-glass -- ansible-playbook ... # a command: Ansible with the key
set -euo pipefail
pub=$(dirname "$(readlink -f "$0")")/../ssh/break-glass.pub
pass show infra/ssh-break-glass | ssh-add -q -t 15m -
if [ "${1:-}" = "--" ]; then
shift
exec "$@" --private-key "$pub"
fi
exec ssh -i "$pub" "$@"
ssh-add - reads the private key from stdin. -t 15m removes it from the agent after 15 minutes.
-i with a .pub file looks odd. ssh then selects the matching key in the agent and offers it first. That matters with IdentitiesOnly yes and MaxAuthTries 3. Ansible’s --private-key works the same way.
Test it now, while everything works, and not on the day you need it.
Step 2: authorized_keys from the inventory
Each host lists its plain keys per account:
# host_vars/vps1.yml
ssh_authorized_keys:
fedora: ["{{ ssh_break_glass_key }}"]
svc: []
ansible: ["{{ ssh_break_glass_key }}"]
The rules: a listed account gets exactly these keys, and nothing else stays in its file. An empty list removes the file. An account that is not listed is left alone, so the role cannot touch accounts it does not know.
# roles/base/tasks/ssh_keys.yml
- name: Accounts' home directories
ansible.builtin.getent:
database: passwd
- name: authorized_keys with exactly the listed keys
ansible.posix.authorized_key:
user: "{{ item.key }}"
key: "{{ item.value | join('\n') }}"
exclusive: true
loop: "{{ ssh_authorized_keys | dict2items | selectattr('value') }}"
loop_control:
label: "{{ item.key }}"
- name: No authorized_keys for cert-only accounts
ansible.builtin.file:
path: "{{ ansible_facts.getent_passwd[item.key][4] }}/.ssh/authorized_keys"
state: absent
loop: "{{ ssh_authorized_keys | dict2items | rejectattr('value') }}"
loop_control:
label: "{{ item.key }}"
exclusive: true is what removes the old keys. The home directory comes from getent, because svc and the admin account do not have to live under /home/<name>.
In the role’s main.yml these tasks come after the flush_handlers from the previous post. On a host’s first run, sshd already trusts the CA when the keys go away.
bootstrap.yml from the Ansible post still installs the control nodes’ plain keys for ansible. That is for the first contact with a new host, before it trusts the CA. The first site.yml run then replaces them with the break-glass key.
Step 3: roll it out, one host at a time
I did this first on a throwaway VM, then host by host.
- Get fresh certificates:
ssh-cert. Check that both are valid:ssh-keygen -Lf ~/.ssh/id_ed25519-cert.puband the same foransible_ed25519-cert.pub. - Open a standby session as the admin account and leave it open:
ssh -M -S ~/.ssh/cm/standby-vps1 -fN fedora@vps1.example.org - Close Ansible’s reused connections, then run the base role:
for s in ~/.ansible/cp/*; do ssh -O exit -o ControlPath="$s" x 2>/dev/null; done ansible-playbook playbooks/site.yml -l vps1 -t base --check ansible-playbook playbooks/site.yml -l vps1 -t base - Test with fresh connections. Each of these must work:And this must fail:
F="-o ControlMaster=no -o ControlPath=none" ssh $F fedora@vps1.example.org true # certificate ssh $F svc@vps1.example.org true # certificate ansible vps1 -m ping # the ansible certificate ssh-break-glass $F fedora@vps1.example.org true # break-glass keyThe copy has nod=$(mktemp -d) && cp ~/.ssh/id_ed25519 "$d/" ssh $F -o IdentityAgent=none -o IdentitiesOnly=yes -i "$d/id_ed25519" fedora@vps1.example.org true rm -r "$d"-cert.pubnext to it, andIdentityAgent=nonekeeps the agent out, so ssh offers only the plain key. Before this step it was inauthorized_keys. Now it is refused. - The second run must report
changed=0. Then close the standby session.
Devices that cannot talk to Bao
My tablet runs Blink, an SSH client for iOS. Blink cannot log in to Bao, but it can use a certificate. So the tablet gets a certificate for 30 days, which I sign on the laptop.
- In Blink, create an Ed25519 key and copy its public part to the laptop.
- Sign it with the
devicerole:ssh-cert device tablet.pub - In Blink: Keys, the key, Certificate, paste.
- Put a reminder into the calendar a few days before the 30 days end. An expired certificate is simply refused.
Blink cannot check host certificates. It asks about each host key once. Compare the fingerprint with the host’s real key before you accept it:
ssh vps1.example.org ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
A second Ansible control node that is not always on gets a 30-day certificate for its ansible key the same way, with a role of its own.
A lost device
For the laptop, the certificates expire within 16 hours. But its Bao token can sign new ones until the token expires. Revoke it at once from another machine:
bao login -no-print -method=userpass username=alice
bao list auth/token/accessors
bao token lookup -accessor <accessor> # find the oidc-alice tokens
bao token revoke -accessor <accessor>
For a device with a 30-day certificate, put its public key into the revoked list and roll it out:
# group_vars/all.yml
ssh_revoked_keys:
- "ssh-ed25519 AAAA...the-tablet's-public-key..."
ansible-playbook playbooks/site.yml -t base
sshd then refuses that key with any certificate. The device’s other keys (GitHub, an SFTP backup box) are plain keys and must be removed there, as before.
When Bao is gone
- Certificates that are already issued keep working until they expire. Hosts check them offline.
- After that:
ssh-break-glass <host>for a login, andssh-break-glass -- ansible-playbook ...for Ansible. - Without pass, too: the provider’s VNC console for the VPS, the local console at home.
- Then bring Bao back, or restore it on another host from its Raft snapshot (the OpenBao post).
How I nearly locked myself out
Two things went wrong during this work. Each one taught me a rule.
fail2ban banned my home IP
During a negative test I connected to a host by its bare FQDN, without the User that my SSH config sets for its short name. ssh therefore used my local user name. That account is not in the host’s AllowUsers, and fail2ban counts such a login as a failure. Together with my negative tests that made three failures within ten minutes, and fail2ban banned my home IP for an hour.
Interestingly, a refused key alone (“Failed publickey”) does not count in fail2ban’s sshd filter. A login as an account that AllowUsers does not allow does count.
I got in again through another of my hosts, which still had a working connection, and unbanned the IP there:
ssh -J other-host vps1.example.org 'sudo fail2ban-client set sshd unbanip 203.0.113.50'
The rule: test negative cases sparingly, with explicit user names, and know how you get in from another IP.
Ansible’s tests passed without a certificate
After the first host, all Ansible tests passed. Then I noticed that my laptop had no Ansible certificate at all. The ssh-cert script signed only the login key at that point. Ansible had passed because it reused its SSH connections from before the change (ControlPersist). A fresh connection would have failed.
Since then ssh-cert signs both keys by default, and I close ~/.ansible/cp/* before every test.
The rules, in short
- Keep a standby session open while you change anything that controls SSH access.
- Test with fresh connections:
-o ControlMaster=no -o ControlPath=none, and close Ansible’s connections. - Test the break-glass path before you need it.
- Keep the principals and the keys in the inventory, and make each list exclusive.
- Short certificates beat revocation. Revoke the token that can sign, not the certificate.
Where this leaves me
Every host trusts one CA key and lists, per account, which principals may log in. My laptop gets certificates for the day with one command and needs no known_hosts entries for my hosts. CI jobs get a certificate for five minutes and can run one command. A reinstalled host gets a new host certificate during its first Ansible run, and nobody has to check a fingerprint over a console. The only plain key left lives in pass.
That was the last post of the series. Part 1, about consolidating the servers, starts with Four servers, three distributions: why I consolidated.
Further reading
- ansible.posix.authorized_key:
exclusiveand the other options. - ssh-add(1):
-tfor a lifetime and-for a key from stdin. - ssh_config(5), ControlMaster: connection reuse, and why it hides a changed login.
- Ansible ssh connection plugin:
control_pathandssh_args. - bao token revoke: revoke by accessor, for a lost laptop.
- sshd_config(5), RevokedKeys: plain key lists and KRLs.
- Blink Shell documentation: keys and certificates on iOS.
- fail2ban: what the sshd filter counts.