Linux Lunatix

Rehearse before you reinstall

Stefan Marx
| 11 minutes

Part 8 of the series that starts with Four servers, three distributions: why I consolidated.

Two of my hosts had to be reinstalled: the Arch VPS with the websites, and the Ubuntu host with mail and Git. Before I touched either of them, I ran the whole reinstall on a local VM: the install, the playbook, the restore of the real backups and the tests. This post shows how. The same approach works for any host you are afraid to touch.

The idea

A rehearsal VM is the real host with three differences:

  1. It runs on a machine at home, on libvirt’s NAT network.
  2. It cannot talk to the world in ways that would hurt: no outgoing mail, no Matrix federation, no certificate requests, no CI jobs.
  3. It never runs the real host’s backup timer or monitoring tunnel.

Everything else is real: the real inventory settings, the real secrets from pass and the real backup repository (read-only). If the rehearsal passes, the cut-over is a repeat of known steps.

Step 1: a VM from the Fedora Cloud image

A provider’s Fedora image is the Fedora Cloud image with cloud-init. So I start from the same image. Download it and check its signature first:

cd /var/lib/libvirt/images/base
curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2
curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-44-1.7-x86_64-CHECKSUM
curl -O https://fedoraproject.org/fedora.gpg
gpgv --keyring ./fedora.gpg Fedora-Cloud-44-1.7-x86_64-CHECKSUM
sha256sum -c --ignore-missing Fedora-Cloud-44-1.7-x86_64-CHECKSUM

Then a small script creates a VM from it. Each VM gets a thin qcow2 overlay on the base image. cloud-init creates the user fedora with my key and passwordless sudo, the same starting point as a fresh VPS:

#!/bin/bash
# rehearsal-vm.sh create <name> <memory MiB> <vcpus> <disk GB> <ssh pubkey file>
# rehearsal-vm.sh destroy <name>
# rehearsal-vm.sh ip <name>
set -euo pipefail

BASE_DIR=/var/lib/libvirt/images/base
IMAGE=${IMAGE:-$BASE_DIR/Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2}
VM_DIR=/var/lib/libvirt/images
export LIBVIRT_DEFAULT_URI=qemu:///system

cmd=${1:?command}; name=${2:?name}

case $cmd in
create)
    mem=${3:?memory MiB}; cpus=${4:?vcpus}; disk=${5:?disk GB}; key=$(cat "${6:?ssh pubkey file}")
    tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
    cat >"$tmp/user-data" <<EOF
#cloud-config
hostname: $name
users:
  - name: fedora
    groups: [wheel]
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys: ["$key"]
EOF
    sudo qemu-img create -q -f qcow2 -F qcow2 -b "$IMAGE" "$VM_DIR/$name.qcow2" "${disk}G"
    virt-install --name "$name" --memory "$mem" --vcpus "$cpus" \
        --disk "$VM_DIR/$name.qcow2,bus=virtio" --import \
        --os-variant fedora-unknown --network network=default \
        --cloud-init user-data="$tmp/user-data" \
        --graphics none --noautoconsole
    ;;
destroy)
    virsh destroy "$name" 2>/dev/null || true
    virsh undefine "$name" --remove-all-storage
    ;;
ip)
    virsh domifaddr "$name" --source lease | awk '/ipv4/ {sub("/.*", "", $4); print $4}'
    ;;
esac
rehearsal-vm.sh create mail1-rehearsal 8192 4 100 ~/.ssh/id_ed25519.pub
rehearsal-vm.sh ip mail1-rehearsal          # e.g. 192.168.122.76

virt-install --cloud-init builds the seed ISO by itself. You need no separate cloud-localds step.

Step 2: an inventory for the VMs

The VMs are in a separate inventory file, which I pass only when I mean it:

# inventory/rehearsal.yml
all:
  hosts:
    mail1-rehearsal:
      ansible_host: 192.168.122.76
      ansible_ssh_common_args: >-
        -J me@vmhost.example.net
        -o UserKnownHostsFile=~/.ssh/known_hosts.rehearsal -o StrictHostKeyChecking=accept-new
      bootstrap_user: fedora
      hostname: mail1-rehearsal
      # use the real host's secrets from pass
      pass_host: mail1
      backup_timer: false
      monitoring_tunnel: false
      # the CI runner would register under the real name and take real jobs
      docker_compose_projects: []
  children:
    fedora:
      hosts: {mail1-rehearsal: }
    mail:
      hosts: {mail1-rehearsal: }

The trick is where the real host’s settings live. They are not in host_vars/mail1.yml, but in group_vars/mail.yml, for a group with one member. The VM joins the same group and gets all of them. The few variables in rehearsal.yml are host variables, and those beat group variables. So the VM can switch off the backup timer, the tunnel and the runner.

The VMs are recreated often. Each new VM has new host keys, so the VMs get their own known_hosts file. My real known_hosts stays clean. accept-new is acceptable here, and only here: the VM is on a NAT network behind my own machine.

Step 3: guards

The rehearsal VM holds the real data and the real credentials. Left alone, it would:

The inventory handles the runner. For the rest I put an nftables table on the VM that drops the dangerous outgoing traffic. Something like this:

table inet rehearsal_guard {
    chain output {
        type filter hook output priority 0; policy accept;
        oifname "lo" accept
        tcp dport { 25, 465, 587, 8448 } drop
        # once the images are pulled: no new connections from the app user
        meta skuid "svc" ct state new drop
    }
}

The app user’s rule goes in only after the first full run, because Podman must pull the images first. DNS needs one more step: unbound marks the root servers as down while the block is on. Restart it after you lift the block for a test.

Step 4: the rollout, in this order

I="-i inventory/hosts.yml -i inventory/rehearsal.yml"

# 1. the ansible user
ansible-playbook $I playbooks/bootstrap.yml -l mail1-rehearsal
# 2. update and reboot at once (see post 5: dnf-automatic would do it mid-run)
ansible $I mail1-rehearsal -b -m shell -a 'dnf -y upgrade && systemctl reboot'
# 3. everything except the apps
ansible-playbook $I playbooks/site.yml -l mail1-rehearsal -t base,monitoring,backup
# 4. restore the state (below), before Traefik starts
# 5. everything
ansible-playbook $I playbooks/site.yml -l mail1-rehearsal
# 6. once more, with fresh connections: changed=0
ansible-playbook $I playbooks/site.yml -l mail1-rehearsal \
    -e ansible_ssh_common_args="-J me@vmhost.example.net -o ControlPath=none -o UserKnownHostsFile=~/.ssh/known_hosts.rehearsal"

Step 3 installs restic and the backup credentials, but no apps. The restore can then put the volumes in place before any container starts on empty data.

Step 5: the restore

The restore script runs on the new host as root and reads the old host’s restic repository with --no-lock. It has three parts, because the order matters:

restore.sh files    # volumes and Traefik's acme.json: before the apps start
restore.sh dbs      # database dumps: after the full run (the databases must run)
restore.sh mailcow  # Mailcow's own backup and its local config: after the full run

The files part copies each old bind mount into a new Podman volume with podman unshare tar (shown in post 2). The dbs part drops the empty databases that the containers created at their first start, then loads the dumps. The mailcow part restores Mailcow’s own config files over the checkout. Then it runs Mailcow’s backup_and_restore.sh restore, which asks four questions. A printf answers them:

printf '1\n0\nY\ny\n' | MAILCOW_BACKUP_LOCATION="$tmp/backup" \
    /opt/mailcow-dockerized/helper-scripts/backup_and_restore.sh restore

Step 6: tests

Test the services against the VM without DNS changes:

ip=192.168.122.76
for h in git.example.org chat.example.org paste.example.org mail1.example.org; do
  curl -s -o /dev/null -w "$h %{http_code} verify=%{ssl_verify_result}\n" \
       --resolve $h:443:$ip https://$h/
done

verify=0 means that the restored certificate is valid for the name. I also compared counts from the old host with the VM: mail domains, mailboxes, aliases, DKIM keys, the number of messages in one INBOX, and the repositories in Forgejo. They must be identical.

On the VM itself:

sudo ausearch -m avc -ts boot          # no SELinux denials
ps -eZ | grep -c container_t           # every container confined

Mailcow on Docker with SELinux enforcing gave no denials at all, which surprised me.

Things a VM cannot test: real mail delivery, certificates from the HTTP challenge (no inbound port 80), the CI runner. I wrote those down as checks for the real cut-over.

A provider without a Fedora image

My VPS provider for the website host offers no Fedora image. It does accept an uploaded qcow2 image, but it has no cloud metadata service. A plain Fedora Cloud image would boot there and wait for cloud-init data that never comes, so there would be no user and no key.

The fix is to bake the cloud-init data into the image, as a NoCloud seed. I built it from a rehearsal VM:

rehearsal-vm.sh create image-build 2048 2 6 ~/.ssh/id_ed25519.pub
ssh fedora@<vm> 'sudo dnf -y upgrade && sudo systemctl reboot'
ssh fedora@<vm> 'sudo bash -s' -- vps2.example.org "$(cat ~/.ssh/id_ed25519.pub)" < seal-image.sh

The script, slightly shortened:

#!/bin/bash
# seal-image.sh <fqdn> <ssh public key>: run as root inside the build VM
set -euo pipefail
fqdn=${1:?fqdn}; key=${2:?ssh public key}

install -d -m 0755 /var/lib/cloud/seed/nocloud
cat > /var/lib/cloud/seed/nocloud/meta-data <<EOF
instance-id: ${fqdn%%.*}-$(date +%Y%m%d)
local-hostname: ${fqdn%%.*}
EOF
cat > /var/lib/cloud/seed/nocloud/user-data <<EOF
#cloud-config
ssh_pwauth: false
disable_root: true
users:
  - name: fedora
    groups: [wheel]
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    lock_passwd: true
    ssh_authorized_keys: ["$key"]
EOF
cat > /etc/cloud/cloud.cfg.d/90-datasource.cfg <<'EOF'
# No metadata service: only the seed in /var/lib/cloud/seed/nocloud
datasource_list: [ NoCloud, None ]
preserve_hostname: true
EOF
restorecon -R /var/lib/cloud /etc/cloud

# The provider's VMs have no serial port: drop console=ttyS0, keep GRUB on the VNC console
grubby --update-kernel=ALL --remove-args="console=ttyS0,115200n8"
sed -i -e 's/ console=ttyS0,115200n8//' \
    -e 's/^GRUB_TERMINAL_INPUT=.*/GRUB_TERMINAL_INPUT="console"/' \
    -e 's/^GRUB_TERMINAL_OUTPUT=.*/GRUB_TERMINAL_OUTPUT="console"/' /etc/default/grub
sed -i '/^GRUB_SERIAL_COMMAND=/d' /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg 2>/dev/null

# First-boot state: new machine-id, new host keys, no network profile bound to this MAC
rm -f /etc/NetworkManager/system-connections/cloud-init-*
cloud-init clean --logs --machine-id
rm -f /etc/ssh/ssh_host_* /home/fedora/.bash_history /root/.bash_history
echo "$fqdn" > /etc/hostname
restorecon /etc/hostname
dnf clean all -q
journalctl --rotate -q
journalctl --vacuum-time=1s -q
fstrim -a
echo SEALED

Then power the VM off and compress its disk into the image:

sudo qemu-img convert -c -O qcow2 /var/lib/libvirt/images/image-build.qcow2 vps2-fedora44.qcow2
sha256sum vps2-fedora44.qcow2 > vps2-fedora44.qcow2.sha256

Some details matter here:

Test the image the way the provider runs it. Mine boots with BIOS, a virtio-scsi disk and no cloud-init ISO:

cp vps2-fedora44.qcow2 test.qcow2 && qemu-img resize test.qcow2 117G
virt-install --name image-test --memory 2048 --vcpus 2 --import \
    --controller type=scsi,model=virtio-scsi --disk test.qcow2,bus=scsi \
    --os-variant fedora-unknown --network network=default --graphics none --noautoconsole
ssh fedora@<test vm> cloud-init status --long     # status: done, DataSourceNoCloud

cloud-init grows the root filesystem to the full disk at the first boot.

The cut-over

With a passed rehearsal, the cut-over is a list:

  1. On the old host: stop what writes (the web proxy, the mail daemons), run a last backup, check its log and note the snapshot ID.
  2. Reinstall the host in the provider’s panel, with the provider’s Fedora image or the uploaded one. The IP address stays.
  3. Check the host key over the console (below).
  4. Inventory: the host’s bootstrap_user, and the host into the fedora group.
  5. The same six rollout steps as on the VM, with -l mail1, and with the snapshot ID for the restore.
  6. The checks that the VM could not do: real mail in both directions, Mailcow’s certificate, the runner online, a deploy of a site, the monitoring targets, the first backup run.

The provider’s reinstall offered Fedora 43 only. Upgrading to 44 right after the bootstrap took under two minutes on the VM:

sudo dnf -y upgrade --refresh
sudo dnf -y system-upgrade download --releasever=44
sudo dnf -y offline reboot

Checking the new host key

After a reinstall, ssh warns that the host key changed. That is expected, and it is exactly the moment an attacker in the middle would use. So I check the new key over a channel that the attacker does not control: the provider’s VNC console.

cloud-init prints the host key fingerprints during the first boot, but they scroll by quickly. A trick that always works:

# keep the new key apart from the real known_hosts
ssh -o UserKnownHostsFile=/tmp/new_host -o StrictHostKeyChecking=accept-new fedora@vps2.example.org \
  "printf '%s\n' \"code 7341: \$(ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub)\" | sudo tee /dev/tty1"

Pick a new random code each time. If the code appears in the VNC console with the fingerprint, the connection reached the real host and not some machine in between. Then compare the fingerprint with ssh-keygen -lf /tmp/new_host, and only then put the key into your real known_hosts.

The second part of this series removes this step: with host certificates, the host proves its identity by itself.

What the rehearsals found

Each of these would have cost me time during a real cut-over, with mail down.

This was the last post of part 1. Part 2 is about SSH: Why SSH certificates.

Further reading

Previous: Monitoring and backups for every host Next: Why SSH certificates

Related content

Monitoring and backups for every host
One generic Ansible role for every app
Ansible groundwork: inventory, bootstrap and a base role