Rehearse before you reinstall
Part 8 of the series that starts with Four servers, three distributions: why I consolidated.
Two of my hosts had to be reinstalled: the Arch VPS with the websites, and the Ubuntu host with mail and Git. Before I touched either of them, I ran the whole reinstall on a local VM: the install, the playbook, the restore of the real backups and the tests. This post shows how. The same approach works for any host you are afraid to touch.
The idea
A rehearsal VM is the real host with three differences:
- It runs on a machine at home, on libvirt’s NAT network.
- It cannot talk to the world in ways that would hurt: no outgoing mail, no Matrix federation, no certificate requests, no CI jobs.
- It never runs the real host’s backup timer or monitoring tunnel.
Everything else is real: the real inventory settings, the real secrets from pass and the real backup repository (read-only). If the rehearsal passes, the cut-over is a repeat of known steps.
Step 1: a VM from the Fedora Cloud image
A provider’s Fedora image is the Fedora Cloud image with cloud-init. So I start from the same image. Download it and check its signature first:
cd /var/lib/libvirt/images/base
curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2
curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-44-1.7-x86_64-CHECKSUM
curl -O https://fedoraproject.org/fedora.gpg
gpgv --keyring ./fedora.gpg Fedora-Cloud-44-1.7-x86_64-CHECKSUM
sha256sum -c --ignore-missing Fedora-Cloud-44-1.7-x86_64-CHECKSUM
Then a small script creates a VM from it. Each VM gets a thin qcow2 overlay on the base image. cloud-init creates the user fedora with my key and passwordless sudo, the same starting point as a fresh VPS:
#!/bin/bash
# rehearsal-vm.sh create <name> <memory MiB> <vcpus> <disk GB> <ssh pubkey file>
# rehearsal-vm.sh destroy <name>
# rehearsal-vm.sh ip <name>
set -euo pipefail
BASE_DIR=/var/lib/libvirt/images/base
IMAGE=${IMAGE:-$BASE_DIR/Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2}
VM_DIR=/var/lib/libvirt/images
export LIBVIRT_DEFAULT_URI=qemu:///system
cmd=${1:?command}; name=${2:?name}
case $cmd in
create)
mem=${3:?memory MiB}; cpus=${4:?vcpus}; disk=${5:?disk GB}; key=$(cat "${6:?ssh pubkey file}")
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
cat >"$tmp/user-data" <<EOF
#cloud-config
hostname: $name
users:
- name: fedora
groups: [wheel]
sudo: "ALL=(ALL) NOPASSWD:ALL"
ssh_authorized_keys: ["$key"]
EOF
sudo qemu-img create -q -f qcow2 -F qcow2 -b "$IMAGE" "$VM_DIR/$name.qcow2" "${disk}G"
virt-install --name "$name" --memory "$mem" --vcpus "$cpus" \
--disk "$VM_DIR/$name.qcow2,bus=virtio" --import \
--os-variant fedora-unknown --network network=default \
--cloud-init user-data="$tmp/user-data" \
--graphics none --noautoconsole
;;
destroy)
virsh destroy "$name" 2>/dev/null || true
virsh undefine "$name" --remove-all-storage
;;
ip)
virsh domifaddr "$name" --source lease | awk '/ipv4/ {sub("/.*", "", $4); print $4}'
;;
esac
rehearsal-vm.sh create mail1-rehearsal 8192 4 100 ~/.ssh/id_ed25519.pub
rehearsal-vm.sh ip mail1-rehearsal # e.g. 192.168.122.76
virt-install --cloud-init builds the seed ISO by itself. You need no separate cloud-localds step.
Step 2: an inventory for the VMs
The VMs are in a separate inventory file, which I pass only when I mean it:
# inventory/rehearsal.yml
all:
hosts:
mail1-rehearsal:
ansible_host: 192.168.122.76
ansible_ssh_common_args: >-
-J me@vmhost.example.net
-o UserKnownHostsFile=~/.ssh/known_hosts.rehearsal -o StrictHostKeyChecking=accept-new
bootstrap_user: fedora
hostname: mail1-rehearsal
# use the real host's secrets from pass
pass_host: mail1
backup_timer: false
monitoring_tunnel: false
# the CI runner would register under the real name and take real jobs
docker_compose_projects: []
children:
fedora:
hosts: {mail1-rehearsal: }
mail:
hosts: {mail1-rehearsal: }
The trick is where the real host’s settings live. They are not in host_vars/mail1.yml, but in group_vars/mail.yml, for a group with one member. The VM joins the same group and gets all of them. The few variables in rehearsal.yml are host variables, and those beat group variables. So the VM can switch off the backup timer, the tunnel and the runner.
The VMs are recreated often. Each new VM has new host keys, so the VMs get their own known_hosts file. My real known_hosts stays clean. accept-new is acceptable here, and only here: the VM is on a NAT network behind my own machine.
Step 3: guards
The rehearsal VM holds the real data and the real credentials. Left alone, it would:
- deliver mail that is waiting in the restored queue,
- federate with other Matrix servers under the real server name,
- ask Let’s Encrypt for certificates of the real domains,
- take real CI jobs as the real runner.
The inventory handles the runner. For the rest I put an nftables table on the VM that drops the dangerous outgoing traffic. Something like this:
table inet rehearsal_guard {
chain output {
type filter hook output priority 0; policy accept;
oifname "lo" accept
tcp dport { 25, 465, 587, 8448 } drop
# once the images are pulled: no new connections from the app user
meta skuid "svc" ct state new drop
}
}
The app user’s rule goes in only after the first full run, because Podman must pull the images first. DNS needs one more step: unbound marks the root servers as down while the block is on. Restart it after you lift the block for a test.
Step 4: the rollout, in this order
I="-i inventory/hosts.yml -i inventory/rehearsal.yml"
# 1. the ansible user
ansible-playbook $I playbooks/bootstrap.yml -l mail1-rehearsal
# 2. update and reboot at once (see post 5: dnf-automatic would do it mid-run)
ansible $I mail1-rehearsal -b -m shell -a 'dnf -y upgrade && systemctl reboot'
# 3. everything except the apps
ansible-playbook $I playbooks/site.yml -l mail1-rehearsal -t base,monitoring,backup
# 4. restore the state (below), before Traefik starts
# 5. everything
ansible-playbook $I playbooks/site.yml -l mail1-rehearsal
# 6. once more, with fresh connections: changed=0
ansible-playbook $I playbooks/site.yml -l mail1-rehearsal \
-e ansible_ssh_common_args="-J me@vmhost.example.net -o ControlPath=none -o UserKnownHostsFile=~/.ssh/known_hosts.rehearsal"
Step 3 installs restic and the backup credentials, but no apps. The restore can then put the volumes in place before any container starts on empty data.
Step 5: the restore
The restore script runs on the new host as root and reads the old host’s restic repository with --no-lock. It has three parts, because the order matters:
restore.sh files # volumes and Traefik's acme.json: before the apps start
restore.sh dbs # database dumps: after the full run (the databases must run)
restore.sh mailcow # Mailcow's own backup and its local config: after the full run
The files part copies each old bind mount into a new Podman volume with podman unshare tar (shown in post 2). The dbs part drops the empty databases that the containers created at their first start, then loads the dumps. The mailcow part restores Mailcow’s own config files over the checkout. Then it runs Mailcow’s backup_and_restore.sh restore, which asks four questions. A printf answers them:
printf '1\n0\nY\ny\n' | MAILCOW_BACKUP_LOCATION="$tmp/backup" \
/opt/mailcow-dockerized/helper-scripts/backup_and_restore.sh restore
Step 6: tests
Test the services against the VM without DNS changes:
ip=192.168.122.76
for h in git.example.org chat.example.org paste.example.org mail1.example.org; do
curl -s -o /dev/null -w "$h %{http_code} verify=%{ssl_verify_result}\n" \
--resolve $h:443:$ip https://$h/
done
verify=0 means that the restored certificate is valid for the name. I also compared counts from the old host with the VM: mail domains, mailboxes, aliases, DKIM keys, the number of messages in one INBOX, and the repositories in Forgejo. They must be identical.
On the VM itself:
sudo ausearch -m avc -ts boot # no SELinux denials
ps -eZ | grep -c container_t # every container confined
Mailcow on Docker with SELinux enforcing gave no denials at all, which surprised me.
Things a VM cannot test: real mail delivery, certificates from the HTTP challenge (no inbound port 80), the CI runner. I wrote those down as checks for the real cut-over.
A provider without a Fedora image
My VPS provider for the website host offers no Fedora image. It does accept an uploaded qcow2 image, but it has no cloud metadata service. A plain Fedora Cloud image would boot there and wait for cloud-init data that never comes, so there would be no user and no key.
The fix is to bake the cloud-init data into the image, as a NoCloud seed. I built it from a rehearsal VM:
rehearsal-vm.sh create image-build 2048 2 6 ~/.ssh/id_ed25519.pub
ssh fedora@<vm> 'sudo dnf -y upgrade && sudo systemctl reboot'
ssh fedora@<vm> 'sudo bash -s' -- vps2.example.org "$(cat ~/.ssh/id_ed25519.pub)" < seal-image.sh
The script, slightly shortened:
#!/bin/bash
# seal-image.sh <fqdn> <ssh public key>: run as root inside the build VM
set -euo pipefail
fqdn=${1:?fqdn}; key=${2:?ssh public key}
install -d -m 0755 /var/lib/cloud/seed/nocloud
cat > /var/lib/cloud/seed/nocloud/meta-data <<EOF
instance-id: ${fqdn%%.*}-$(date +%Y%m%d)
local-hostname: ${fqdn%%.*}
EOF
cat > /var/lib/cloud/seed/nocloud/user-data <<EOF
#cloud-config
ssh_pwauth: false
disable_root: true
users:
- name: fedora
groups: [wheel]
sudo: "ALL=(ALL) NOPASSWD:ALL"
lock_passwd: true
ssh_authorized_keys: ["$key"]
EOF
cat > /etc/cloud/cloud.cfg.d/90-datasource.cfg <<'EOF'
# No metadata service: only the seed in /var/lib/cloud/seed/nocloud
datasource_list: [ NoCloud, None ]
preserve_hostname: true
EOF
restorecon -R /var/lib/cloud /etc/cloud
# The provider's VMs have no serial port: drop console=ttyS0, keep GRUB on the VNC console
grubby --update-kernel=ALL --remove-args="console=ttyS0,115200n8"
sed -i -e 's/ console=ttyS0,115200n8//' \
-e 's/^GRUB_TERMINAL_INPUT=.*/GRUB_TERMINAL_INPUT="console"/' \
-e 's/^GRUB_TERMINAL_OUTPUT=.*/GRUB_TERMINAL_OUTPUT="console"/' /etc/default/grub
sed -i '/^GRUB_SERIAL_COMMAND=/d' /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg 2>/dev/null
# First-boot state: new machine-id, new host keys, no network profile bound to this MAC
rm -f /etc/NetworkManager/system-connections/cloud-init-*
cloud-init clean --logs --machine-id
rm -f /etc/ssh/ssh_host_* /home/fedora/.bash_history /root/.bash_history
echo "$fqdn" > /etc/hostname
restorecon /etc/hostname
dnf clean all -q
journalctl --rotate -q
journalctl --vacuum-time=1s -q
fstrim -a
echo SEALED
Then power the VM off and compress its disk into the image:
sudo qemu-img convert -c -O qcow2 /var/lib/libvirt/images/image-build.qcow2 vps2-fedora44.qcow2
sha256sum vps2-fedora44.qcow2 > vps2-fedora44.qcow2.sha256
Some details matter here:
preserve_hostname: true. cloud-init sets the hostname withhostnamectlso early that D-Bus is not up yet. That fails, andcloud-init statusthen reports “degraded”. The hostname is already in/etc/hostname, and the base role sets it again anyway.- The serial console. The provider’s VMs have no serial port. With
console=ttyS0on the kernel command line,serial-gettyrestarts every few seconds, and GRUB shows its menu only on the serial line. You would never see it in the VNC console. cloud-init clean --machine-id. Without a new machine-id, every host made from the image would have the same one, and the same DHCP client ID too.- The image contains only the user
fedorawith my public key. No secret is in it.
Test the image the way the provider runs it. Mine boots with BIOS, a virtio-scsi disk and no cloud-init ISO:
cp vps2-fedora44.qcow2 test.qcow2 && qemu-img resize test.qcow2 117G
virt-install --name image-test --memory 2048 --vcpus 2 --import \
--controller type=scsi,model=virtio-scsi --disk test.qcow2,bus=scsi \
--os-variant fedora-unknown --network network=default --graphics none --noautoconsole
ssh fedora@<test vm> cloud-init status --long # status: done, DataSourceNoCloud
cloud-init grows the root filesystem to the full disk at the first boot.
The cut-over
With a passed rehearsal, the cut-over is a list:
- On the old host: stop what writes (the web proxy, the mail daemons), run a last backup, check its log and note the snapshot ID.
- Reinstall the host in the provider’s panel, with the provider’s Fedora image or the uploaded one. The IP address stays.
- Check the host key over the console (below).
- Inventory: the host’s
bootstrap_user, and the host into thefedoragroup. - The same six rollout steps as on the VM, with
-l mail1, and with the snapshot ID for the restore. - The checks that the VM could not do: real mail in both directions, Mailcow’s certificate, the runner online, a deploy of a site, the monitoring targets, the first backup run.
The provider’s reinstall offered Fedora 43 only. Upgrading to 44 right after the bootstrap took under two minutes on the VM:
sudo dnf -y upgrade --refresh
sudo dnf -y system-upgrade download --releasever=44
sudo dnf -y offline reboot
Checking the new host key
After a reinstall, ssh warns that the host key changed. That is expected, and it is exactly the moment an attacker in the middle would use. So I check the new key over a channel that the attacker does not control: the provider’s VNC console.
cloud-init prints the host key fingerprints during the first boot, but they scroll by quickly. A trick that always works:
# keep the new key apart from the real known_hosts
ssh -o UserKnownHostsFile=/tmp/new_host -o StrictHostKeyChecking=accept-new fedora@vps2.example.org \
"printf '%s\n' \"code 7341: \$(ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub)\" | sudo tee /dev/tty1"
Pick a new random code each time. If the code appears in the VNC console with the fingerprint, the connection reached the real host and not some machine in between. Then compare the fingerprint with ssh-keygen -lf /tmp/new_host, and only then put the key into your real known_hosts.
The second part of this series removes this step: with host certificates, the host proves its identity by itself.
What the rehearsals found
- Pi-hole binds its DNS port to an address on the WireGuard interface. The WireGuard role therefore has to run before the apps, not after them.
- Fedora’s
dockerdalready runs with--selinux-enabled. Adding it todaemon.jsonas well stops the daemon. Also,moby-enginecreates no/etc/dockerdirectory. - Element’s nginx cannot bind port 80 as a rootless user. It has a setting for another port.
- Mailcow’s
update.shmakes local Git commits in its checkout, so my role checks out the release tag and not the branch. - The first
baserun locked out theansibleuser on a fresh host (theAllowUsersissue in post 5). A reused SSH connection had hidden it during the first rehearsal.
Each of these would have cost me time during a real cut-over, with mail down.
This was the last post of part 1. Part 2 is about SSH: Why SSH certificates.
Further reading
- Fedora Cloud: the qcow2 images used for the rehearsal VMs.
- cloud-init NoCloud datasource: the seed that the custom image carries.
- libvirt documentation: domains, networks and storage.
- virt-install: all options, including
--cloud-initand--import. - virt-customize: changes a disk image without booting it. An alternative to a build VM.
- Upgrading Fedora using the DNF system upgrade: the upgrade path from the end of this post.