Four servers, three distributions: why I consolidated
Where I started
I run a few small servers for myself, my family and a couple of friends. Over the years they grew the way such things grow: one at a time, each with the tools I liked at that moment. When I finally wrote down what I had, it looked like this:
| Host | Distribution | Runs | Managed with |
|---|---|---|---|
mail1.example.org | Ubuntu 24.04 | Mailcow, Forgejo, Matrix, a paste bin, about 35 containers | Docker Compose |
vps2.example.org | Arch Linux | three static websites | rootless Podman, by hand |
vps1.example.org | Fedora Cloud | a CI runner, a push notification server | rootless Podman, quadlets |
home1.example.net | Fedora Server | monitoring, media, DNS, about 20 containers | rootless Podman, quadlets |
(All names in this series are examples. My real hosts have other names.)
Four servers, three distributions and four ways to manage them. The configuration was in four Git repositories. A script copied shared files between them, when I remembered to run it. How a host was set up was in READMEs, in shell history and in my head.
Nothing was broken. But every change was slow, because I first had to remember how this host did things. And I knew that a reinstall of the mail server would cost me a weekend of reading my own old notes.
What I wanted
I wrote down four goals before I touched anything:
- One distribution everywhere: Fedora.
- The OS setup and every app rolled out with Ansible, from one repository.
- Every app as a rootless Podman quadlet: a systemd unit, no Compose file, no root daemon.
- A reinstall of any host is a routine job: install, run the playbook, restore the data.
The decisions, and why
Fedora everywhere
I already had two Fedora hosts, and I liked how they behaved. Fedora ships a current Podman with quadlet support, systemd, firewalld and SELinux, all of them integrated and enabled by default. A new release comes every six months, and dnf system-upgrade takes a few minutes. Ubuntu and Arch would work too. But with one distribution there is one package manager, one firewall and one set of paths, and my Ansible roles can be much simpler.
Rootless Podman with quadlets
A quadlet is a small ini file, for example ~/.config/containers/systemd/blog.container. systemd turns it into a normal service at boot. Each container is then a unit that you start, stop and read logs from like any other service. Rootless means that the containers run as a normal user. A container that breaks out lands in an unprivileged account, and with SELinux it lands in a confined domain too.
The next post shows the move from Compose to quadlets in detail.
Docker only where nothing else works
Mailcow is the exception. It supports only Docker with Compose, and its update script expects exactly that. A mail server is not where I want to experiment, so Mailcow stays on Docker. Everything else on that host moves to rootless Podman.
One repository
All hosts, all apps, all docs in one Git repository. A change that touches three hosts is one commit. The repository is the truth: if a setting is not in it, it does not exist.
Secrets in pass, not in Git
I already use pass (GPG-encrypted files in a Git repository) for my passwords. Ansible can read from it with the community.general.passwordstore lookup. So there is no Ansible Vault and no encrypted blob in the infrastructure repository. The repository can be public in principle, and nothing secret leaks with it.
SELinux enforcing, no exceptions
Many container guides start with setenforce 0. I wanted the opposite. Every host is enforcing, and the playbook stops if one is not. Every local policy change is code in the repository. Post 4 is about this.
Ansible owns configuration, not versions
Ansible writes the quadlets, config files and secrets. It does not pin image versions. Images are updated by podman auto-update (daily). Images I build myself are updated by their CI pipelines. With this split, a playbook run is predictable: it changes what I changed in Git and nothing else.
The strategy: adopt first, then rebuild
The obvious way would be to write roles, reinstall a server and see what breaks. I did not want that for hosts that other people use.
So I turned it around:
- Adopt the hosts that already run Fedora. I wrote the roles against the live hosts. I ran
ansible-playbook --check --diffagain and again and changed the code until the diff was empty. At that point the code describes the host exactly as it is. The real run then changes nothing. This tests the roles on real systems and puts nothing at risk. - Rehearse on a VM. For the hosts that must be reinstalled, I made a local Fedora VM with the same settings. I ran the full rollout on it and restored the real backups into it. Then I tested the services against the VM.
- Reinstall for real. Only after the rehearsal passed did I reinstall the real host. Same steps, same order.
This takes longer than a “big bang”. But at no point was a service down because of untested code.
The phases
| Phase | What | Host(s) |
|---|---|---|
| 0 | Repository, collections, inventory, ansible user | all |
| 1 | Write the roles, adopt the first Fedora host | vps1 |
| 2 | Adopt the home server (monitoring, about 20 apps) | home1 |
| 3 | Rehearsal VMs | VMs on a spare box |
| 4 | Reinstall the Arch host with Fedora | vps2 |
| 5 | Reinstall the Ubuntu mail host with Fedora | mail1 |
| 6 | Clean up: archive the old repositories |
The repository layout
This is the layout I ended up with. The next posts go through each part.
ops/
├── ansible.cfg
├── requirements.yml # pinned collections
├── inventory/
│ ├── hosts.yml # hosts and groups
│ ├── group_vars/all.yml # settings for every host
│ ├── group_vars/<group>.yml
│ └── host_vars/<host>.yml # app_user, apps: [...], firewall, backup, ...
├── playbooks/
│ ├── bootstrap.yml # first contact: creates the ansible user
│ └── site.yml # everything else, with tags
├── roles/
│ ├── base/ # packages, sshd, firewalld, fail2ban, updates, SELinux
│ ├── podman/ # the app user, linger, socket, auto-update timer
│ ├── quadlet_app/ # one generic role for every app
│ ├── monitoring_agent/ # WireGuard tunnel, node_exporter
│ ├── backup/ # restic, timer, healthchecks
│ └── ...
├── apps/<name>/ # one directory per app: quadlets, files, app.yml
├── selinux/ # local policy modules as source
├── tools/ # small scripts: rehearsal VM, image build, restore
└── docs/
The most important idea is in apps/. There is no hand-written role per service. There is one generic role, and one directory per app with its quadlet files and config. Adding an app means adding a directory and putting its name into the host’s apps: list. Post 6 shows how that role works.
How I check that it works
For every phase I used the same tests:
ansible-playbook site.yml --check --diffshows no changes on an adopted host.- A second real run reports
changed=0. A role that changes something on every run hides real changes, so I treat that as a bug. - After each phase all monitoring targets are up and no alert fires.
- After a reinstall the backup restore works. That was also my first real restore test in years.
The series
- This post: why and how.
- From Docker Compose to rootless Podman quadlets
- Traefik in front of rootless containers
- SELinux enforcing with containers
- Ansible groundwork: inventory, bootstrap and a base role
- One generic Ansible role for every app
- Monitoring and backups for every host
- Rehearse before you reinstall
A second part of the series is about SSH. I replaced all authorized_keys files with short-lived SSH certificates from my own CA, run on OpenBao. It starts with Why SSH certificates.
Next: From Docker Compose to rootless Podman quadlets.
Further reading
- Fedora Server documentation: installation, administration and the release cycle.
- Podman: the project home, with links to the docs and the blog.
- Ansible documentation: the entry point for everything in part 1.
- pass, the standard Unix password manager: GPG files in a git repository, the secrets store of this series.
- Getting started with SELinux (Fedora Quick Docs): modes, contexts and the first commands.
- Mailcow documentation: the one Docker exception in my setup.