Linux Lunatix

Four servers, three distributions: why I consolidated

Stefan Marx
| 7 minutes

Where I started

I run a few small servers for myself, my family and a couple of friends. Over the years they grew the way such things grow: one at a time, each with the tools I liked at that moment. When I finally wrote down what I had, it looked like this:

HostDistributionRunsManaged with
mail1.example.orgUbuntu 24.04Mailcow, Forgejo, Matrix, a paste bin, about 35 containersDocker Compose
vps2.example.orgArch Linuxthree static websitesrootless Podman, by hand
vps1.example.orgFedora Clouda CI runner, a push notification serverrootless Podman, quadlets
home1.example.netFedora Servermonitoring, media, DNS, about 20 containersrootless Podman, quadlets

(All names in this series are examples. My real hosts have other names.)

Four servers, three distributions and four ways to manage them. The configuration was in four Git repositories. A script copied shared files between them, when I remembered to run it. How a host was set up was in READMEs, in shell history and in my head.

Nothing was broken. But every change was slow, because I first had to remember how this host did things. And I knew that a reinstall of the mail server would cost me a weekend of reading my own old notes.

What I wanted

I wrote down four goals before I touched anything:

  1. One distribution everywhere: Fedora.
  2. The OS setup and every app rolled out with Ansible, from one repository.
  3. Every app as a rootless Podman quadlet: a systemd unit, no Compose file, no root daemon.
  4. A reinstall of any host is a routine job: install, run the playbook, restore the data.

The decisions, and why

Fedora everywhere

I already had two Fedora hosts, and I liked how they behaved. Fedora ships a current Podman with quadlet support, systemd, firewalld and SELinux, all of them integrated and enabled by default. A new release comes every six months, and dnf system-upgrade takes a few minutes. Ubuntu and Arch would work too. But with one distribution there is one package manager, one firewall and one set of paths, and my Ansible roles can be much simpler.

Rootless Podman with quadlets

A quadlet is a small ini file, for example ~/.config/containers/systemd/blog.container. systemd turns it into a normal service at boot. Each container is then a unit that you start, stop and read logs from like any other service. Rootless means that the containers run as a normal user. A container that breaks out lands in an unprivileged account, and with SELinux it lands in a confined domain too.

The next post shows the move from Compose to quadlets in detail.

Docker only where nothing else works

Mailcow is the exception. It supports only Docker with Compose, and its update script expects exactly that. A mail server is not where I want to experiment, so Mailcow stays on Docker. Everything else on that host moves to rootless Podman.

One repository

All hosts, all apps, all docs in one Git repository. A change that touches three hosts is one commit. The repository is the truth: if a setting is not in it, it does not exist.

Secrets in pass, not in Git

I already use pass (GPG-encrypted files in a Git repository) for my passwords. Ansible can read from it with the community.general.passwordstore lookup. So there is no Ansible Vault and no encrypted blob in the infrastructure repository. The repository can be public in principle, and nothing secret leaks with it.

SELinux enforcing, no exceptions

Many container guides start with setenforce 0. I wanted the opposite. Every host is enforcing, and the playbook stops if one is not. Every local policy change is code in the repository. Post 4 is about this.

Ansible owns configuration, not versions

Ansible writes the quadlets, config files and secrets. It does not pin image versions. Images are updated by podman auto-update (daily). Images I build myself are updated by their CI pipelines. With this split, a playbook run is predictable: it changes what I changed in Git and nothing else.

The strategy: adopt first, then rebuild

The obvious way would be to write roles, reinstall a server and see what breaks. I did not want that for hosts that other people use.

So I turned it around:

  1. Adopt the hosts that already run Fedora. I wrote the roles against the live hosts. I ran ansible-playbook --check --diff again and again and changed the code until the diff was empty. At that point the code describes the host exactly as it is. The real run then changes nothing. This tests the roles on real systems and puts nothing at risk.
  2. Rehearse on a VM. For the hosts that must be reinstalled, I made a local Fedora VM with the same settings. I ran the full rollout on it and restored the real backups into it. Then I tested the services against the VM.
  3. Reinstall for real. Only after the rehearsal passed did I reinstall the real host. Same steps, same order.

This takes longer than a “big bang”. But at no point was a service down because of untested code.

The phases

PhaseWhatHost(s)
0Repository, collections, inventory, ansible userall
1Write the roles, adopt the first Fedora hostvps1
2Adopt the home server (monitoring, about 20 apps)home1
3Rehearsal VMsVMs on a spare box
4Reinstall the Arch host with Fedoravps2
5Reinstall the Ubuntu mail host with Fedoramail1
6Clean up: archive the old repositories

The repository layout

This is the layout I ended up with. The next posts go through each part.

ops/
├── ansible.cfg
├── requirements.yml            # pinned collections
├── inventory/
│   ├── hosts.yml               # hosts and groups
│   ├── group_vars/all.yml      # settings for every host
│   ├── group_vars/<group>.yml
│   └── host_vars/<host>.yml    # app_user, apps: [...], firewall, backup, ...
├── playbooks/
│   ├── bootstrap.yml           # first contact: creates the ansible user
│   └── site.yml                # everything else, with tags
├── roles/
│   ├── base/                   # packages, sshd, firewalld, fail2ban, updates, SELinux
│   ├── podman/                 # the app user, linger, socket, auto-update timer
│   ├── quadlet_app/            # one generic role for every app
│   ├── monitoring_agent/       # WireGuard tunnel, node_exporter
│   ├── backup/                 # restic, timer, healthchecks
│   └── ...
├── apps/<name>/                # one directory per app: quadlets, files, app.yml
├── selinux/                    # local policy modules as source
├── tools/                      # small scripts: rehearsal VM, image build, restore
└── docs/

The most important idea is in apps/. There is no hand-written role per service. There is one generic role, and one directory per app with its quadlet files and config. Adding an app means adding a directory and putting its name into the host’s apps: list. Post 6 shows how that role works.

How I check that it works

For every phase I used the same tests:

The series

  1. This post: why and how.
  2. From Docker Compose to rootless Podman quadlets
  3. Traefik in front of rootless containers
  4. SELinux enforcing with containers
  5. Ansible groundwork: inventory, bootstrap and a base role
  6. One generic Ansible role for every app
  7. Monitoring and backups for every host
  8. Rehearse before you reinstall

A second part of the series is about SSH. I replaced all authorized_keys files with short-lived SSH certificates from my own CA, run on OpenBao. It starts with Why SSH certificates.

Next: From Docker Compose to rootless Podman quadlets.

Further reading

Previous: A note about git merge Next: From Docker Compose to rootless Podman quadlets

Related content

Rehearse before you reinstall
Monitoring and backups for every host
One generic Ansible role for every app