Ansible groundwork: inventory, bootstrap and a base role
Part 5 of the series that starts with Four servers, three distributions: why I consolidated.
This post sets up the parts that every host shares. At the end you have a control node, an inventory, a playbook for the first contact with a host, and a base role. That role hardens sshd, runs firewalld and fail2ban, installs updates every night and caps the journal.
The control node
I run Ansible from my laptop, and from a second machine as a spare. Both use the same pinned versions in a virtualenv. The distribution’s Ansible package changes with each upgrade, and I want the same behaviour on both machines:
python3 -m venv ~/.local/share/ops-venv
~/.local/share/ops-venv/bin/pip install "ansible-core==2.22.0" ansible-lint yamllint
export PATH=~/.local/share/ops-venv/bin:$PATH
The collections are pinned too, and installed into the repository:
# requirements.yml
collections:
- name: ansible.posix
version: "2.2.2"
- name: community.general
version: "14.0.0"
- name: containers.podman
version: "1.21.2"
ansible-galaxy collection install -r requirements.yml -p collections
Each control node has its own key for Ansible, ~/.ssh/ansible_ed25519, separate from my login key. I can then revoke one machine without touching the other.
ansible.cfg
[defaults]
inventory = inventory/hosts.yml
roles_path = roles
collections_path = collections
remote_user = ansible
private_key_file = ~/.ssh/ansible_ed25519
interpreter_python = auto_silent
callback_result_format = yaml
forks = 4
# Handlers run even when a later task fails
force_handlers = True
[privilege_escalation]
become = True
[diff]
always = True
[ssh_connection]
pipelining = True
Two settings are worth an explanation.
force_handlers runs the notified handlers even if a later task fails. Without it, this can happen: a task changes the sshd config and notifies “Reload sshd”, and then a later task fails. The handler never runs. On the next run the config file is unchanged, so nothing notifies the handler again. sshd runs with the old config until somebody notices.
diff always shows every change as a diff. Together with --check this makes a run a review: I see what would change before it changes.
The inventory
# inventory/hosts.yml
all:
hosts:
mail1:
ansible_host: mail1.example.org
bootstrap_user: admin
tunnel_ip: 10.99.0.11
vps1:
ansible_host: vps1.example.org
bootstrap_user: fedora
tunnel_ip: 10.99.0.13
app_user: svc
vps2:
ansible_host: vps2.example.org
bootstrap_user: fedora
tunnel_ip: 10.99.0.12
home1:
ansible_host: home1.example.net
bootstrap_user: admin
tunnel_ip: 10.99.0.1
app_user: svc
children:
vps:
hosts: {mail1: , vps1: , vps2: }
home:
hosts: {home1: }
mail:
hosts: {mail1: }
websites:
hosts: {vps2: }
# The hosts that site.yml manages. A host joins after its reinstall with Fedora.
fedora:
hosts: {vps1: , home1: , vps2: }
The fedora group is how I adopted the hosts one by one. A host that still ran Ubuntu or Arch was in the inventory, but site.yml did not touch it. It joined the group after its reinstall.
Everything else goes into group_vars/ and host_vars/. A host file says what is special about that host:
# inventory/host_vars/vps1.yml
hostname: vps1.example.org
timezone: Europe/Berlin
ssh_allow_users: [fedora, svc]
firewall_public_services: [http, https, ssh]
dnf_automatic_reboot: when-needed
unprivileged_port_start: 80
apps: [traefik, ntfy, forgejo-runner]
Secrets from pass
I keep all secrets in pass on the control node. The lookup plugin reads them during the run:
- name: Tunnel private key
ansible.builtin.copy:
dest: /etc/wireguard/wg-mon.key
content: "{{ lookup('community.general.passwordstore', 'infra/' ~ inventory_hostname ~ '/wg-mon-key') }}\n"
mode: "0600"
no_log: true
diff: false
no_log and diff: false keep the secret out of the output. Without diff: false, diff always would print the old and the new key.
Compared with Ansible Vault, the repository contains nothing encrypted at all, and there is no vault password to share between the control nodes. A secret is changed in one place, in pass, and the next run rolls it out.
First contact: bootstrap.yml
A fresh host has one admin account that the provider set up (fedora on a Fedora Cloud image). bootstrap.yml logs in as that account once and creates the ansible user. Every other playbook then uses that user.
# playbooks/bootstrap.yml
- name: Bootstrap the ansible user
hosts: all
remote_user: "{{ bootstrap_user }}"
vars:
ansible_ssh_private_key_file: ~/.ssh/id_ed25519
tasks:
- name: Ansible user exists
ansible.builtin.user:
name: ansible
comment: Ansible
shell: /bin/bash
system: true
create_home: true
- name: Only the control-node keys may log in
ansible.posix.authorized_key:
user: ansible
key: "{{ ansible_authorized_keys | join('\n') }}"
exclusive: true
# The dnf module needs it, and Fedora Server does not ship it
- name: dnf bindings for Ansible
ansible.builtin.command: dnf install -y python3-libdnf5
args:
creates: /usr/lib64/python3*/site-packages/libdnf5
when: ansible_facts.pkg_mgr == 'dnf5'
- name: Passwordless sudo for ansible
ansible.builtin.copy:
dest: /etc/sudoers.d/ansible
content: "ansible ALL=(ALL) NOPASSWD: ALL\n"
mode: "0440"
validate: visudo -cf %s
- name: Read effective sshd settings
ansible.builtin.command: sshd -T
register: sshd_effective
changed_when: false
# AllowUsers lines add up; only needed where such a list is in force
- name: Allow ansible over SSH where AllowUsers is set
ansible.builtin.copy:
dest: /etc/ssh/sshd_config.d/01-ansible.conf
content: "AllowUsers ansible\n"
mode: "0600"
validate: sshd -t -f %s
when: sshd_effective.stdout_lines | select('match', '^allowusers ') | list | length > 0
notify: Reload sshd
handlers:
- name: Reload sshd
ansible.builtin.service:
name: sshd
state: reloaded
ansible-playbook playbooks/bootstrap.yml -l vps1 # the admin has passwordless sudo
ansible-playbook playbooks/bootstrap.yml -l vps1 -K # sudo asks for a password
ansible vps1 -m ping
validate: is your friend for every file that can lock you out. Ansible writes the new file to a temporary path, runs the command on it, and installs it only if the command succeeds. A sudoers file with a typo can no longer break sudo.
site.yml
# playbooks/site.yml
- name: Fedora hosts
hosts: fedora
roles:
- {role: base, tags: base}
- {role: podman, tags: [podman, apps, backup]}
- {role: quadlet_app, tags: apps}
- {role: monitoring_agent, tags: monitoring}
- {role: backup, tags: backup}
- name: Mail server
hosts: mail:&fedora
roles:
- {role: mailcow, tags: [docker, mailcow]}
mail:&fedora is the intersection of the two groups. The mail host gets its Mailcow role only after it runs Fedora.
I always run it like this:
ansible-playbook playbooks/site.yml -l vps1 --check # what would change
ansible-playbook playbooks/site.yml -l vps1
ansible-playbook playbooks/site.yml -l vps1 -t base # only one part
The base role
The base role does the same things on every host. It starts with two checks: the host runs Fedora, and SELinux is enforcing (see the previous post). Then it does the actual work.
sshd
One drop-in file. Fedora’s sshd_config reads /etc/ssh/sshd_config.d/*.conf first, and for most options the first value wins. So a file called 00-hardening.conf overrides the distribution’s defaults in 50-redhat.conf.
# /etc/ssh/sshd_config.d/00-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
GSSAPIAuthentication no
{% if ssh_allow_users %}
AllowUsers {{ (ssh_allow_users + ['ansible']) | unique | join(' ') }}
{% endif %}
# Ed25519 only; post-quantum key exchange first; AEAD ciphers and ETM MACs only
HostKey /etc/ssh/ssh_host_ed25519_key
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
PubkeyAcceptedAlgorithms ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com
KexAlgorithms mlkem768x25519-sha256,sntrup761x25519-sha512,curve25519-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com
X11Forwarding no
AllowAgentForwarding no
AllowStreamLocalForwarding no
PermitTunnel no
AllowTcpForwarding no
MaxAuthTries 3
LoginGraceTime 20
LogLevel VERBOSE
- name: SSH hardening
ansible.builtin.template:
src: sshd-hardening.conf.j2
dest: /etc/ssh/sshd_config.d/00-hardening.conf
mode: "0644"
validate: sshd -t -f %s
notify: Reload sshd
Do not put Match blocks into such a drop-in. A Match block lasts until the end of the whole configuration, so it would also catch the rest of the main sshd_config.
The AllowUsers line always adds ansible. The first version did not. On a fresh test host, the first base run then locked the ansible user out of the host. I did not notice at first, because Ansible kept using its open SSH connection. More about that below.
Check what sshd really uses with sudo sshd -T | grep -i allowusers.
fail2ban
# /etc/fail2ban/jail.d/sshd.local
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1
[sshd]
enabled = true
backend = systemd
# aggressive: also scanners that disconnect before auth or fail the key exchange
mode = aggressive
maxretry = 3
findtime = 10m
bantime = 1h
bantime.increment = true
bantime.maxtime = 1w
[recidive]
enabled = true
bantime = 1w
findtime = 1d
maxretry = 5
The recidive jail reads fail2ban’s own log file. On a fresh host that file does not exist yet, and the jail, and with it fail2ban, fails to start. So the role creates the file first:
- name: fail2ban log file for the recidive jail
ansible.builtin.file:
path: /var/log/fail2ban.log
state: touch
mode: "0600"
setype: fail2ban_log_t
modification_time: preserve
access_time: preserve
modification_time: preserve keeps the task idempotent. A plain state: touch reports a change on every run.
firewalld
Only what the host lists in firewall_public_services is open in the default zone:
- name: Open services in the default zone
ansible.posix.firewalld:
service: "{{ item }}"
permanent: true
immediate: true
state: enabled
loop: "{{ firewall_public_services }}"
permanent and immediate together write the permanent config and apply it at once. No reload is needed, and a reboot changes nothing.
Updates every night
- name: dnf5-automatic settings
community.general.ini_file:
path: /etc/dnf/automatic.conf
section: "{{ item.section }}"
option: "{{ item.option }}"
value: "{{ item.value }}"
mode: "0644"
loop:
- {section: commands, option: apply_updates, value: "yes"}
- {section: commands, option: download_updates, value: "yes"}
- {section: commands, option: upgrade_type, value: default}
- {section: commands, option: reboot, value: "{{ dnf_automatic_reboot }}"}
- {section: commands, option: reboot_command, value: "\"shutdown -r +5 'Rebooting after applying automatic updates'\""}
- {section: emitters, option: emit_via, value: stdio}
- name: dnf5-automatic timer
ansible.builtin.systemd_service:
name: dnf5-automatic.timer
enabled: true
state: started
reboot is when-needed on the VPS (a reboot after a new kernel or glibc) and never on the home server, which I reboot myself.
A cap for the journal
- name: Journal size cap
ansible.builtin.copy:
dest: /etc/systemd/journald.conf.d/00-size.conf
content: |
[Journal]
SystemMaxUse=500M
mode: "0644"
notify: Restart journald
Fedora’s default is 10% of the filesystem, up to 4 GB. On a small VPS with chatty containers that is a lot. The old Ubuntu mail host had 2.3 GB of journal when its root filesystem reached 91%.
Things to watch
- Ansible reuses its SSH connections (
ControlPersist). After a change to sshd, a run that passes may still use the old connection. Close the connections before you test such a change:for s in ~/.ansible/cp/*; do ssh -O exit -o ControlPath="$s" x 2>/dev/null; done - Update a fresh host and reboot it before the first full run. Otherwise dnf5-automatic installs a new kernel during the rollout and schedules a reboot in five minutes. From then on,
pam_nologinlets only root log in, and the run fails halfway. --checkmode has limits. A task that depends on a package installed earlier in the same run fails in check mode, because the package is not there yet. Usewhen: not ansible_check_modefor such tasks, or accept the error on a host’s very first check run.
Check
ansible-playbook playbooks/site.yml -l vps1 -t base
ansible-playbook playbooks/site.yml -l vps1 -t base # the second run: changed=0
ssh -o ControlMaster=no -o ControlPath=none ansible@vps1.example.org true && echo ok
Next: One generic Ansible role for every app.
Further reading
- Ansible tips and tricks: the official list of good practices.
- Sample Ansible setup: a directory layout for inventories, group_vars and roles.
- Good practices for Ansible (Red Hat CoP): role design, naming and variables. Opinionated, and worth reading.
- How to build your inventory: groups, host_vars and group_vars.
- community.general.passwordstore lookup: secrets from pass, with all options.
- ansible-lint: catches many mistakes before a run.
- firewalld documentation: zones, services and rich rules.
- fail2ban: jails and filters.
- dnf5 automatic: the settings in
automatic.conf.