Linux Lunatix

Ansible groundwork: inventory, bootstrap and a base role

Stefan Marx
| 9 minutes

Part 5 of the series that starts with Four servers, three distributions: why I consolidated.

This post sets up the parts that every host shares. At the end you have a control node, an inventory, a playbook for the first contact with a host, and a base role. That role hardens sshd, runs firewalld and fail2ban, installs updates every night and caps the journal.

The control node

I run Ansible from my laptop, and from a second machine as a spare. Both use the same pinned versions in a virtualenv. The distribution’s Ansible package changes with each upgrade, and I want the same behaviour on both machines:

python3 -m venv ~/.local/share/ops-venv
~/.local/share/ops-venv/bin/pip install "ansible-core==2.22.0" ansible-lint yamllint
export PATH=~/.local/share/ops-venv/bin:$PATH

The collections are pinned too, and installed into the repository:

# requirements.yml
collections:
  - name: ansible.posix
    version: "2.2.2"
  - name: community.general
    version: "14.0.0"
  - name: containers.podman
    version: "1.21.2"
ansible-galaxy collection install -r requirements.yml -p collections

Each control node has its own key for Ansible, ~/.ssh/ansible_ed25519, separate from my login key. I can then revoke one machine without touching the other.

ansible.cfg

[defaults]
inventory = inventory/hosts.yml
roles_path = roles
collections_path = collections
remote_user = ansible
private_key_file = ~/.ssh/ansible_ed25519
interpreter_python = auto_silent
callback_result_format = yaml
forks = 4
# Handlers run even when a later task fails
force_handlers = True

[privilege_escalation]
become = True

[diff]
always = True

[ssh_connection]
pipelining = True

Two settings are worth an explanation.

force_handlers runs the notified handlers even if a later task fails. Without it, this can happen: a task changes the sshd config and notifies “Reload sshd”, and then a later task fails. The handler never runs. On the next run the config file is unchanged, so nothing notifies the handler again. sshd runs with the old config until somebody notices.

diff always shows every change as a diff. Together with --check this makes a run a review: I see what would change before it changes.

The inventory

# inventory/hosts.yml
all:
  hosts:
    mail1:
      ansible_host: mail1.example.org
      bootstrap_user: admin
      tunnel_ip: 10.99.0.11
    vps1:
      ansible_host: vps1.example.org
      bootstrap_user: fedora
      tunnel_ip: 10.99.0.13
      app_user: svc
    vps2:
      ansible_host: vps2.example.org
      bootstrap_user: fedora
      tunnel_ip: 10.99.0.12
    home1:
      ansible_host: home1.example.net
      bootstrap_user: admin
      tunnel_ip: 10.99.0.1
      app_user: svc
  children:
    vps:
      hosts: {mail1: , vps1: , vps2: }
    home:
      hosts: {home1: }
    mail:
      hosts: {mail1: }
    websites:
      hosts: {vps2: }
    # The hosts that site.yml manages. A host joins after its reinstall with Fedora.
    fedora:
      hosts: {vps1: , home1: , vps2: }

The fedora group is how I adopted the hosts one by one. A host that still ran Ubuntu or Arch was in the inventory, but site.yml did not touch it. It joined the group after its reinstall.

Everything else goes into group_vars/ and host_vars/. A host file says what is special about that host:

# inventory/host_vars/vps1.yml
hostname: vps1.example.org
timezone: Europe/Berlin
ssh_allow_users: [fedora, svc]
firewall_public_services: [http, https, ssh]
dnf_automatic_reboot: when-needed
unprivileged_port_start: 80
apps: [traefik, ntfy, forgejo-runner]

Secrets from pass

I keep all secrets in pass on the control node. The lookup plugin reads them during the run:

- name: Tunnel private key
  ansible.builtin.copy:
    dest: /etc/wireguard/wg-mon.key
    content: "{{ lookup('community.general.passwordstore', 'infra/' ~ inventory_hostname ~ '/wg-mon-key') }}\n"
    mode: "0600"
  no_log: true
  diff: false

no_log and diff: false keep the secret out of the output. Without diff: false, diff always would print the old and the new key.

Compared with Ansible Vault, the repository contains nothing encrypted at all, and there is no vault password to share between the control nodes. A secret is changed in one place, in pass, and the next run rolls it out.

First contact: bootstrap.yml

A fresh host has one admin account that the provider set up (fedora on a Fedora Cloud image). bootstrap.yml logs in as that account once and creates the ansible user. Every other playbook then uses that user.

# playbooks/bootstrap.yml
- name: Bootstrap the ansible user
  hosts: all
  remote_user: "{{ bootstrap_user }}"
  vars:
    ansible_ssh_private_key_file: ~/.ssh/id_ed25519
  tasks:
    - name: Ansible user exists
      ansible.builtin.user:
        name: ansible
        comment: Ansible
        shell: /bin/bash
        system: true
        create_home: true

    - name: Only the control-node keys may log in
      ansible.posix.authorized_key:
        user: ansible
        key: "{{ ansible_authorized_keys | join('\n') }}"
        exclusive: true

    # The dnf module needs it, and Fedora Server does not ship it
    - name: dnf bindings for Ansible
      ansible.builtin.command: dnf install -y python3-libdnf5
      args:
        creates: /usr/lib64/python3*/site-packages/libdnf5
      when: ansible_facts.pkg_mgr == 'dnf5'

    - name: Passwordless sudo for ansible
      ansible.builtin.copy:
        dest: /etc/sudoers.d/ansible
        content: "ansible ALL=(ALL) NOPASSWD: ALL\n"
        mode: "0440"
        validate: visudo -cf %s

    - name: Read effective sshd settings
      ansible.builtin.command: sshd -T
      register: sshd_effective
      changed_when: false

    # AllowUsers lines add up; only needed where such a list is in force
    - name: Allow ansible over SSH where AllowUsers is set
      ansible.builtin.copy:
        dest: /etc/ssh/sshd_config.d/01-ansible.conf
        content: "AllowUsers ansible\n"
        mode: "0600"
        validate: sshd -t -f %s
      when: sshd_effective.stdout_lines | select('match', '^allowusers ') | list | length > 0
      notify: Reload sshd

  handlers:
    - name: Reload sshd
      ansible.builtin.service:
        name: sshd
        state: reloaded
ansible-playbook playbooks/bootstrap.yml -l vps1        # the admin has passwordless sudo
ansible-playbook playbooks/bootstrap.yml -l vps1 -K     # sudo asks for a password
ansible vps1 -m ping

validate: is your friend for every file that can lock you out. Ansible writes the new file to a temporary path, runs the command on it, and installs it only if the command succeeds. A sudoers file with a typo can no longer break sudo.

site.yml

# playbooks/site.yml
- name: Fedora hosts
  hosts: fedora
  roles:
    - {role: base, tags: base}
    - {role: podman, tags: [podman, apps, backup]}
    - {role: quadlet_app, tags: apps}
    - {role: monitoring_agent, tags: monitoring}
    - {role: backup, tags: backup}

- name: Mail server
  hosts: mail:&fedora
  roles:
    - {role: mailcow, tags: [docker, mailcow]}

mail:&fedora is the intersection of the two groups. The mail host gets its Mailcow role only after it runs Fedora.

I always run it like this:

ansible-playbook playbooks/site.yml -l vps1 --check     # what would change
ansible-playbook playbooks/site.yml -l vps1
ansible-playbook playbooks/site.yml -l vps1 -t base     # only one part

The base role

The base role does the same things on every host. It starts with two checks: the host runs Fedora, and SELinux is enforcing (see the previous post). Then it does the actual work.

sshd

One drop-in file. Fedora’s sshd_config reads /etc/ssh/sshd_config.d/*.conf first, and for most options the first value wins. So a file called 00-hardening.conf overrides the distribution’s defaults in 50-redhat.conf.

# /etc/ssh/sshd_config.d/00-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
GSSAPIAuthentication no
{% if ssh_allow_users %}
AllowUsers {{ (ssh_allow_users + ['ansible']) | unique | join(' ') }}
{% endif %}

# Ed25519 only; post-quantum key exchange first; AEAD ciphers and ETM MACs only
HostKey /etc/ssh/ssh_host_ed25519_key
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
PubkeyAcceptedAlgorithms ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com
KexAlgorithms mlkem768x25519-sha256,sntrup761x25519-sha512,curve25519-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

X11Forwarding no
AllowAgentForwarding no
AllowStreamLocalForwarding no
PermitTunnel no
AllowTcpForwarding no
MaxAuthTries 3
LoginGraceTime 20
LogLevel VERBOSE
- name: SSH hardening
  ansible.builtin.template:
    src: sshd-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/00-hardening.conf
    mode: "0644"
    validate: sshd -t -f %s
  notify: Reload sshd

Do not put Match blocks into such a drop-in. A Match block lasts until the end of the whole configuration, so it would also catch the rest of the main sshd_config.

The AllowUsers line always adds ansible. The first version did not. On a fresh test host, the first base run then locked the ansible user out of the host. I did not notice at first, because Ansible kept using its open SSH connection. More about that below.

Check what sshd really uses with sudo sshd -T | grep -i allowusers.

fail2ban

# /etc/fail2ban/jail.d/sshd.local
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1

[sshd]
enabled = true
backend = systemd
# aggressive: also scanners that disconnect before auth or fail the key exchange
mode = aggressive
maxretry = 3
findtime = 10m
bantime = 1h
bantime.increment = true
bantime.maxtime = 1w

[recidive]
enabled = true
bantime = 1w
findtime = 1d
maxretry = 5

The recidive jail reads fail2ban’s own log file. On a fresh host that file does not exist yet, and the jail, and with it fail2ban, fails to start. So the role creates the file first:

- name: fail2ban log file for the recidive jail
  ansible.builtin.file:
    path: /var/log/fail2ban.log
    state: touch
    mode: "0600"
    setype: fail2ban_log_t
    modification_time: preserve
    access_time: preserve

modification_time: preserve keeps the task idempotent. A plain state: touch reports a change on every run.

firewalld

Only what the host lists in firewall_public_services is open in the default zone:

- name: Open services in the default zone
  ansible.posix.firewalld:
    service: "{{ item }}"
    permanent: true
    immediate: true
    state: enabled
  loop: "{{ firewall_public_services }}"

permanent and immediate together write the permanent config and apply it at once. No reload is needed, and a reboot changes nothing.

Updates every night

- name: dnf5-automatic settings
  community.general.ini_file:
    path: /etc/dnf/automatic.conf
    section: "{{ item.section }}"
    option: "{{ item.option }}"
    value: "{{ item.value }}"
    mode: "0644"
  loop:
    - {section: commands, option: apply_updates, value: "yes"}
    - {section: commands, option: download_updates, value: "yes"}
    - {section: commands, option: upgrade_type, value: default}
    - {section: commands, option: reboot, value: "{{ dnf_automatic_reboot }}"}
    - {section: commands, option: reboot_command, value: "\"shutdown -r +5 'Rebooting after applying automatic updates'\""}
    - {section: emitters, option: emit_via, value: stdio}

- name: dnf5-automatic timer
  ansible.builtin.systemd_service:
    name: dnf5-automatic.timer
    enabled: true
    state: started

reboot is when-needed on the VPS (a reboot after a new kernel or glibc) and never on the home server, which I reboot myself.

A cap for the journal

- name: Journal size cap
  ansible.builtin.copy:
    dest: /etc/systemd/journald.conf.d/00-size.conf
    content: |
      [Journal]
      SystemMaxUse=500M
    mode: "0644"
  notify: Restart journald

Fedora’s default is 10% of the filesystem, up to 4 GB. On a small VPS with chatty containers that is a lot. The old Ubuntu mail host had 2.3 GB of journal when its root filesystem reached 91%.

Things to watch

Check

ansible-playbook playbooks/site.yml -l vps1 -t base
ansible-playbook playbooks/site.yml -l vps1 -t base   # the second run: changed=0
ssh -o ControlMaster=no -o ControlPath=none ansible@vps1.example.org true && echo ok

Next: One generic Ansible role for every app.

Further reading

Previous: SELinux enforcing with containers Next: One generic Ansible role for every app

Related content

Rehearse before you reinstall
Monitoring and backups for every host
One generic Ansible role for every app