One generic Ansible role for every app
Part 6 of the series that starts with Four servers, three distributions: why I consolidated.
My hosts run close to 30 apps, some of them with several containers. I did not want 30 Ansible roles. This post shows the one role that deploys all of them, and how I used it to take over running hosts without restarting anything.
The idea
Each app is a directory:
apps/forgejo/
├── app.yml # optional: secrets, directories, units
├── quadlets/ # -> ~svc/.config/containers/systemd/
│ ├── forgejo.container
│ ├── forgejo-postgres.container
│ ├── forgejo.network
│ ├── forgejo-data.volume
│ └── forgejo-db.volume
├── files/ # -> ~svc/ (same relative paths)
│ └── .config/forgejo/...
└── tasks.yml # optional: extra tasks (downloads, checkouts)
A host lists its apps:
# host_vars/vps1.yml
app_user: svc
apps: [traefik, ntfy, forgejo-runner]
The role quadlet_app loops over the list. For each app it does the same steps:
- Read
app.yml. - Create the Podman secrets it names, from pass.
- Copy
files/andquadlets/into the app user’s home. Files that end in.j2are templates. - If anything changed: run
daemon-reloadand restart the app’s units. - Make sure the units run.
An app that did not change is not touched. That last point matters most. A playbook run on a host with 20 apps restarts exactly the apps I changed.
Why plain files and not a module
containers.podman.podman_container can create containers from Ansible, and newer versions can write quadlets too. I chose plain quadlet files in the repository instead:
- They are the same files that are on the host.
diffshows exactly what changes. - I can copy one to a test machine and run it without Ansible.
- The Podman documentation for quadlets applies 1:1. There is no translation layer whose options I must look up.
app.yml
Most apps need no app.yml at all. When one is there, it can say:
# apps/forgejo/app.yml
secrets:
- {name: forgejo-db-password, pass: infra/<host>/forgejo-db-password}
# apps/traefik/app.yml
secrets:
- {name: dns-api-token, pass: infra/<host>/dns-api-token, if: traefik_dns_challenge}
dirs:
- {path: .config/traefik, mode: "0700"}
- {path: traefik}
- {path: traefik/acme, mode: "0700"}
| Key | Meaning |
|---|---|
secrets | Podman secrets: name, pass entry (<host> is replaced), optional if (a host variable that must be true) |
secret_files | files from files/ with mode 0600 that never show up in a diff |
dirs | directories to create, with an optional mode |
units | the units to restart; default: one .service per .container |
enable | plain user units or timers to enable and start |
restart | restart, or stop-start for containers that share a network namespace |
The role, step by step
The role’s main.yml creates the quadlet directory once, then includes app.yml for each app:
# roles/quadlet_app/tasks/main.yml
- name: Quadlet directory
ansible.builtin.file:
path: "{{ app_home }}/{{ item }}"
state: directory
owner: "{{ app_user }}"
group: "{{ app_user }}"
loop: [.config, .config/containers, .config/containers/systemd]
- name: Apps
ansible.builtin.include_tasks: app.yml
loop: "{{ apps }}"
loop_control:
loop_var: app_name
app_home, app_uid and app_gid come from the podman role, which creates the user, enables linger and starts the user’s podman.socket and podman-auto-update.timer.
Read the definition and the source tree
- name: "{{ app_name }}: definition"
ansible.builtin.set_fact:
app: "{{ app_defaults | combine(app_file | from_yaml or {}) }}"
app_src: "{{ apps_dir }}/{{ app_name }}"
vars:
app_file: "{{ lookup('ansible.builtin.file', apps_dir ~ '/' ~ app_name ~ '/app.yml', errors='ignore') | default('', true) }}"
app_defaults: {secrets: [], secret_files: [], dirs: [], units: null, enable: [], restart: restart}
- name: "{{ app_name }}: source tree"
ansible.builtin.set_fact:
app_tree:
files: "{{ lookup('community.general.filetree', app_src ~ '/files', wantlist=true) | selectattr('state', 'eq', 'file') | list }}"
dirs: "{{ lookup('community.general.filetree', app_src ~ '/files', wantlist=true) | selectattr('state', 'eq', 'directory')
| map(attribute='path') | map('community.general.dict_kv', 'path') | list }}"
quadlets: "{{ lookup('community.general.filetree', app_src ~ '/quadlets', wantlist=true) | selectattr('state', 'eq', 'file') | list }}"
The filetree lookup walks a directory on the control node and returns every file with its relative path and mode. That is all the role needs to know about an app.
Secrets
- name: "{{ app_name }}: podman secrets"
containers.podman.podman_secret:
name: "{{ item.name }}"
data: "{{ lookup('community.general.passwordstore', item.pass | replace('<host>', inventory_hostname)) }}"
state: present
loop: "{{ app.secrets }}"
loop_control:
label: "{{ item.name }}"
when: item.if is not defined or lookup('ansible.builtin.vars', item.if) | bool
become_user: "{{ app_user }}"
environment:
XDG_RUNTIME_DIR: /run/user/{{ app_uid }}
no_log: true
register: app_secrets_result
become_user plus XDG_RUNTIME_DIR is how every task in this role talks to the rootless user’s Podman and systemd. become_user to an unprivileged user needs the acl package on the host, so that Ansible can hand its temporary files to that user. The base role installs it.
Files and quadlets
Plain files are copied, and .j2 files are rendered. Executables stay executable. Secret files get mode 0600 and no diff:
- name: "{{ app_name }}: files"
ansible.builtin.copy:
src: "{{ item.src }}"
dest: "{{ app_home }}/{{ item.path }}"
owner: "{{ app_user }}"
group: "{{ app_user }}"
mode: "{{ '0600' if item.path in app.secret_files else ('0755' if item.mode[1] in '157' else '0644') }}"
loop: "{{ app_tree.files | rejectattr('path', 'search', '\\.j2$') }}"
loop_control:
label: "{{ item.path }}"
no_log: "{{ item.path in app.secret_files }}"
register: app_files_result
- name: "{{ app_name }}: templated files"
ansible.builtin.template:
src: "{{ item.src }}"
dest: "{{ app_home }}/{{ item.path | regex_replace('\\.j2$', '') }}"
owner: "{{ app_user }}"
group: "{{ app_user }}"
mode: "0644"
loop: "{{ app_tree.files | selectattr('path', 'search', '\\.j2$') }}"
loop_control:
label: "{{ item.path }}"
register: app_templates_result
The quadlets work the same way, with .config/containers/systemd/ as the destination. Why are only .j2 files templates? Some apps ship files that contain their own {{ }} syntax: Grafana dashboards, Prometheus alert rules. Rendering those with Jinja would break them. So a file is a template only when its name says so.
A quadlet template is useful for values that differ per host:
# apps/traefik/quadlets/traefik.container.j2 (part)
UserNS=keep-id:uid={{ app_uid }},gid={{ app_gid }}
User={{ app_uid }}:{{ app_gid }}
{% for v in traefik_extra_volumes | default([]) %}
Volume={{ v }}
{% endfor %}
{% if traefik_dns_challenge %}
Secret=dns-api-token,type=env,target=CF_DNS_API_TOKEN
{% endif %}
Restart only what changed
Every task above registers its result. The role collects all of them and asks one question: did anything change for this app?
- name: "{{ app_name }}: units"
ansible.builtin.set_fact:
app_units: "{{ app.units if app.units is not none else
(app_tree.quadlets | map(attribute='path') | map('regex_replace', '\\.j2$', '')
| select('match', '.*\\.container$') | map('regex_replace', '\\.container$', '.service') | list) }}"
app_changed: "{{ (app_secrets_result.results + app_files_result.results + app_templates_result.results
+ app_quadlets_result.results + app_quadlet_templates_result.results
+ (app_extra_results | default([]))) | selectattr('changed', 'defined') | selectattr('changed') | list | length > 0 }}"
- name: "{{ app_name }}: reload and restart"
when: app_changed
become_user: "{{ app_user }}"
environment:
XDG_RUNTIME_DIR: /run/user/{{ app_uid }}
block:
- name: "{{ app_name }}: daemon-reload"
ansible.builtin.systemd_service:
daemon_reload: true
scope: user
- name: "{{ app_name }}: restart"
ansible.builtin.systemd_service:
name: "{{ item }}"
scope: user
state: restarted
loop: "{{ app_units }}"
when: app.restart == 'restart'
# Containers that share another's network namespace must be gone before it can be
# replaced: stop in reverse order, then start in order
- name: "{{ app_name }}: stop (reverse order)"
ansible.builtin.systemd_service:
name: "{{ item }}"
scope: user
state: stopped
loop: "{{ app_units | reverse | list }}"
when: app.restart == 'stop-start'
- name: "{{ app_name }}: start"
ansible.builtin.systemd_service:
name: "{{ item }}"
scope: user
state: started
loop: "{{ app_units }}"
when: app.restart == 'stop-start'
- name: "{{ app_name }}: running"
ansible.builtin.systemd_service:
name: "{{ item }}"
scope: user
state: started
loop: "{{ app_units }}"
become_user: "{{ app_user }}"
environment:
XDG_RUNTIME_DIR: /run/user/{{ app_uid }}
Why not handlers? Handlers run at the end of the play, and they run once per handler name, however many apps notified it. I want the restart right after the app’s own files changed, and only for that app’s units. Handlers cannot carry “which app” without tricks. A plain when: app_changed is easier to read.
Restarting is also how a container sees a changed config file. Ansible writes a new file (a new inode), and a running container keeps the old one in its mount.
Extra tasks
Some apps need more than files: a Git checkout to build from, a binary to download. An optional tasks.yml in the app directory is included, and it can add its results to app_extra_results to trigger the restart:
- name: "{{ app_name }}: extra tasks"
ansible.builtin.include_tasks: "{{ app_src }}/tasks.yml"
when: (app_src ~ '/tasks.yml') is file
Order matters
Apps run in the order of the apps list. If one app provides a file that another app mounts, the provider must come first. On a fresh host, Traefik fails to start if a password file it mounts does not exist yet:
# the landing page app provides the htpasswd file that Traefik mounts
apps: [landing, traefik, ntfy, forgejo-runner]
traefik_extra_volumes:
- "%h/.config/traefik/landing_htpasswd:/landing_htpasswd:ro,Z"
Adopting a running host
This is how I moved the existing quadlets of a running host into the role without a single restart:
- Copy the host’s quadlets and config files into
apps/<name>/, one app at a time. - Run check mode with a diff against the live host:
ansible-playbook playbooks/site.yml -l home1 -t apps --check --diff - Every difference is one of two things. Either the repository is wrong (fix the file), or the host has a change that was never written down (decide which version is right).
- Repeat until the run shows
changed=0in check mode. - Run it for real. Nothing changes, so nothing restarts.
The loop also shows what nobody wrote down: an option changed by hand on the host, a file that exists only there. Each of those is now either in the repository or gone.
The test: changed=0
After every change I run the playbook twice. The second run must report changed=0. A task that reports a change every time hides the real changes, and it may restart an app on every run. Typical causes:
commandorshelltasks withoutchanged_when.file: state=touchwithoutmodification_time: preserve.- A template that renders a timestamp or a random value.
- A module that compares in a different format than it writes (for example YAML key order).
Run it with fresh SSH connections, too (-o ControlPath=none). The reason is in the previous post.
Check
ansible-playbook playbooks/site.yml -l vps1 -t apps
ansible-playbook playbooks/site.yml -l vps1 -t apps # changed=0
ssh vps1 sudo -u svc XDG_RUNTIME_DIR=/run/user/1001 systemctl --user list-units 'traefik*' 'ntfy*'
Next: Monitoring and backups for every host.
Further reading
- Ansible roles: the role layout, defaults and dependencies.
- Handlers: notify, listen and
flush_handlers. - Check mode and diff mode: what
--check --diffcan and cannot predict. - containers.podman collection: the modules for secrets, images and containers. The alternative to plain files.
- podman-systemd.unit(5): the quadlet reference again, because the role only copies these files.
- Good practices for Ansible (Red Hat CoP): the chapter on roles fits this post.