Linux Lunatix

One generic Ansible role for every app

Stefan Marx
| 9 minutes

Part 6 of the series that starts with Four servers, three distributions: why I consolidated.

My hosts run close to 30 apps, some of them with several containers. I did not want 30 Ansible roles. This post shows the one role that deploys all of them, and how I used it to take over running hosts without restarting anything.

The idea

Each app is a directory:

apps/forgejo/
├── app.yml                     # optional: secrets, directories, units
├── quadlets/                   # -> ~svc/.config/containers/systemd/
│   ├── forgejo.container
│   ├── forgejo-postgres.container
│   ├── forgejo.network
│   ├── forgejo-data.volume
│   └── forgejo-db.volume
├── files/                      # -> ~svc/ (same relative paths)
│   └── .config/forgejo/...
└── tasks.yml                   # optional: extra tasks (downloads, checkouts)

A host lists its apps:

# host_vars/vps1.yml
app_user: svc
apps: [traefik, ntfy, forgejo-runner]

The role quadlet_app loops over the list. For each app it does the same steps:

  1. Read app.yml.
  2. Create the Podman secrets it names, from pass.
  3. Copy files/ and quadlets/ into the app user’s home. Files that end in .j2 are templates.
  4. If anything changed: run daemon-reload and restart the app’s units.
  5. Make sure the units run.

An app that did not change is not touched. That last point matters most. A playbook run on a host with 20 apps restarts exactly the apps I changed.

Why plain files and not a module

containers.podman.podman_container can create containers from Ansible, and newer versions can write quadlets too. I chose plain quadlet files in the repository instead:

app.yml

Most apps need no app.yml at all. When one is there, it can say:

# apps/forgejo/app.yml
secrets:
  - {name: forgejo-db-password, pass: infra/<host>/forgejo-db-password}
# apps/traefik/app.yml
secrets:
  - {name: dns-api-token, pass: infra/<host>/dns-api-token, if: traefik_dns_challenge}
dirs:
  - {path: .config/traefik, mode: "0700"}
  - {path: traefik}
  - {path: traefik/acme, mode: "0700"}
KeyMeaning
secretsPodman secrets: name, pass entry (<host> is replaced), optional if (a host variable that must be true)
secret_filesfiles from files/ with mode 0600 that never show up in a diff
dirsdirectories to create, with an optional mode
unitsthe units to restart; default: one .service per .container
enableplain user units or timers to enable and start
restartrestart, or stop-start for containers that share a network namespace

The role, step by step

The role’s main.yml creates the quadlet directory once, then includes app.yml for each app:

# roles/quadlet_app/tasks/main.yml
- name: Quadlet directory
  ansible.builtin.file:
    path: "{{ app_home }}/{{ item }}"
    state: directory
    owner: "{{ app_user }}"
    group: "{{ app_user }}"
  loop: [.config, .config/containers, .config/containers/systemd]

- name: Apps
  ansible.builtin.include_tasks: app.yml
  loop: "{{ apps }}"
  loop_control:
    loop_var: app_name

app_home, app_uid and app_gid come from the podman role, which creates the user, enables linger and starts the user’s podman.socket and podman-auto-update.timer.

Read the definition and the source tree

- name: "{{ app_name }}: definition"
  ansible.builtin.set_fact:
    app: "{{ app_defaults | combine(app_file | from_yaml or {}) }}"
    app_src: "{{ apps_dir }}/{{ app_name }}"
  vars:
    app_file: "{{ lookup('ansible.builtin.file', apps_dir ~ '/' ~ app_name ~ '/app.yml', errors='ignore') | default('', true) }}"
    app_defaults: {secrets: [], secret_files: [], dirs: [], units: null, enable: [], restart: restart}

- name: "{{ app_name }}: source tree"
  ansible.builtin.set_fact:
    app_tree:
      files: "{{ lookup('community.general.filetree', app_src ~ '/files', wantlist=true) | selectattr('state', 'eq', 'file') | list }}"
      dirs: "{{ lookup('community.general.filetree', app_src ~ '/files', wantlist=true) | selectattr('state', 'eq', 'directory')
                | map(attribute='path') | map('community.general.dict_kv', 'path') | list }}"
      quadlets: "{{ lookup('community.general.filetree', app_src ~ '/quadlets', wantlist=true) | selectattr('state', 'eq', 'file') | list }}"

The filetree lookup walks a directory on the control node and returns every file with its relative path and mode. That is all the role needs to know about an app.

Secrets

- name: "{{ app_name }}: podman secrets"
  containers.podman.podman_secret:
    name: "{{ item.name }}"
    data: "{{ lookup('community.general.passwordstore', item.pass | replace('<host>', inventory_hostname)) }}"
    state: present
  loop: "{{ app.secrets }}"
  loop_control:
    label: "{{ item.name }}"
  when: item.if is not defined or lookup('ansible.builtin.vars', item.if) | bool
  become_user: "{{ app_user }}"
  environment:
    XDG_RUNTIME_DIR: /run/user/{{ app_uid }}
  no_log: true
  register: app_secrets_result

become_user plus XDG_RUNTIME_DIR is how every task in this role talks to the rootless user’s Podman and systemd. become_user to an unprivileged user needs the acl package on the host, so that Ansible can hand its temporary files to that user. The base role installs it.

Files and quadlets

Plain files are copied, and .j2 files are rendered. Executables stay executable. Secret files get mode 0600 and no diff:

- name: "{{ app_name }}: files"
  ansible.builtin.copy:
    src: "{{ item.src }}"
    dest: "{{ app_home }}/{{ item.path }}"
    owner: "{{ app_user }}"
    group: "{{ app_user }}"
    mode: "{{ '0600' if item.path in app.secret_files else ('0755' if item.mode[1] in '157' else '0644') }}"
  loop: "{{ app_tree.files | rejectattr('path', 'search', '\\.j2$') }}"
  loop_control:
    label: "{{ item.path }}"
  no_log: "{{ item.path in app.secret_files }}"
  register: app_files_result

- name: "{{ app_name }}: templated files"
  ansible.builtin.template:
    src: "{{ item.src }}"
    dest: "{{ app_home }}/{{ item.path | regex_replace('\\.j2$', '') }}"
    owner: "{{ app_user }}"
    group: "{{ app_user }}"
    mode: "0644"
  loop: "{{ app_tree.files | selectattr('path', 'search', '\\.j2$') }}"
  loop_control:
    label: "{{ item.path }}"
  register: app_templates_result

The quadlets work the same way, with .config/containers/systemd/ as the destination. Why are only .j2 files templates? Some apps ship files that contain their own {{ }} syntax: Grafana dashboards, Prometheus alert rules. Rendering those with Jinja would break them. So a file is a template only when its name says so.

A quadlet template is useful for values that differ per host:

# apps/traefik/quadlets/traefik.container.j2 (part)
UserNS=keep-id:uid={{ app_uid }},gid={{ app_gid }}
User={{ app_uid }}:{{ app_gid }}
{% for v in traefik_extra_volumes | default([]) %}
Volume={{ v }}
{% endfor %}
{% if traefik_dns_challenge %}
Secret=dns-api-token,type=env,target=CF_DNS_API_TOKEN
{% endif %}

Restart only what changed

Every task above registers its result. The role collects all of them and asks one question: did anything change for this app?

- name: "{{ app_name }}: units"
  ansible.builtin.set_fact:
    app_units: "{{ app.units if app.units is not none else
                   (app_tree.quadlets | map(attribute='path') | map('regex_replace', '\\.j2$', '')
                    | select('match', '.*\\.container$') | map('regex_replace', '\\.container$', '.service') | list) }}"
    app_changed: "{{ (app_secrets_result.results + app_files_result.results + app_templates_result.results
                      + app_quadlets_result.results + app_quadlet_templates_result.results
                      + (app_extra_results | default([]))) | selectattr('changed', 'defined') | selectattr('changed') | list | length > 0 }}"

- name: "{{ app_name }}: reload and restart"
  when: app_changed
  become_user: "{{ app_user }}"
  environment:
    XDG_RUNTIME_DIR: /run/user/{{ app_uid }}
  block:
    - name: "{{ app_name }}: daemon-reload"
      ansible.builtin.systemd_service:
        daemon_reload: true
        scope: user

    - name: "{{ app_name }}: restart"
      ansible.builtin.systemd_service:
        name: "{{ item }}"
        scope: user
        state: restarted
      loop: "{{ app_units }}"
      when: app.restart == 'restart'

    # Containers that share another's network namespace must be gone before it can be
    # replaced: stop in reverse order, then start in order
    - name: "{{ app_name }}: stop (reverse order)"
      ansible.builtin.systemd_service:
        name: "{{ item }}"
        scope: user
        state: stopped
      loop: "{{ app_units | reverse | list }}"
      when: app.restart == 'stop-start'

    - name: "{{ app_name }}: start"
      ansible.builtin.systemd_service:
        name: "{{ item }}"
        scope: user
        state: started
      loop: "{{ app_units }}"
      when: app.restart == 'stop-start'

- name: "{{ app_name }}: running"
  ansible.builtin.systemd_service:
    name: "{{ item }}"
    scope: user
    state: started
  loop: "{{ app_units }}"
  become_user: "{{ app_user }}"
  environment:
    XDG_RUNTIME_DIR: /run/user/{{ app_uid }}

Why not handlers? Handlers run at the end of the play, and they run once per handler name, however many apps notified it. I want the restart right after the app’s own files changed, and only for that app’s units. Handlers cannot carry “which app” without tricks. A plain when: app_changed is easier to read.

Restarting is also how a container sees a changed config file. Ansible writes a new file (a new inode), and a running container keeps the old one in its mount.

Extra tasks

Some apps need more than files: a Git checkout to build from, a binary to download. An optional tasks.yml in the app directory is included, and it can add its results to app_extra_results to trigger the restart:

- name: "{{ app_name }}: extra tasks"
  ansible.builtin.include_tasks: "{{ app_src }}/tasks.yml"
  when: (app_src ~ '/tasks.yml') is file

Order matters

Apps run in the order of the apps list. If one app provides a file that another app mounts, the provider must come first. On a fresh host, Traefik fails to start if a password file it mounts does not exist yet:

# the landing page app provides the htpasswd file that Traefik mounts
apps: [landing, traefik, ntfy, forgejo-runner]
traefik_extra_volumes:
  - "%h/.config/traefik/landing_htpasswd:/landing_htpasswd:ro,Z"

Adopting a running host

This is how I moved the existing quadlets of a running host into the role without a single restart:

  1. Copy the host’s quadlets and config files into apps/<name>/, one app at a time.
  2. Run check mode with a diff against the live host:
    ansible-playbook playbooks/site.yml -l home1 -t apps --check --diff
    
  3. Every difference is one of two things. Either the repository is wrong (fix the file), or the host has a change that was never written down (decide which version is right).
  4. Repeat until the run shows changed=0 in check mode.
  5. Run it for real. Nothing changes, so nothing restarts.

The loop also shows what nobody wrote down: an option changed by hand on the host, a file that exists only there. Each of those is now either in the repository or gone.

The test: changed=0

After every change I run the playbook twice. The second run must report changed=0. A task that reports a change every time hides the real changes, and it may restart an app on every run. Typical causes:

Run it with fresh SSH connections, too (-o ControlPath=none). The reason is in the previous post.

Check

ansible-playbook playbooks/site.yml -l vps1 -t apps
ansible-playbook playbooks/site.yml -l vps1 -t apps      # changed=0
ssh vps1 sudo -u svc XDG_RUNTIME_DIR=/run/user/1001 systemctl --user list-units 'traefik*' 'ntfy*'

Next: Monitoring and backups for every host.

Further reading

Previous: Ansible groundwork: inventory, bootstrap and a base role Next: Monitoring and backups for every host

Related content

Rehearse before you reinstall
Monitoring and backups for every host
Ansible groundwork: inventory, bootstrap and a base role