Linux Lunatix

Traefik in front of rootless containers

Stefan Marx
| 7 minutes

Part 3 of the series that starts with Four servers, three distributions: why I consolidated.

At the end of this post Traefik runs as a rootless quadlet. It gets Let’s Encrypt certificates and finds the apps by their container labels, as it did with Docker. You will also know which network mode to choose, and why.

The problem with rootless networking

With Docker, Traefik sits on a bridge network with the apps, and the root daemon forwards ports 80 and 443 to it. Traefik sees the real client address.

Rootless Podman cannot create real bridges in the host’s network namespace. It runs the container network in a user namespace. A helper process (rootlessport, or pasta) accepts connections on the host’s port and forwards them into the namespace. On the way the client address gets lost: Traefik then sees every request as coming from the helper. Access logs, rate limits, IP allow lists and fail2ban on the proxy then do not work.

There are two ways to deal with this:

ModeClient IPApps reached atUse it when
Traefik on the host networkreal127.0.0.1:<port> published by each appthe host is on the internet
Traefik on a Podman bridgelostthe container IP on the bridgeonly you and your LAN use it

I use both. My two internet-facing VPS run Traefik on the host network. My home server is reachable only from the LAN and through WireGuard, so the client addresses there are my own devices anyway. That host keeps the bridge network, which was there first and needs fewer changes.

Option A: Traefik on the host network

The quadlet

# ~svc/.config/containers/systemd/traefik.container
[Unit]
Description=Traefik reverse proxy
After=network-online.target

[Container]
Image=docker.io/library/traefik:v3
AutoUpdate=registry
ContainerName=traefik
# Host network namespace: real client IPs, binds 80/443 directly
Network=host
# Run as svc itself, so the files it writes belong to svc on the host
UserNS=keep-id:uid=1001,gid=1001
User=1001:1001
Volume=%h/traefik/acme:/acme:Z
Volume=%h/.config/traefik/traefik.yml:/etc/traefik/traefik.yml:ro,Z
Volume=%h/.config/traefik/dynamic:/dynamic:ro,Z
# The rootless Podman socket, read-only, as Traefik's "docker" provider
Volume=%t/podman/podman.sock:/var/run/docker.sock:ro,z
Secret=dns-api-token,type=env,target=CF_DNS_API_TOKEN

[Service]
Restart=always

[Install]
WantedBy=default.target

Replace 1001 with the UID and GID of your app user (id svc). Some notes:

The static configuration

# ~svc/.config/traefik/traefik.yml
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"

# On the host network, ClientHost in this log is the real client address.
# Written to stdout, so it ends up in the journal.
accessLog:
  format: json
  fields:
    headers:
      defaultMode: drop
      names:
        User-Agent: keep
        Referer: keep

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    # Apps publish their port on 127.0.0.1. Route to that address, not to the
    # container IP, which the host network namespace cannot reach.
    useBindPortIP: true
  file:
    directory: /dynamic
    watch: true

certificatesResolvers:
  default:
    acme:
      email: "admin@example.org"
      storage: /acme/acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"

useBindPortIP: true is the key line for this mode. Each app publishes its port on localhost (PublishPort=127.0.0.1:3000:3000). Traefik reads the port mapping from the container’s labels and metadata through the socket and connects to 127.0.0.1:3000. Each app needs its own host port.

The certificate challenge

Let’s Encrypt has to check that you control the domain. Traefik supports three ways:

acme.json holds the certificates and the ACME account key. Keep it in your backup. After a reinstall, restore it before Traefik starts. Traefik then serves the old certificates and asks Let’s Encrypt for nothing. Let’s Encrypt has rate limits, and a rehearsal or two reinstalls in a row can hit them.

Shared middlewares in a file

Things that many apps use, like security headers, go into the file provider:

# ~svc/.config/traefik/dynamic/dynamic_conf.yml
http:
  middlewares:
    secureHeaders:
      headers:
        browserXssFilter: true
        contentTypeNosniff: true
        frameDeny: true
        stsSeconds: 31536000

tls:
  options:
    default:
      minVersion: VersionTLS12

An app uses it with Label=traefik.http.routers.<name>.middlewares=secureHeaders@file.

An app behind it

# ~svc/.config/containers/systemd/blog.container
[Container]
ContainerName=blog
Image=git.example.org/alice/blog:latest
AutoUpdate=registry
PublishPort=127.0.0.1:8101:80
Label=traefik.enable=true
Label="traefik.http.routers.blog.rule=Host(`www.example.org`) || Host(`example.org`)"
Label=traefik.http.routers.blog.entrypoints=https
Label=traefik.http.routers.blog.tls.certresolver=default
Label=traefik.http.routers.blog.middlewares=secureHeaders@file
Label=traefik.http.services.blog.loadbalancer.server.port=80

[Service]
Restart=always

[Install]
WantedBy=default.target

Quote the whole Label= value if it contains spaces or ||. Otherwise systemd splits it.

Real client IPs in the app, too

Traefik sees the client, but the app behind it sees 127.0.0.1. For nginx, tell it to trust Traefik’s X-Forwarded-For header:

# realip.conf, mounted into the nginx container at /etc/nginx/conf.d/realip.conf
# Traefik replaces any X-Forwarded-For sent by clients. The app port is published
# on 127.0.0.1 only, so the only peer that can connect is the local Traefik.
set_real_ip_from 0.0.0.0/0;
set_real_ip_from ::/0;
real_ip_header X-Forwarded-For;

The wide set_real_ip_from looks wrong at first. It is safe here only because nothing but Traefik can reach the app’s port.

Option B: Traefik on a Podman bridge network

On the home server Traefik and the apps share a Podman network called proxy. Traefik publishes the ports, and the apps publish none.

# proxy.network
[Network]
NetworkName=proxy
# traefik.container (bridge mode, shortened)
[Container]
ContainerName=traefik
Image=docker.io/library/traefik:v3
AutoUpdate=registry
PublishPort=80:80/tcp
PublishPort=443:443/tcp
Volume=%t/podman/podman.sock:/run/podman/podman.sock:ro,z
Volume=%h/.config/traefik/traefik.yml:/etc/traefik/traefik.yml:ro,z
Volume=traefik-letsencrypt.volume:/letsencrypt:z
Secret=dns-api-token,type=env,target=CF_DNS_API_TOKEN
Network=proxy.network:ip=10.89.0.2

[Service]
Restart=always

[Install]
WantedBy=default.target

An app joins the network with Network=proxy.network:ip=10.89.0.20. I give each container a fixed IP, so that other configs (Prometheus targets, for example) can use it. In this mode the provider section of traefik.yml has no useBindPortIP, because Traefik reaches the container IPs directly.

Ports below 1024 are still needed for PublishPort=80:80, because the rootless helper binds them as the user.

Which one should you pick?

If anybody on the internet uses the service, choose the host network. You will want the client addresses the first time somebody hammers your login page. If only you and your devices use it, the bridge is simpler. The apps then need no host ports, and no app can be reached past Traefik on the host.

Check

systemctl --user daemon-reload
systemctl --user start traefik blog
journalctl --user -u traefik -f        # look for "certificate obtained"
curl -sI https://www.example.org/ | head -3
# Host network mode: the access log must show your real IP as ClientHost
journalctl --user -u traefik -n 5 -o cat | jq -r .ClientHost

Things to watch

Next: SELinux enforcing with containers.

Further reading

Previous: From Docker Compose to rootless Podman quadlets Next: SELinux enforcing with containers

Related content

One generic Ansible role for every app
SELinux enforcing with containers
From Docker Compose to rootless Podman quadlets