Traefik in front of rootless containers
Part 3 of the series that starts with Four servers, three distributions: why I consolidated.
At the end of this post Traefik runs as a rootless quadlet. It gets Let’s Encrypt certificates and finds the apps by their container labels, as it did with Docker. You will also know which network mode to choose, and why.
The problem with rootless networking
With Docker, Traefik sits on a bridge network with the apps, and the root daemon forwards ports 80 and 443 to it. Traefik sees the real client address.
Rootless Podman cannot create real bridges in the host’s network namespace. It runs the container network in a user namespace. A helper process (rootlessport, or pasta) accepts connections on the host’s port and forwards them into the namespace. On the way the client address gets lost: Traefik then sees every request as coming from the helper. Access logs, rate limits, IP allow lists and fail2ban on the proxy then do not work.
There are two ways to deal with this:
| Mode | Client IP | Apps reached at | Use it when |
|---|---|---|---|
| Traefik on the host network | real | 127.0.0.1:<port> published by each app | the host is on the internet |
| Traefik on a Podman bridge | lost | the container IP on the bridge | only you and your LAN use it |
I use both. My two internet-facing VPS run Traefik on the host network. My home server is reachable only from the LAN and through WireGuard, so the client addresses there are my own devices anyway. That host keeps the bridge network, which was there first and needs fewer changes.
Option A: Traefik on the host network
The quadlet
# ~svc/.config/containers/systemd/traefik.container
[Unit]
Description=Traefik reverse proxy
After=network-online.target
[Container]
Image=docker.io/library/traefik:v3
AutoUpdate=registry
ContainerName=traefik
# Host network namespace: real client IPs, binds 80/443 directly
Network=host
# Run as svc itself, so the files it writes belong to svc on the host
UserNS=keep-id:uid=1001,gid=1001
User=1001:1001
Volume=%h/traefik/acme:/acme:Z
Volume=%h/.config/traefik/traefik.yml:/etc/traefik/traefik.yml:ro,Z
Volume=%h/.config/traefik/dynamic:/dynamic:ro,Z
# The rootless Podman socket, read-only, as Traefik's "docker" provider
Volume=%t/podman/podman.sock:/var/run/docker.sock:ro,z
Secret=dns-api-token,type=env,target=CF_DNS_API_TOKEN
[Service]
Restart=always
[Install]
WantedBy=default.target
Replace 1001 with the UID and GID of your app user (id svc). Some notes:
Network=hostneedsnet.ipv4.ip_unprivileged_port_start=80(see the previous post). Without it Traefik cannot bind 80 and 443.%his the user’s home directory and%tis$XDG_RUNTIME_DIR. These are systemd specifiers, and quadlets expand them.- The Podman socket must be enabled for the user:
systemctl --user enable --now podman.socket. - The socket mount needs an SELinux module, so that a confined container may connect to it. Post 4 shows it. Without the module Traefik logs “permission denied” on the socket.
The static configuration
# ~svc/.config/traefik/traefik.yml
entryPoints:
http:
address: ":80"
http:
redirections:
entryPoint:
to: https
scheme: https
https:
address: ":443"
# On the host network, ClientHost in this log is the real client address.
# Written to stdout, so it ends up in the journal.
accessLog:
format: json
fields:
headers:
defaultMode: drop
names:
User-Agent: keep
Referer: keep
providers:
docker:
endpoint: "unix:///var/run/docker.sock"
exposedByDefault: false
# Apps publish their port on 127.0.0.1. Route to that address, not to the
# container IP, which the host network namespace cannot reach.
useBindPortIP: true
file:
directory: /dynamic
watch: true
certificatesResolvers:
default:
acme:
email: "admin@example.org"
storage: /acme/acme.json
dnsChallenge:
provider: cloudflare
resolvers:
- "1.1.1.1:53"
- "8.8.8.8:53"
useBindPortIP: true is the key line for this mode. Each app publishes its port on localhost (PublishPort=127.0.0.1:3000:3000). Traefik reads the port mapping from the container’s labels and metadata through the socket and connects to 127.0.0.1:3000. Each app needs its own host port.
The certificate challenge
Let’s Encrypt has to check that you control the domain. Traefik supports three ways:
- The DNS challenge: Traefik writes a TXT record through your DNS provider’s API. It works for wildcard certificates and for hosts that are not reachable from the internet. It needs an API token, which I keep in a Podman secret (
Secret=dns-api-token,...). Use your provider’s name inprovider:. Traefik’s ACME library supports many providers, and each one has its own environment variable for the token. - The TLS challenge (
tlsChallenge: {}): Let’s Encrypt connects to port 443. No token is needed, but the host must be reachable from the internet. I use it for domains whose DNS is at a provider without an API. - The HTTP challenge: the same as the TLS challenge, but on port 80.
acme.json holds the certificates and the ACME account key. Keep it in your backup. After a reinstall, restore it before Traefik starts. Traefik then serves the old certificates and asks Let’s Encrypt for nothing. Let’s Encrypt has rate limits, and a rehearsal or two reinstalls in a row can hit them.
Shared middlewares in a file
Things that many apps use, like security headers, go into the file provider:
# ~svc/.config/traefik/dynamic/dynamic_conf.yml
http:
middlewares:
secureHeaders:
headers:
browserXssFilter: true
contentTypeNosniff: true
frameDeny: true
stsSeconds: 31536000
tls:
options:
default:
minVersion: VersionTLS12
An app uses it with Label=traefik.http.routers.<name>.middlewares=secureHeaders@file.
An app behind it
# ~svc/.config/containers/systemd/blog.container
[Container]
ContainerName=blog
Image=git.example.org/alice/blog:latest
AutoUpdate=registry
PublishPort=127.0.0.1:8101:80
Label=traefik.enable=true
Label="traefik.http.routers.blog.rule=Host(`www.example.org`) || Host(`example.org`)"
Label=traefik.http.routers.blog.entrypoints=https
Label=traefik.http.routers.blog.tls.certresolver=default
Label=traefik.http.routers.blog.middlewares=secureHeaders@file
Label=traefik.http.services.blog.loadbalancer.server.port=80
[Service]
Restart=always
[Install]
WantedBy=default.target
Quote the whole Label= value if it contains spaces or ||. Otherwise systemd splits it.
Real client IPs in the app, too
Traefik sees the client, but the app behind it sees 127.0.0.1. For nginx, tell it to trust Traefik’s X-Forwarded-For header:
# realip.conf, mounted into the nginx container at /etc/nginx/conf.d/realip.conf
# Traefik replaces any X-Forwarded-For sent by clients. The app port is published
# on 127.0.0.1 only, so the only peer that can connect is the local Traefik.
set_real_ip_from 0.0.0.0/0;
set_real_ip_from ::/0;
real_ip_header X-Forwarded-For;
The wide set_real_ip_from looks wrong at first. It is safe here only because nothing but Traefik can reach the app’s port.
Option B: Traefik on a Podman bridge network
On the home server Traefik and the apps share a Podman network called proxy. Traefik publishes the ports, and the apps publish none.
# proxy.network
[Network]
NetworkName=proxy
# traefik.container (bridge mode, shortened)
[Container]
ContainerName=traefik
Image=docker.io/library/traefik:v3
AutoUpdate=registry
PublishPort=80:80/tcp
PublishPort=443:443/tcp
Volume=%t/podman/podman.sock:/run/podman/podman.sock:ro,z
Volume=%h/.config/traefik/traefik.yml:/etc/traefik/traefik.yml:ro,z
Volume=traefik-letsencrypt.volume:/letsencrypt:z
Secret=dns-api-token,type=env,target=CF_DNS_API_TOKEN
Network=proxy.network:ip=10.89.0.2
[Service]
Restart=always
[Install]
WantedBy=default.target
An app joins the network with Network=proxy.network:ip=10.89.0.20. I give each container a fixed IP, so that other configs (Prometheus targets, for example) can use it. In this mode the provider section of traefik.yml has no useBindPortIP, because Traefik reaches the container IPs directly.
Ports below 1024 are still needed for PublishPort=80:80, because the rootless helper binds them as the user.
Which one should you pick?
If anybody on the internet uses the service, choose the host network. You will want the client addresses the first time somebody hammers your login page. If only you and your devices use it, the bridge is simpler. The apps then need no host ports, and no app can be reached past Traefik on the host.
Check
systemctl --user daemon-reload
systemctl --user start traefik blog
journalctl --user -u traefik -f # look for "certificate obtained"
curl -sI https://www.example.org/ | head -3
# Host network mode: the access log must show your real IP as ClientHost
journalctl --user -u traefik -n 5 -o cat | jq -r .ClientHost
Things to watch
- Test VMs and Let’s Encrypt. I restored
acme.jsoninto every test VM before Traefik’s first start, and blocked the VM’s outgoing connections while I tested. A VM that asks for certificates of the real domains counts against the rate limits of the live host. - Test against a VM without touching DNS:
curl --resolve www.example.org:443:<vm-ip> https://www.example.org/. With the restoredacme.jsonthe certificate is even valid. - Bridge mode and fail2ban. On the bridge, the client address that Traefik logs is the helper’s address. A ban on that would block everyone. So do not point fail2ban at the access log in bridge mode.
Next: SELinux enforcing with containers.
Further reading
- Traefik documentation: entrypoints, routers, services and middlewares.
- Traefik Docker provider: the provider that reads the labels. It works with the Podman socket.
- Traefik ACME: the HTTP, TLS and DNS challenges and the
acme.jsonstorage. - lego DNS providers: the list of DNS APIs that Traefik supports, with the environment variables each one needs.
- podman-network(1): bridge networks for rootless containers.