Operationalizing an Arr Stack: Health Invariants and Stateful Alerting

A stack is not finished when every container says Up.

For a multi-service media pipeline, the useful question is whether the relationships between the services are still correct:

Is the torrent client still behind the VPN?
Does its listen port still match the VPN provider’s forwarded port?
Can Sonarr and Radarr still reach the downloader and indexer bridge?
Are management UIs bound only where expected?
Is the cleanup layer still pointing at the intended services?
Is the post-import checker still read-only?

I turned those assumptions into executable health invariants and designed the monitoring system around state transitions rather than repeated alerts.

Container health is necessary but insufficient

Docker can tell you that a container is running. That does not prove the application is correctly wired into the rest of the system.

For example, all of the following can happen while every container remains Up:

qBittorrent listen port no longer matches VPN forwarded port
Sonarr points at an obsolete hostname
Jackett is reachable only through a stale network path
management UI is accidentally wildcard-published
Tdarr server API is exposed to the host unnecessarily
cleanup service points at an old endpoint

The health check therefore validates invariants, not only process state.

A useful health model

The stack-wide health script checks several categories.

Compose and lifecycle

Compose parses
expected services exist
services are running
restart policy matches policy
containers belong to the intended Compose project

VPN boundary

Gluetun healthy
only intended services share Gluetun namespace
input firewall contains only required ports
VPN IPv4 works
unexpected IPv6 Internet path is absent
provider-forwarded port exists

Torrent transport

qBittorrent API reachable
listen port == provider forwarded port
network interface == VPN tunnel
random port disabled
UPnP disabled
TCP listener present
UDP listener present
no stale default listener

Control-plane routing

Sonarr/Radarr on normal Docker network
Recyclarr/Cleanuparr on normal Docker network
Sonarr/Radarr can reach qBit via Gluetun service name
Sonarr/Radarr can reach Jackett via Gluetun service name

Management boundary

expected UIs bound only to trusted management path
no wildcard publication
no host publication for internal-only services

Application policy

Cleanuparr healthy
local-address authentication bypass disabled
Recyclarr read-only root filesystem
Recyclarr expected schedule and targets
Tdarr authentication enabled
Tdarr server/node API keys agree
Tdarr media mount read-only
Tdarr internal server listening but not host-published
Tdarr has no GPU device

Capacity

root filesystem below threshold
media/download filesystem below threshold

The script exits non-zero when a hard invariant fails.

Why exact host bindings are worth checking

A Compose file can accidentally regress from:

<trusted-management-address>:<port>

back to:

0.0.0.0:<port>

without affecting whether the application itself starts.

That is precisely the sort of configuration drift a health script should detect.

The same applies to ports that should not exist on the host at all. If an internal application server is only used by a process in the same container, its host publication should be treated as a failed invariant rather than an innocent extra mapping.

Authentication should be verified before exposure changes

One management service was initially loopback-only. Before moving it to the trusted management network, the account state was checked and a real login was verified.

The sequence was:

verify account initialized
verify local-address auth bypass disabled
verify password login works
only then change host binding
verify again from a fresh browser session

A second service required a slightly more involved bootstrap because enabling Web UI authentication also meant its internal node needed an API key.

The safe sequence was:

generate private API key
keep UI loopback-only
enable authentication
configure internal node with matching key
verify node reconnects
create/login to UI account
confirm auth errors stop
only then expose UI to trusted management network

This order prevents “enable auth and hope” deployments where an internal worker silently stops functioning after the UI is secured.

Build monitoring around state transitions

Running a health script every minute is easy. Sending a notification every minute during an outage is not useful.

The local watchdog should remember the previous state:

previous       current       action
--------       -------       ------
UNKNOWN/PASS   PASS          nothing
UNKNOWN/PASS   FAIL          send one DOWN
FAIL           FAIL          suppress duplicate
FAIL           PASS          send one RECOVERED
PASS           PASS          nothing

The important property is that notification behavior depends on the transition, not simply on the current exit code.

A simple local watchdog design

The planned local architecture is:

systemd timer
    |
    v
watchdog service
    |
    v
stack health script
    |
    +--> save latest output
    +--> compare state with previous run
    +--> notify on transition
    `--> preserve last failure for diagnosis

Persistent state can live somewhere such as:

/var/lib/<watchdog-name>/
├── state
├── last-health.txt
└── last-failure.txt

The notifier should be a small adapter so the state machine does not care whether delivery is through Home Assistant, ntfy, Gotify, Pushover, email, or another service.

Do not auto-restart on the first failure

Automatic remediation is attractive, but it can erase the evidence needed to understand a failure.

For the initial deployment, the watchdog should:

detect
record
notify

—not restart.

After real incidents have been observed, narrow remediations can be added for failure modes that are well understood and safe to automate.

For example, a stale application process may eventually justify a restart. A VPN/authentication/route failure may not.

A local watchdog cannot detect its own disappearance

There is one unavoidable blind spot:

host/LXC down
     |
     +--> timer cannot run
     +--> health script cannot run
     `--> local notifier cannot run

That requires an external dead-man monitor on a different failure domain.

The external check can be much simpler than the local one. It does not need to understand every application. It only needs to answer something like:

Is the host reachable?
Is one trusted management endpoint alive?
Is a heartbeat still fresh?

The local watchdog provides depth. The external monitor provides liveness from the outside.

Three notification layers work well together

A complete monitoring strategy can use three layers:

Layer 1: native application notifications
         - best event context
         - import/queue/cleanup-specific events

Layer 2: local stateful watchdog
         - deep cross-service invariants
         - DOWN / RECOVERED transitions

Layer 3: external dead-man monitor
         - host/container disappearance
         - independent failure domain

No single layer has to solve every problem.

Preserve evidence when a health check fails

A useful watchdog should retain the full health output rather than reducing everything to “red” or “green.”

A failure notification can be short:

Media stack DOWN: 3 failed invariants

while the local evidence contains the actual details:

VPN healthy
forwarded port present
qBit port mismatch
Sonarr reachable
Radarr reachable

That immediately narrows the incident from “the stack is broken” to “torrent transport state drifted.”

The final operational goal

The point of all of this is not maximal complexity. It is to make the system answer, quickly and unambiguously:

What failed?
What is still safe?
What is still available?
Did the failure recover?
Do I need to intervene?

A good Docker media stack is not just automated. It is observable, bounded, and recoverable without guessing.