Watchdogs

A deploy gate proves the host was healthy at deploy time. Some faults start later and fail silently: supervisorctl status says RUNNING, requests are served, and the damage piles up somewhere nobody looks. A watchdog is a small cron job that looks there.

When a Service Needs One

Write a watchdog for a fault that is silent and cumulative: every one below was found by a human, days late:

Fault	Why nothing else reports it	Detector
Orphaned uwsgi workers	supervisor is no longer their parent; the live workers serve fine	uwsgi processes parented to init
A queue with no consumer	the broker accepts the message; no task ever fails	messages > 0 and consumers = 0
A nightly job that “ran”	cron exited 0; the job did nothing useful	the job’s own outcome, not its exit code

Not a watchdog: anything a metric plus an alert threshold already covers (CPU, disk, HTTP 5xx rate). Use a metric and an alert threshold in the monitoring you already run for those.

Anatomy

The reference layout (your repository, not the service repo):

deploy/templates/<name>-watchdog.sh        cron entrypoint: gather, call the verdict, relay
deploy/templates/cron-<name>-watchdog.tmpl every 15 minutes, output appended to a log
main/<name>_verdict.py                     the decision — pure Python, unit-tested

1. Gather raw state in shell: one ps, one rabbitmqctl list_queues. No logic.
2. Decide in Python, with tests. Logic inside a shell heredoc has no tests, and a detector with a bug in its verdict is worse than none: it reports health.
3. Can’t look ≠ healthy. A read that fails is its own verdict (NO_READ) and alerts like a real fault. A detector that cannot see must never say OK.
4. Confirm on two consecutive runs. A process reparented for a second during a restart, or a queue drained between two reads, must not page anyone. Persist what this run saw in a state file; report only what the previous run also saw.
5. Clear on the transition only. Send “recovered” when the state goes from fault to OK, not on every healthy run.
6. Load the env file explicitly. Cron gives a script almost no environment, so a flag the script expects to inherit is unset.

Run every watchdog on the same cadence (15 minutes) unless there is a measured reason not to. With two-run confirmation, a real fault surfaces in about 30 minutes.

Report-Only First, Then Arm

A new detector ships report-only: it runs, logs its verdict, and sends nothing.

1. Deploy it. Let it run on a live host for a few cycles.
2. Triage its log against ground truth: the same ps / rabbitmqctl a human would run.
3. When the verdicts match reality, set <NAME>_WATCHDOG_ARMED=1 in the source env file (deploy/<app>.env) and run your quick deploy.

Never arm it in the generated /etc/<app>.env: the next converge rewrites that file and the detector goes quiet without saying so. A detector that cries wolf trains everyone to skim the channel it posts to. The triage step is what keeps the channel worth reading.

Every Alert Has a Runbook

The alert body names its runbook section, and the section in the service’s deployment doc has three parts:

• Reading the situation by hand: the exact commands, and what healthy output looks like.
• Clearing it: a safe, targeted fix, followed by a check that the service is still up.
• Arming the watchdog: the flag and the quick deploy.

See Process Supervision → Verify It for the orphaned-worker commands.

Related

• Reference Setup: the post-deploy gate that runs before a watchdog.
• Process Supervision: what stops orphans being created.

Next

Deploy Patterns