Process Supervision

Supervisor’s defaults are wrong for a process that forks. A stop that times out sends SIGKILL to the program’s main pid only; its children reparent to init and keep running, invisible to supervisorctl status. Every restart can leak them, and every deploy restarts.

What It Cost

Measured on two production hosts: orphaned uwsgi workers had piled up for days, both containers sat near full CPU pressure while the host underneath was mostly idle. The app answered every request correctly the whole time; the live workers were fine. A human found it on a CPU graph.

The chain:

1. The unit said stopsignal = QUIT; the ini said die-on-term = true. Under die-on-term, uWSGI treats SIGQUIT as a brutal reload, not a shutdown.
2. Supervisor’s stop asked the master to reload. It killed its workers, spawned new ones, and never exited.
3. After stopwaitsecs supervisor sent SIGKILL to the master alone (the program declared no group-kill) and the freshly spawned workers reparented to init.
4. Each orphan still held supervisor’s stdout pipe, whose read end was now closed. Every log line failed with EPIPE; uWSGI’s SIGPIPE handler logs “SIGPIPE: writing to a closed pipe” to the same fd. A flood of failed write()s, one core each, forever.

Two independent defects, each survivable alone: the wrong signal made the master outlive its stop window, and the missing group-kill turned that into a permanent leak. The rules below close both.

The Rules

1. Every program sets stopasgroup = true and killasgroup = true. The stop signal and the final SIGKILL then reach the whole process group. Set both on every program, including ones that do not fork today. A uniform rule is one a test can check.
2. The stop signal means shutdown for that server, in its current configuration. Look it up in the server’s source or docs, not in another unit file.
3. stopwaitsecs covers the longest in-flight work you are willing to wait for. The default is 10 seconds.
4. Templates are guarded by tests that fail when a rule regresses.

uwsgi

What each signal does to the uWSGI 2.0 master (core/master.c):

Signal	die-on-term = false	die-on-term = true
SIGTERM	brutal reload	shutdown
SIGQUIT	shutdown	brutal reload
SIGINT	shutdown	shutdown
SIGHUP	graceful reload	graceful reload

Use SIGINT: it is a shutdown in both modes, so the unit stays correct if someone flips die-on-term later.

; deploy/templates/supervisor-uwsgi.conf.tmpl
[program:@APP@-uwsgi]
command = @VENV@/bin/uwsgi --ini /etc/uwsgi/vassals/@[email protected]
user = webapp
autostart = true
autorestart = true
stopsignal = INT
stopasgroup = true
killasgroup = true
redirect_stderr = true
stdout_logfile = @LOG@/uwsgi.log

; deploy/templates/uwsgi.ini.tmpl — the lines that matter here
[uwsgi]
master      = true
processes   = 4
die-on-term = true
; workers exit when the master dies — covers a master killed from outside supervisor
; (the OOM killer, a stray kill -9), which group-kill cannot see
no-orphans  = true
harakiri    = 300

SIGINT kills the stack immediately, so stopwaitsecs barely matters for uwsgi: in-flight requests are cut on restart. That is the accepted trade for a routine deploy. Zero-downtime reloads (SIGHUP, chain reload) are a separate decision, not the default.

celery worker

SIGTERM is Celery’s warm shutdown: stop taking tasks, finish the running ones, exit. The prefork pool children must receive it too, which is what stopasgroup does. Celery’s own supervisor example (extra/supervisord/celeryd.conf) sets it for that reason.

; deploy/templates/supervisor-celery.conf.tmpl
[program:@APP@-celery]
command = @VENV@/bin/celery --workdir=@SVC@ -A main worker --queues=...
user = webapp
autostart = true
autorestart = true
stopsignal = TERM
stopasgroup = true
killasgroup = true
; Celery's example value. Tasks longer than this are killed on restart and must be safe
; to re-run — size it to the tasks you actually have, not to --time-limit.
stopwaitsecs = 600
environment = @SUPERVISOR_ENV@

The --queues list is not free text either: a queue a task publishes to and no worker consumes swallows messages with no error anywhere. Reconcile it against Celery’s task registry in a test.

celery beat

One process, no children. The group settings are there for uniformity.

; deploy/templates/supervisor-beat.conf.tmpl
[program:@APP@-beat]
command = @VENV@/bin/celery --workdir=@SVC@ -A main beat --schedule=@RUN@/beat-schedule
user = webapp
autostart = true
autorestart = true
stopsignal = TERM
stopasgroup = true
killasgroup = true
numprocs = 1
environment = @SUPERVISOR_ENV@

Run exactly one beat per environment. Two beats double every scheduled task.

Verify It

On the host, after any restart:

supervisorctl status                                        # every program RUNNING
ps -eo pid,ppid,etimes,pcpu,comm -C uwsgi                   # one master + `processes` workers
ps -eo pid,ppid,etimes,pcpu,comm -C uwsgi | awk '$2==1'     # orphans — must print nothing

The last line belongs in the post-deploy gate and in a watchdog: the gate catches a leak at deploy time, the watchdog catches one created between deploys.

Orphans serve no traffic and never exit on their own. Kill the ones parented to init, never “every uwsgi”, which takes the live master with them:

ps -eo pid,ppid --no-headers -C uwsgi | awk '$2==1 {print $1}' | xargs -r kill -9
supervisorctl status @APP@-uwsgi                            # must still be RUNNING

Guard It in CI

Parse the templates in the service’s test suite. Supervisor programs are INI; a regex over key = value lines is enough:

import re
from pathlib import Path

TEMPLATES = Path(__file__).resolve().parents[1] / "deploy" / "templates"
SHUTDOWN_SIGNALS = {"INT"}  # a uWSGI shutdown whatever die-on-term says


def _setting(text: str, key: str) -> str | None:
    match = re.search(rf"^{key}\s*=\s*(\S+)", text, re.M)
    return match.group(1) if match else None


def test_every_program_kills_its_whole_group():
    for unit in TEMPLATES.glob("supervisor-*.conf.tmpl"):
        text = unit.read_text()
        for key in ("stopasgroup", "killasgroup"):
            assert _setting(text, key) == "true", f"{unit.name}: {key} must be true"


def test_uwsgi_stop_signal_is_a_shutdown():
    text = (TEMPLATES / "supervisor-uwsgi.conf.tmpl").read_text()
    assert _setting(text, "stopsignal") in SHUTDOWN_SIGNALS

Related

• Reference Setup: the host and the deploy rail these units belong to.
• Watchdogs: detecting orphans between deploys.

Next

Watchdogs