Restarts and Failures

Scope: how systemd decides a service "started," what it does when one dies, and where it gives up — with a real crash loop dissected.

Type=: what "started" means

Type tells systemd how to know the service is up:

Type	systemd considers it started when...	Use for
simple	the process is running	almost everything; the default
oneshot	the command finishes (exit 0 = success)	batch jobs, setup steps — timers' partner
forking	the process forks and the parent exits	legacy daemons that insist on self-backgrounding
notify	the program sends an sd_notify signal	programs with real readiness signaling

Two mortal traps: a forking-style script under Type=simple that backgrounds itself and exits — systemd sees the main process exit and concludes the service stopped, then cleans up the actual worker. And a oneshot that never exits — the unit sits in activating forever and anything ordered after it waits. Match the type to the program's actual shape; the demo and most modern programs are simple.

Restart=: what happens when it dies

Policy	Restarts when
no (default)	never
on-failure	nonzero exit, signal death (kill, segfault), timeout, watchdog
on-success	only on clean exit 0
always	any exit at all — even a deliberate stop is honored only via systemctl stop, which overrides
on-abnormal / on-abort	signals/timeouts only, not nonzero exits

RestartSec= is the pause before each attempt (the demo used 2s). A clean exit under on-failure is respected — the demo's SIGTERM trap exiting 0 is why a deliberate stop doesn't fight the restart policy.

The crash loop, and giving up

examples/mere-flaky.service runs a script that exits 1 immediately, with Restart=on-failure, RestartSec=1. Watched live:

systemd[1]: mere-flaky.service: Scheduled restart job, restart counter is at 5.
systemd[1]: mere-flaky.service: Start request repeated too quickly.
systemd[1]: mere-flaky.service: Failed with result 'exit-code'.

and final status:

× mere-flaky.service - Crashes immediately, restarts until systemd gives up
     Active: failed (Result: exit-code) since ...
    Process: 16440 ExecStart=/usr/local/bin/mere-flaky.sh (code=exited, status=1/FAILURE)

systemd's patience is a start limit: by default, 5 starts within 10 seconds (StartLimitBurst, StartLimitIntervalSec — settable per unit in [Unit]) and it stops trying, leaving the unit failed. This is deliberate protection: a broken service shouldn't burn a machine forever, and a human should look. After fixing the cause:

sudo systemctl reset-failed mere-flaky.service
sudo systemctl start mere-flaky.service

reset-failed clears the failed state and the restart counters. Restarting without it fails instantly with the same "repeated too quickly" story — a classic ten-minute confusion the first time you meet it.

Exit statuses you'll actually see

In status, under Process: / Main PID::

• status=0/SUCCESS — clean.
• status=1/FAILURE (or any small number) — the program exited with that code. Your bug, or your program honestly reporting failure.
• status=203/EXEC — the executable couldn't be started at all: path wrong, not executable, bad shebang. systemd's own error range (200s) means "my side of the fence" — see Troubleshooting.
• code=killed, signal=KILL/SEGV — something shot it: OOM killer (check journalctl -b -1 for "Out of memory" lines) or a segfault.