Skip to content

Troubleshooting

Scope: the failures you'll actually meet, symptom → fix. Messages are verbatim from real runs (systemd 255) unless marked (documented).

"I edited the unit and nothing changed"

Symptom: edits to a .service file have no effect.

Fix: sudo systemctl daemon-reload. systemd caches units in memory; until told, it runs the old version. This is the cause so often that it's the first entry. After reloading, restart the service — a reload alone doesn't restart anything either.

"Failed to start nosuch-thing.service: Unit nosuch-thing.service not found."

Failed to start nosuch-thing.service: Unit nosuch-thing.service not found.

Symptom: start fails instantly.

Fix: the unit doesn't exist under that name. Typo, wrong host, or the file was placed but daemon-reload never ran so systemd hasn't seen it. systemctl list-unit-files | grep thing shows what systemd knows about.

status=203/EXEC — the binary never ran

× mere-missing.service - Points at a binary that does not exist
     Active: failed (Result: exit-code) since ...
    Process: 16390 ExecStart=/usr/local/bin/does-not-exist-anywhere (code=exited, status=203/EXEC)
   Main PID: 16390 (code=exited, status=203/EXEC)

Symptom: instant failure, before any of your program's output.

Fix: 203/EXEC means systemd couldn't exec the command: the path doesn't exist, the file isn't executable (chmod +x), it's a script with a bad or CRLF-mangled shebang line, or ExecStart used a relative path — always absolute. The 200-series codes are systemd's own: 200/CHDIR (bad WorkingDirectory), 201/NICE, 202/IO, 212/CREDENTIALS and friends. When the status is 200+something, debug the unit; when it's a small number like 1, that's your program's own exit code — debug the program.

"Start request repeated too quickly"

systemd[1]: mere-flaky.service: Scheduled restart job, restart counter is at 5.
systemd[1]: mere-flaky.service: Start request repeated too quickly.
systemd[1]: mere-flaky.service: Failed with result 'exit-code'.

Symptom: a crashing service is failed, and manual start attempts bounce off.

Fix: systemd hit the start limit (default 5 in 10s) protecting the machine from a crash loop, and latched the failed state. Read the journal for the cause (journalctl -u <unit> — your program's stderr is there), fix it, then:

sudo systemctl reset-failed <unit>
sudo systemctl start <unit>

Restarting without fixing buys exactly five more crashes.

The service "runs" but isn't your new code

Symptom: behavior unchanged after deploying new files.

Fix, in order: you edited the script but forgot restart (a running service keeps the old process); you edited the unit but forgot daemon-reload then restart; or there are two copies of the program and ExecStart points at the other one — status shows the exact binary and PID, and systemctl cat <unit> shows the unit as systemd actually parsed it (drop-ins included).

Unknown directives silently ignored

/etc/systemd/system/mere-broken.service:7: Unknown key name 'Typee' in section 'Service', ignoring.

Symptom: a setting you wrote has no effect — most cruelly, a misspelled Restart= means no restart policy.

Fix: typos in unit files are ignored, not errors, by design (forward compatibility). systemd-analyze verify <file> prints the complaints; run it on every new unit, and after "a setting isn't working," verify before theorizing.

Service starts as root but can't read "its" files

Symptom: after adding User= or sandboxing, the service fails with permission errors in the journal.

Fix: the walls are working — aim them. Files owned by root that the new user can't read, a data directory outside ProtectSystem=strict's ReadWritePaths=, state written to a home directory ProtectHome now hides. The pattern for real services: a dedicated user, a StateDirectory= (systemd creates and owns /var/lib/<name> for you), and ReadWritePaths naming exactly that. See Hardening.

Stops are slow, or stop "hangs"

Symptom: (documented) systemctl stop takes the full 90 seconds.

Fix: your program ignores or mishandles SIGTERM — the polite request systemd sends first. The demo's trap handler is the shape of the fix: catch TERM, clean up, exit. systemd escalates to SIGKILL after TimeoutStopSec; the journal's "State 'stop-sigterm' timed out. Killing." line names the moment patience ran out.