Skip to content

Logs with journalctl

Scope: the journal — where everything your services print actually goes, and the half-dozen journalctl forms that answer every real question.

The one idea

Services don't own log files anymore. stdout and stderr from every unit flow into the journal, a single, interleaved, timestamped stream kept by systemd-journald. One stream means one question-asking tool, and no more "which file did it log to, and did it rotate."

The everyday forms

journalctl -u mere-demo.service            # this unit, all of it
journalctl -u mere-demo.service -n 20      # last 20 lines
journalctl -u mere-demo.service -f         # follow (tail -f for services)
journalctl -u mere-demo.service --since "10 minutes ago"
journalctl -u mere-demo.service --since today
journalctl -b                              # this boot, everything
journalctl -b -1                           # the *previous* boot (the crash you're debugging)

Real lines from the demo service:

Oct 16 18:31:27 htch mere-demo.sh[16299]: mere-demo: heartbeat 1 (pid 16299)

Anatomy: timestamp, host, the unit's process name + PID, message. systemd's own messages about the unit (Started..., Scheduled restart job...) live in the same stream, attributed to systemd[1] — which is why debugging a service starts with its journal: the supervisor's account and the program's account are already merged, in order.

Filters that pay rent

  • -p err — priority at or above error: journalctl -p err -b. The journal carries syslog priorities; services logging at the right priority make this filter gold after an incident.
  • --grep — journalctl -u myapp --grep "timeout", server-side grep over the binary journal.
  • -o json — structured output when a script needs fields, not prose.
  • --disk-usage and journalctl --rotate / --vacuum-time=7d — how big the journal is, and trimming it. Size limits live in /etc/systemd/journald.conf (SystemMaxUse=).

Persistence: the honest footnote

Whether the journal survives a reboot depends on Storage= in journald.conf and the existence of /var/log/journal/: persistent keeps it across boots (needed for that -b -1 trick), volatile keeps it in RAM only, and the default auto means "persistent if /var/log/journal exists." Many distros ship persistent; some minimal installs don't. Check once, on each machine you care about, before the reboot you need to debug. (Documented — the mechanism is config, and this guide's lab verified journal reading and restart lines, not a reboot.)

The contract with your programs

Everything above works because the demo script obeyed two rules, and your programs should too:

  1. Log to stdout/stderr. No private log files unless you have a real reason; the journal is the reason's replacement.
  2. Don't double-log timestamps. journald stamps every line itself. Programs that prepend their own timestamps make journal lines stutter — and make -o json consumers sad. (Frameworks with native journal support can also emit priorities per line; plain text at default priority is fine for mortals.)