Skip to content

Controlling the Jobs

--jobs (short form -j) decides how many jobs run at once. It is the single most important dial the tool has, and the timing table in chapter 4 showed why: the same eight one-second sleeps finished soonest with four slots, slowest with one — even on a busy lab machine where starting jobs cost more than the sleeps themselves.

The forms of -j

-j 4        exactly 4 slots
-j 0        as many as possible (one per input if need be)
-j 150%     one and a half slots per CPU core
-j -1       one slot fewer than the core count
-j +2       two slots more than the core count

With no -j at all, parallel uses one slot per CPU core. That is the right default for CPU-bound work — converting, compressing, computing. Other work wants other numbers:

  • Waiting work (downloads, remote calls, anything mostly idle) can run far above the core count; the jobs spend their lives waiting, not computing.
  • Disk-bound work often wants fewer slots than cores. Twelve jobs reading one disk are one disk working twelve ways; chapter 13 returns to this.
  • -j 0 is a foot-gun with long lists: ten thousand inputs means up to ten thousand simultaneous processes. It has its place — short, known lists of quick jobs — and nowhere else.

Starting gently and stopping early

  • --delay SECONDS waits between job starts, so a thousand jobs do not all hit a service, or a disk, in the same instant.
  • --timeout SECONDS ends a job that runs longer than the limit, and the job log records it as failed.
  • --halt now,fail=1 stops the whole run as soon as one job fails, instead of working through a long list that is already known to be broken. Its gentler cousin --halt soon,fail=1 lets running jobs finish first.

Check: transcript t06b ran parallel --halt now,fail=1 test -f {} ::: present-x absent present-y with the two present- files real and absent not. Parallel named the failed job — This job failed: test -f absent — and the run ended there with exit status 1, instead of quietly working through a list already known to be broken. The same transcript's --timeout 1 run ended a five-second sleep at one second, warned about it, and counted it failed.

The exit status is a summary

When the run ends, parallel's own exit status is the number of failed jobs, capped at 100 — or 0 when everything succeeded. Transcript t09 shows it plainly: one failed job out of three, parallel exit: 1. A script around a parallel run can trust that number, and chapter 9 shows how the job log says which jobs failed.