The Moving Parts

A GNU Parallel run has four parts. Naming them now makes every later chapter simpler.

The input list

The raw material: one input per job. Inputs come from arguments after :::, from files named after ::::, or from standard input when you pipe a list in. Chapter 7 covers each source. An input is usually one line of text — a filename, most often — and parallel treats it as text, quoting it before handing it to the shell.

The command template

The command you write, with replacement strings where the input should go:

parallel gzip -k {} ::: *.txt

{} is the placeholder for the input. There are more: {/} is the input with its directory removed, {//} just the directory, {.} the input with its extension removed,{/.}the bare name. Chapter 5 is the full table. If your command has no replacement string at all, parallel appends{}` to the end of it.

The template is a shell command line. Pipes, redirections and && inside it mean what they mean in the shell — which also means a stray | in your template is a real pipe, as the lab found out in transcript t02.

Jobs and slots

Each combination of template and input is a job. Parallel keeps a set of slots — --jobs decides how many — and runs one job per slot until the list is done. When a job finishes, its slot is refilled with the next input. By default there is one slot per CPU core.

Jobs are numbered from 1 in input order. That sequence number turns up in the job log, in --results file names, and in the {#} replacement string.

The record

Parallel can write down what it did:

• --joblog FILE — one line per job: start time, run time, exit value, and the command that ran (chapter 9).
• --results DIR — each job's output saved to its own file.
• --bar, --progress, --eta — a live view while the work runs.

None of this exists unless you ask for it, except the exit status of the whole run: parallel exits with the number of failed jobs (up to 100), so scripts can tell that something went wrong.

What runs where

Jobs run under /bin/sh — or your shell with env_parallel — as child processes of the parallel process itself. Output from each job is collected and printed as a block when the job finishes, so two jobs' lines do not interleave on your terminal; chapter 8 covers the options that change that.