GNU Parallel for Mere Mortals

Running a command once is easy. Running it five hundred times — once per file, once per name, once per line of a list — is where afternoons go. GNU Parallel is the tool that takes a command and a list and runs the command over the whole list, several jobs at a time, with the inputs slotted into the command by replacement strings, the number of running jobs under your control, the output kept in an order you can reason about, and a log of every job afterwards. It is a single Perl program, it runs as your ordinary user, and it has been doing this one job since 2007. This guide explains how it works in those terms: where the input comes from, how jobs are controlled, what the records prove, how the same list can be spread over several machines — with the limits stated plainly, including the ones the lab ran into.

The lab behind this guide ran the real thing: GNU Parallel 20231122, the version in Ubuntu 24.04's repositories, on a small two-core Linux machine. The system was exercised end to end: first jobs from argument lists and pipes, all five everyday replacement strings and the positional forms, job control timed with sleeps, bulk checksums over a set of generated files with a job log kept and totalled, dry runs followed by a real compression batch, every input source (command line lists combined and linked, argument files, standard input, and a stream cut into blocks), output in finish order and in input order, streamed and grouped output, per-job results filed to disk, a failed run repaired and only its failures re-run, and the citation behaviour of the packaged build investigated down to the source of the shipped script.

What the lab could not do, it did not fake. The lab machine had no SSH server — a probe is in the transcripts — so running jobs on other machines (--sshlogin, file transfer and return) is taught from the upstream manual and labelled as such wherever it appears. The lab machine was also a shared sandbox under heavy load from other work while the timing runs were captured, and the transcripts say so: the sleep timings prove the ordering the job slots promise, and their absolute values carry the sandbox's job-start overhead, not a benchmark of the tool. criteria.md keeps the full ledger of what was run and what was only read.

What this guide covers

What GNU Parallel is — the mental model: one command, one list, several jobs at a time — and what it is not (a speed-up for a single job, a scheduler)
The moving parts — the input list, the command template, jobs and slots, and the record a run can keep
Installing GNU Parallel — the package install, the other parallel that is not this one, and the citation notice — what it is, and why the Debian and Ubuntu package never shows it
Your first parallel jobs — from a shell loop to :::, pipes, --dry-run, and a first timing run
Replacement strings — {} {/} {//} {.} {/.} and the positional {1} {2}, with the one-extension rule shown, not asserted
Controlling the jobs — -j in all its forms, starting gently, stopping early, and the exit status that counts the failures
Where the input comes from — ::: lists and their combinations, --link and :::+ pairs, :::: argument files, standard input, and --pipe blocks
Order and output — finish order by default, input order with -k, and streamed output with --ungroup and --line-buffer
Progress and records — --bar while it runs; the job log, --results, --resume and --retry-failed after it ends
Recipes — checksum a tree, compress a folder, derive new files from old, rename in pairs, summarise every file
Parallel on other machines — --sshlogin and moving files with --trc — documented from the manual, because the lab had no SSH server, and labelled that way
The neighbours — the shell loop, xargs -P, find -exec, make -j and fd --exec: what each trades, and when each is the right reach
Limits — shared bottlenecks, job-start overhead, held output and memory, no dependencies, no scheduler, stated plainly
Troubleshooting — the template pipe that was a real pipe, scrambled output, functions that vanish inside jobs, the impostor parallel, and the citation notice in the logs
Glossary — the terms, collected in one place

What it doesn't cover

Job dependencies and scheduling across days (that is make, a workflow engine or a cluster scheduler's subject, and chapter 12 says why). The full option list — GNU Parallel has hundreds of options, and this guide teaches the everyday core properly rather than all of them thinly; the {= … =} Perl expressions get a pointer, not a chapter. Running under Windows shells other than through WSL or Cygwin, and anything about citing the tool in academic work beyond settling its notice, are likewise pointed at, not taught.

How to use this guide

Read chapter 1 for the model and chapter 2 before running anything — knowing the four parts turns later confusion into a named problem. Chapter 3 installs the tool; chapter 4 runs your first jobs in the same sitting. Chapter 5 is not the optional chapter: replacement strings are where every real batch is won or lost, and chapter 9 is the other one not to skip — the first time a long run fails halfway, a job log and --retry-failed are the difference between an hour and a minute. Keep --dry-run from chapter 4 as a standing habit; the troubleshooting chapter assumes it. The lab's transcripts live in gnu-parallel-lab/transcripts/; its notes are examples/lab-notes.txt, and the file generator the lab used is gnu-parallel-lab/make-files.sh.

Part of the series

This guide is one of the For Mere Mortals guides: plain-language guides to the tools programmers and makers use.

Related guides:

License

The writing in this guide is licensed under CC BY 4.0. Code examples and lab scripts are licensed under the MIT License.