Skip to content

What GNU Parallel Is

GNU Parallel runs the same command many times at once. You give it a command and a list of inputs — files, names, numbers, whatever you have — and it runs the command once per input, several at a time, using the cores your machine would otherwise leave idle.

parallel echo hello ::: one two three

runs echo hello one, echo hello two and echo hello three as three separate jobs, overlapping in time.

That is the whole idea. Everything else in this guide — the replacement strings, the job control, the logs — exists to make that one idea safe and useful on real work:

  • The list can come from anywhere: the command line, a file, a pipe, or several lists combined (chapter 7).
  • The command can use each input in several ways: the full path, just the file name, the name with its extension removed, and more (chapter 5).
  • You decide how many run at once, and the output arrives in an order you can reason about (chapters 6 and 8).
  • It keeps records — a log of every job, its time and its exit status — so a run of five hundred jobs can be checked afterwards (chapter 9).

GNU Parallel is a single Perl program, written by Ole Tange and maintained since 2007. It is not a scheduler, not a cluster system, and not a programming language. It runs under your shell, as your user, with your permissions, and it starts and finishes with the command you typed.

What it is not

Not a speed-up for a single job. One input still takes as long as it takes. Parallel helps when you have many inputs and a machine with spare cores or spare waiting time.

Not magic for slow disks or networks. If every job waits on the same disk or the same download, running twelve at once can be slower than running four. Chapter 13 is honest about this.

Not a replacement for understanding the command. Parallel runs the command you give it, as many times as you say, including the destructive ones. Chapter 4 starts with --dry-run for a reason.

Where it earns its keep

The everyday shape of the work:

  • checksum, compress or convert every file in a folder tree;
  • run the same check against every line of a list;
  • shrink, resize, rename or re-encode a whole batch instead of babysitting a loop;
  • spread a long list across several machines over SSH (chapter 11).

If you have ever written a for loop and then watched one CPU core work while the rest idled, you already know the problem this tool solves.