The Neighbours

GNU Parallel is not the only way to run things at once. Each neighbour below is the right tool for something; the point of the chapter is knowing which job you actually have.

The shell loop

for f in *.txt; do gzip -k "$f"; done

Free, obvious, already in your fingers. Its costs: one job at a time, no records, and error handling is whatever you remembered to write. For five files it beats everything in this guide. For five hundred files that take a minute each, it is a day of your life.

xargs -P

The traditional paralleliser, in every Unix base system:

ls *.txt | xargs -P 4 -I{} gzip -k {}

Honest strengths: it is already there, and for "run this over that list, N at a time" it does the job. Where it thins out against parallel: output from jobs interleaves on the terminal, there are no replacement strings beyond whole-input substitution, no job log, no keep-order, no resume, no remote hosts. If your batch needs any of those, you have outgrown it; if it needs none, xargs earns its keep.

find -exec … +

find can run a command over its own results, batching many files per invocation with +. It overlaps nothing — jobs run one after another — but for quick per-file commands the batching alone removes most of the overhead, and there is nothing new to learn. Parallel is what you reach for when the per-file work is heavy enough that the cores matter.

make -j

If your jobs have dependencies — B needs A's output, C needs both — you do not have a list problem, you have a build problem. make -j runs a graph of targets with the dependencies honoured. Parallel deliberately knows nothing about dependencies; forcing it to pretend otherwise ends in scripts that re-implement make, badly.

fd --exec and friends

Modern finders like fd have their own parallel --exec. Inside their home turf — "every file matching this, run that" — they are pleasant and quick. They stop where parallel starts: multiple input sources, derived names, logs, remote machines.

The rule of thumb

Loop for a handful. xargs when it is already enough. Parallel when the list is long, the jobs are heavy, the names need surgery, or the run deserves a record. make when jobs depend on each other.