Your First Parallel Jobs

Everything in this chapter was run for real; the full captures are transcripts t01 to t03 in the lab notes.

From a loop to a list

A shell loop does one thing at a time:

$ for n in one two three; do echo hello $n; done
hello one
hello two
hello three

The parallel version hands the same work to GNU Parallel as a command plus a list. The list follows ::::

$ parallel echo hello ::: one two three
hello one
hello two
hello three

Read it as: run echo hello once per input, with the input placed where {} would go — and since the command has no {} in it, the input is appended:

Check: parallel echo ::: one two three prints one, two, three, each from its own echo job.

The list can just as well arrive on standard input, which is how parallel slots into pipelines:

$ printf 'apple\nbanana\ncherry\n' | parallel echo got {}
got apple
got banana
got cherry

Look before you leap: --dry-run

Parallel will run a command hundreds of times without asking. Before any batch that changes files, ask it to print the commands instead of running them:

$ parallel --dry-run gzip -k {} ::: samples/north/*.txt
gzip -k samples/north/notes-01.txt
gzip -k samples/north/notes-02.txt
...

If the printed commands are not exactly the ones you meant, fix the template now, while fixing is free. Transcript t06 shows a dry run followed by the real run, and chapter 14 returns to this.

What "in parallel" means here

With several inputs and a slow command, jobs overlap. Eight one-second sleeps — transcript t03, on the lab's two-core machine:

Command	Wall time
parallel -j1 sleep 1 ::: 1 2 3 4 5 6 7 8	47.6s
parallel sleep 1 ::: 1 2 3 4 5 6 7 8 (default: 2 jobs)	26.2s
parallel -j4 sleep 1 ::: 1 2 3 4 5 6 7 8	15.7s

-j is the number of job slots (chapter 6). The default is one slot per CPU core. The sleeps themselves account for only 8, 4 and 2 of those seconds — the rest is the cost of starting jobs on the lab machine, a shared sandbox that was busy with other work at the time. On a quiet machine the same three runs land close to 8, 4 and 2 seconds.

Both halves of that result are the lesson. The waits overlapped exactly as the slot counts say they should — and starting a job is never free, which is why chapter 13 warns against spending parallel on jobs that take milliseconds. Real commands behave the same way when their time is spent waiting or spread across cores, and chapter 13 explains when they don't.