Skip to content

Where the Input Comes From

A parallel run needs at least one source of inputs. Transcript t05 ran each of these for real.

From the command line: :::

The most common source. Everything after ::: is the list:

parallel echo ::: one two three

Give two lists and you get every combination of them — the Cartesian product:

$ parallel -k echo {1}-{2} ::: A B C ::: 1 2
A-1
A-2
B-1
B-2
C-1
C-2

{1} is the item from the first list, {2} from the second. Six jobs, not five: three times two. Combinations grow multiplicatively, which is power and hazard in one — three lists of a hundred are a million jobs.

Sometimes two lists belong together item by item, not in combination. --link walks them in step:

$ parallel -k --link echo {1}-{2} ::: A B C ::: 1 2 3
A-1
B-2
C-3

The spelling :::+ does the same for a single argument position: the lists join into one stream of pairs. Chapter 10's rename recipe uses it:

parallel mv {1} {2} ::: old1 old2 :::+ new1 new2

runs mv old1 new1 and mv old2 new2.

From files: :::: and --a

:::: reads inputs from a file, one per line:

$ parallel -k echo found :::: names.txt
found notes-01.txt
found notes-03.txt
found notes-05.txt

--a names.txt (long form --arg-file) does exactly the same; the two spellings exist so that file sources and command line sources can be mixed in one run. Inputs from a file combine with ::: lists the same way lists combine with each other.

From a pipe: standard input

With no source given, the list is standard input, one job per line:

ls samples/*/*.txt | parallel sha256sum {}

This is the form that composes with the rest of the shell: whatever can print a list — find, ls, a database query, a previous command — can feed parallel.

One caution: the job commands also inherit standard input, and by default parallel feeds them /dev/null, so a command that tries to read interactively from the terminal gets nothing rather than stealing list lines. That is the behaviour you want.

From a stream of data: --pipe

A different mode entirely. Instead of one job per line, --pipe cuts the input stream into blocks and hands each block to a job's standard input:

$ seq 1 25 | parallel --pipe -N10 wc -l
10
10
5

Three jobs, each counting the lines in its block. This is the shape for processing big text — logs especially — where the work is on the data, not on the filenames. -N10 asked for blocks of ten lines; --block sizes blocks by bytes instead.