Skip to content

Parallel on Other Machines

Read this first: this chapter is documented-only. The guide's lab had no SSH server to connect to — transcript t10 shows the probe: nothing listening on port 22, and ssh localhost refused. So the mechanics below come from the upstream manual, not from a lab run, and they are presented as what the tool promises, ready for you to verify on machines you control.

The idea

With --sshlogin, the list of jobs is spread across machines instead of job slots on one machine. Parallel logs into each host over SSH, runs its share of the jobs there, and brings the output back to your terminal, interleaved with the results from every other host — and with a job log that records which host ran what in its Host column (the : in this guide's local job logs means "this machine").

parallel --sshlogin user@host1,user@host2 echo {} ::: one two three

A host given without -j contributes as many slots as it has cores; the list of hosts can also live in a file, one per line, for a standing set of machines.

The requirements

  • Key-based SSH login to every host. Parallel cannot stop to type passwords mid-run.
  • The same command must exist on each host. Parallel itself can be copied over automatically when a host lacks it; your convert or gzip cannot.
  • Inputs must be reachable. A job that names samples/notes-01.txt needs that file on the host it lands on — a shared filesystem — unless you ask parallel to move files for you.

Moving files: --transfer, --return, --trc

For hosts that do not share storage, parallel can ship each input to the host, run there, and ship named results back:

  • --transfer sends each input file to the host before its job runs;
  • --return FILE brings a named result file back;
  • --cleanup deletes the copies afterwards;
  • --trc FILE is the three combined (transfer, return, cleanup), the form most people actually use.

Output files come back into the working directory, so run remote batches from a folder you made for them.

When it is worth it — and when not

Worth it when the jobs are CPU-heavy, the inputs are small or already shared, and you genuinely have idle machines. Not worth it when the work is moving large files to save small computation, or when one machine's cores already finish the list before you could finish typing the hostnames. A single machine with the right -j (chapter 6) is the common case; this chapter is for the days it is not.