I have an sshloginfile with 12 hostnames in it. All machines are identical. I used parallel like this:
$ parallel --controlmaster --sshdelay 0.3 --joblog joblog.txt --sshloginfile ~/etc/parallel/hosts 'echo {} | /shared/nfs/process-file' :::: filelist.txt | tee -a result.txt
/shared/nfs/process-file is a shell script and is accessible on all hosts
filelist.txt is a file with 6247 paths to .gz files in it. These .gz files are processed by the process-file script.
While the parallel was running I noticed that all of the jobs were being allocated to one of the hosts in the hostfile, host05. In fact, the first time a host other than host05 was allocated a job was on sequence number 622.
The final distribution of how many jobs ran on each of the 12 hosts is:
$ cat joblog.txt | awk '{ print $2 }' | sort | uniq -c
2 host01
538 host02
1 host03
323 host04
4696 host05
1 host07
2 host08
2 host09
493 host10
3 host11
3 host12
1 Host
$
As you can see, the distribution is very imbalanced.
Later in the job run, I did start seeing messages like the following: