Hello --
GNU parallel is a great piece of software. I use it all the time.
From time to time, I run into a file handle limit issue when I have many jobs to run (1M+):
parallel: Warning: No more file handles. parallel: Warning: Raising ulimit -n or /etc/security/limits.conf may help.
I hadn't realized that it was related to --keep-order, but Ole's comment on
SO suggest's that it is.
* What the relationship is between is between keep-order, the number of jobs, and the number of file handles used? Is there any relationship with the number of cores as well? [On some machines, I don't have control over the file limit, so better understanding this interplay would be useful]
* Could someone also clarify the expected behavior on stdout? When multiple jobs (e.g. A and B) are running, is it possible for the output to become interleaved (without --keep-order)? In other words (each line represents output from job A or B),
1) Interleaved:
2a) Or am I guaranteed ....
A
A
A
B
B
B
2b) OR
B
B
B
A
A
A
For my purposes, either 2a or 2b would be acceptable, but 1 would not. Do I need to specify --keep-order to guarantee 1 doesn't occur.
Thanks for the help,
Shyam
PS. Ole, we just published a
paper that used GNU parallel. Of course, it is cited.