How far does GNU Parallel scale?

[Ole Tange]

I rarely run more than 200 jobs in parallel. But I was curious how
many jobs I could safely run. This starts 30000 sleeps:

  seq 1000000000 | parallel -N1000 -j120 --pipe parallel -j0 sleep

I have let those stay in the background for a day now, and the only
issue I have had so far is that 'ps aux' takes > 1 sec to run because
it has to list > 30000 processes.

I also tried going higher, but GNU Parallel does not seem to act
predictably when hitting the system hits the 32767 process limit. But
that might also be caused by the rest of the system acting weird when
hitting that limit.

So using the fan out approach (parallel --pipe parallel) GNU Parallel
scales well to 30000 processes without any changes to my GNU/Linux
system.

If you are running GNU Parallel on another platform (and you are not
afraid to crash it), it would be interesting what the (safe) limit is
on other systems.


/Ole