Re: Properly scheduling multi-threaded/process jobs

[Ole Tange]

On Tue, Aug 26, 2014 at 2:15 PM, Douglas A. Augusto <daaugusto@gmail.com> wrote:

> Is there a way to inform GNU Parallel that a certain job consumes more than
> one slot so that the job is properly scheduled?

There is no such way per job, but there is per invocation of GNU
Parallel using the % in --jobs:

  parallel --S server1,server2 --jobs 50% my_2_core_command ::: *

This will run one instance of my_2_core_command for every 2 cores in
server1 and server2.

I have never in practice seen or even heard of a situation where the
number of cores used depended on the arguments, so I don't think it
happens very often for GNU Parallel use cases.

It does, of course, happen often for cluster queue systems, but GNU
Parallel is not intended to become a full replacement for cluster
queue systems like PBS, Sun Grid Engine and Torque.

If many users start to request the option and can argue why a full
cluster queue systems is not the right way to go for them, I might be
persuaded to consider it.

> More specifically, I am looking for an option "--slots" like:
>
>    parallel -j 3 --slots {3} "echo {1} sleeping processes for {2} seconds; 
> mpiexec -n {1} sleep {2}" ::: 2 3 ::: 5 10 ::: 2 3

I assume you want to add an '--xapply' to that - otherwise it does not
make any sense to me.

If your goal is to not overload a server, look at --load: The idea
here being that --load 100% will only start another job if there is at
least 1 core sitting idle.

  parallel -j 100% --load 100% --xapply "echo {1} sleeping processes
for {2} seconds; mpiexec -n {1} sleep {2}" ::: 2 3 ::: 5 10

(Crappy example because 'sleep' will make the core be sitting idle).


/Ole