Re: Properly scheduling multi-threaded/process jobs

[Douglas A. Augusto]

On 26/08/2014 at 15:55,
Ole Tange <tange@gnu.org> wrote:

> There is no such way per job, but there is per invocation of GNU
> Parallel using the % in --jobs:

Hi,

Thanks for the quick reply. I'm aware of that invocation, but it doesn't
suffice (see at the end).

> I have never in practice seen or even heard of a situation where the
> number of cores used depended on the arguments, so I don't think it
> happens very often for GNU Parallel use cases.

I'm facing this situation right now. I've a 40-core machine and lots of MPI
jobs ranging from 1 to 32 processes each. I'd like to run simultaneously as
many jobs as possible, but they cannot exceed 40 slots/processes.

> If many users start to request the option and can argue why a full
> cluster queue systems is not the right way to go for them, I might be
> persuaded to consider it.

Nowadays many tools explore some sort of parallelism, be it multi-thread or
multi-process. Handling those tools properly would extend the applicability of
GNU Parallel. GNU Parallel is straightforward to use and has so many useful
features that switching to a full-fledged queue system would overcomplicate
things and the user would lose all those GNU Parallel's nice features.

> > More specifically, I am looking for an option "--slots" like:
> >
> >    parallel -j 3 --slots {3} "echo {1} sleeping processes for {2} seconds; 
> > mpiexec -n {1} sleep {2}" ::: 2 3 ::: 5 10 ::: 2 3
> 
> I assume you want to add an '--xapply' to that - otherwise it does not
> make any sense to me.

Perhaps the example is oversimplified, but the idea is to run for each number
of processes ({1}) many different arguments (in this case, given by {2}). Thus,
I want to run 'sleep 5' and 'sleep 10' using both 2 and 3 instances each:

   mpiexec -n 2 sleep 5
   mpiexec -n 2 sleep 10
   mpiexec -n 3 sleep 5
   mpiexec -n 3 sleep 10

Although very simple, this example illustrates what I'm facing at the moment.

> If your goal is to not overload a server, look at --load: The idea
> here being that --load 100% will only start another job if there is at
> least 1 core sitting idle.

I've considered that, but the problem is that if the load is for example 90%,
currently GNU Parallel will launch the next job regardless of how much slots it
requires. For instance, if the next job is a 32-thread OpenMP application, then
launching it on a 40-core system 90% loaded will certainly almost double the
system load.

Another way of approaching all that is by exposing (via an interface) the GNU
Parallel's scheduler so that the user could provide his or her own customized
scheduler. This would also allow many other crazy things as well...

-- 
Douglas A. Augusto