Re: Properly scheduling multi-threaded/process jobs

[Rob Sargent]

On 08/26/2014 10:52 AM, Douglas A. Augusto wrote:

On 26/08/2014 at 15:55,
Ole Tange <tange@gnu.org> wrote:

There is no such way per job, but there is per invocation of GNU
Parallel using the % in --jobs:

Hi,

Thanks for the quick reply. I'm aware of that invocation, but it doesn't
suffice (see at the end).

I have never in practice seen or even heard of a situation where the
number of cores used depended on the arguments, so I don't think it
happens very often for GNU Parallel use cases.

I'm facing this situation right now. I've a 40-core machine and lots of MPI
jobs ranging from 1 to 32 processes each. I'd like to run simultaneously as
many jobs as possible, but they cannot exceed 40 slots/processes.

We use tools (genetic analysis) which have explicit parameters for processor consumption and which when left to their own devices use variable numbers of core during a given run. We gauge their typical cpu load and parallelize accordingly.  I'm not too worried about exceeding cores/slots (within reason).  More runnables than cores to me just means maximal cpu usage. (Think back to the single cpu days: there was always more than one runnable, and that was a good thing.)

If many users start to request the option and can argue why a full
cluster queue systems is not the right way to go for them, I might be
persuaded to consider it.

Nowadays many tools explore some sort of parallelism, be it multi-thread or
multi-process. Handling those tools properly would extend the applicability of
GNU Parallel. GNU Parallel is straightforward to use and has so many useful
features that switching to a full-fledged queue system would overcomplicate
things and the user would lose all those GNU Parallel's nice features.

More specifically, I am looking for an option "--slots" like:

   parallel -j 3 --slots {3} "echo {1} sleeping processes for {2} seconds; mpiexec -n {1} sleep {2}" ::: 2 3 ::: 5 10 ::: 2 3

Are you saying you know beforehand which job needs how many cores?  If so (before Ole builds your feature) could you group them into say small, medium and large and parallel those groups with reasonable --jobs each?