Re: Properly scheduling multi-threaded/process jobs

[Douglas A. Augusto]

On 26/08/2014 at 11:04,
Rob Sargent <robjsargent@gmail.com> wrote:

> We use tools (genetic analysis) which have explicit parameters for processor
> consumption and which when left to their own devices use variable numbers of
> core during a given run. We gauge their typical cpu load and parallelize
> accordingly.  I'm not too worried about exceeding cores/slots (within
> reason).  More runnables than cores to me just means maximal cpu usage.
> (Think back to the single cpu days: there was always more than one runnable,
> and that was a good thing.)

Hi,

The ability to tell GNU Parallel that a certain job requires 'n' slots can
also, indirectly, be used when the resource is memory. In this way one could
assign 1 to the less memory hungry job and more than 1 to all others,
proportionally.

> Are you saying you know beforehand which job needs how many cores?  If so
> (before Ole builds your feature) could you group them into say small, medium
> and large and parallel those groups with reasonable --jobs each?

Yes, I know beforehand how many cores a job needs, this is an input parameter.
Grouping them is exactly what I did, but the drawback is that this imposes an
unnecessary synchronism point (barrier) between groups--this is specially bad
if there are unbalanced long-running jobs within a group.

-- 
Douglas A. Augusto