[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Properly scheduling multi-threaded/process jobs
|
From: |
Douglas A. Augusto |
|
Subject: |
Re: Properly scheduling multi-threaded/process jobs |
|
Date: |
Sun, 31 Aug 2014 13:19:25 -0300 |
|
User-agent: |
Mutt/1.5.23 (2014-03-12) |
On 31/08/2014 at 09:59,
Ole Tange <ole@tange.dk> wrote:
> So while I am happy to learn that you appreciate GNU Parallel's
> features, then instead of expanding GNU Parallel into a full queuing
> system I would prefer seeing an extension that would make it possible
> for GNU Parallel to interface to real queuing systems with proper
> resource management (such as Torque). That way we can re-use
> development already done instead of re-inventing the wheel.
Hi, I couldn't agree more. I don't know the internals of GNU Parallel's
scheduling kernel, but perhaps an intermediate approach would be to make it
easy to plug in user-defined schedulers by means of an interface.
Regarding the handling of multi-process jobs within GNU Parallel, for now I'm
using a work-around: instead of running the job directly I use a wrapper that
only starts the job if there are enough resources to run it; otherwise it will
sleep for awhile (of course there are some side-effects, such as making
inconsistent the use of --timeout).
--
Douglas A. Augusto