parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Uses of GNU Parallel


From: Ole Tange
Subject: Re: Uses of GNU Parallel
Date: Mon, 13 Feb 2017 13:45:36 +0100

On Mon, Feb 13, 2017 at 11:11 AM, Prince Sibanda
<1princesibanda@gmail.com> wrote:

> However, once one of these two cases starts running, i want to be able to
> issue interactively a command to stop feeding certain types of files from
> the joblist. I also want to be able to prioritise the jobs in joblist so
> that those are run first. I would also like to be able to insert new jobs
> into the joblist with a certain priority level, so that if the inserted is a
> high priority job for example, it is run next as soon as any of the
> currently running jobs has finished. I would like to be able to say skip a
> certain job, or repeat a certain job, take a certain job out of joblist,
> etc. All this i want to be able to do when one of those two cases has
> already started running.

You are describing a job queue system.

GNU Parallel was not built as a job queue system, but can be used as a
very minimal queue.

GNU Parallel is not designed for interactivity - it has very few
interactive features. It is not designed for removing jobs from the
queue, and it has no concept of a priority level.

Extending GNU Parallel to a proper job queue system is outside the
scope of GNU Parallel, and even if someone made a patch for this, I
would probably be reluctant to include it - as it would have to
re-write huge sections of GNU Parallel.

Some GNU Parallel users use Slurm. I would imagine GNU Parallel is
useful for generating and submitting jobs to Slurm, and I would be
open to making a few changes to make GNU Parallel interface better
with Slurm, if there are obvious improvement ideas.

Slurm already has the concept of priority, and it is possible to
remove jobs from the queue, so my guess is that it will be easier for
you to extend Slurm to meet your needs, and I encourage you to see if
Slurm or some of the alternatives meet your needs already.

Other alternatives include Torque and Rocks.


/Ole



reply via email to

[Prev in Thread] Current Thread [Next in Thread]