Re: xargs -P and load balancers

[James Youngman]

On Fri, Oct 10, 2008 at 7:26 AM, Sven Hartrumpf
<address@hidden> wrote:
> Hi all.
>
> If I use xargs -P for parallel excution of commands distributed
> by ssh over several machines, the start-up by xargs is too
> fast (yes, not too slow :-) ) so that the load balancer
> (that selects the best node for the current ssh job) will not
> catch up with the newly started processes.
> Therefore the processes are badly distributed.
>
> A solution would be be an xargs option like
> --delay-parallel-start=<SECS>
> which waits <SECS> seconds before starting a new parallel job.
>
> How can this be achieved currently?

I have seen good solutions to this problem that use pull-based
approaches; machines collect chunks of work from some shared storage,
complete it, and then go back for more.   The advantage of this
approach is that it works well for situations where backends are
unequal in performance, or some of the machines have an independent
workload.  There are downsides too though, which is that there is a
certain amount of additional complexity and you need to guard against
cases like having 0 remaining backends...

James.