Re: Regarding GNU parallel (was: Re: Debian packaging)

[Ole Tange]

2010/12/3 Rogério Brito <rbrito@ime.usp.br>:

> Oh, just a generic question/comment about the usage of GNU parallel:
>
> Let's say that one have a lot of files to work on, with just a single
> command (to make things simpler). And let's suppose that:
>
> * the files reside in a central NFS server;
> * the files are big, so transferring them back and forth can negativate the
> effects of GNU parallel;
> * we want to process the files in the central NFS server with multiple (say,
> 10 hosts), and they all have the same commands installed, but they are of
> different architectures;
> * as expected, there is one host were we will issue the commands.
>
> Is there any recommended/standard way of invoking GNU parallel for this
> case?

If we assume that the NFS is mounted on the same dir on all the
compute hosts, then you should be able to do:

find /nfs/the/files/dir | parallel -j+0 -S host1,host2...host10 command

So you will simply ignore the --transfer --return --cleanup. GNU
Parallel is designed to not care about architecture. If it does _not_
work, then it is a bug.

If the NFS is slow so you are waiting for it, it can often help to
spawn a few more processes: -j+4. Or you can combine parallel and sem:
http://lists.gnu.org/archive/html/parallel/2010-11/msg00020.html


/Ole