Re: using Parallel for several Parallel codes

[Ole Tange]

On Wed, Nov 6, 2013 at 7:25 PM, Hojat Ghorbanidehno
<hojjatgh@stanford.edu> wrote:
> Hi
>
> I want to submit about 100 jobs at the same time, each of them are parallel 
> codes, and I want to give 24 processors to each one( I need to get 2400 
> processors). I was wondering if it is possible to do it with Parallel. All 
> examples that I saw was for serial codes, so I presumed it will give each job 
> just 1 processor but I want to give each job 24 processor.
> is it possible?

GNU Parallel does not decide how many CPUs each job uses. That is done
by the OS and the program. But if your program spawns 24 threads, then
you will be using 2400 threads by running 100 jobs in parallel:

    cat arguments.txt | parallel -j100 my24thread_program

If your computer has 2400 cores then this should use all of your cores.

You can also tell GNU Parallel to figure the number of jobs out by
itself (4% will run one job for every 25 cores in the system):

    cat arguments.txt | parallel -j4% my24thread_program

If you really *do* have a single computer with 2400 cores, I would
love to see a picture of it. Most of us only have that amount of cores
spread over multiple machines in a cluster. If that proves to be the
case for you, too, then read about --sshloginfile.


/Ole