[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Spreading parallel across nodes on HPC system

From: Rob Sargent
Subject: Re: Spreading parallel across nodes on HPC system
Date: Fri, 11 Nov 2022 00:37:45 -0700
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.2

On 11/11/22 00:05, Ken Mankoff wrote:
Hi Rob,

On 2022-11-10 at 21:21 +01, Rob Sargent <> wrote:
I do this, in slurm bash script, to get the number of jobs I want to
run (turns out it's better for me to not load the full hyper-threaded

   cores=`grep -c processor /proc/cpuinfo`
   cores=$(( $cores / 2 ))

   parallel --jobs $cores etc :::: <file with list of jobs>

or sometimes the same jobs many times with

   parallel --jobs $cores etc ::: {1..300}
I apologize if I am missing something, but I don't see how this solves distributing to different hosts (nodes), where each host may have a different number of CPUs or cores.


Definition of "job" is part of the problem.  Mine either take over the host (internally multi-threaded) or I use the above to keep the machine busy with a list of jobs (or same job n times)
I asked the local cluster folks if I could queue up all my jobs and request just on core for each job but they preferred I let parallel keep the machine busy:  one slurm job versus hundreds.

How do you mix slurm and parallel hostfile?

reply via email to

[Prev in Thread] Current Thread [Next in Thread]