[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Distributed parallel: how to measure how many cores are really used
From: |
Ole Tange |
Subject: |
Re: Distributed parallel: how to measure how many cores are really used on remote machine? |
Date: |
Mon, 13 Dec 2010 11:06:30 +0100 |
On Mon, Dec 13, 2010 at 8:54 AM, Maciej Pilichowski
<pilichowski.maciej@gmail.com> wrote:
> Hello,
>
> I have local computer with 2 cores, and remote one with 4 cores. I run
> parallel always with -j+0.
>
> It seems that around 50% of the job is done locally, and 50% on remote
> machine. While I would expect around 20% and 80%.
This can be because it is much faster to start a job on the local
machine. If your jobs only take a few seconds or if your jobs take a
few minutes but transfer files, then the startup time can be
overwhelming.
A quick test is:
seq 1 20 | parallel --progress -S .. sleep
The workaround may be something like:
parallel -j+2 sem -j+0 your_command
That will start 2 extra jobs but sem will only start one per cpu. The
remaining 2 jobs are allowed to do file transfer, though.
> I am wondering if parallel checks the cores locally, gets the answer, 2,
> and so running 2 tasks on remote machine too. Or if the transferring files
> is so limiting.
>
> So -- how to measure how many cores are really used on remote machine?
GNU Parallel runs this command on the remote machine to figure it out:
parallel --number-of-cores
/Ole