[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Problem with thousands of small jobs.
From: |
xmoon 2000 |
Subject: |
Problem with thousands of small jobs. |
Date: |
Fri, 6 Feb 2015 13:20:06 +0000 |
Hi,
I need to run about 4,000 jobs that each take around 20 seconds to complete
using:
cat /tmp/parList | parallel -j 28 --eta;
On my 32 core machine works OK, BUT there is a "lull" in processing
every few seconds as new jobs are started, once the current crop have
completed. I assume this is due to an overhead in starting jobs that
is only noticable because my jobs are so short.
Is there any way I could make this more efficient, so my cores are
fully utilised and getting through the whole process is faster?
Moon
- Problem with thousands of small jobs.,
xmoon 2000 <=
- Re: Problem with thousands of small jobs., Felipe Alvarez, 2015/02/06
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/06
- Re: Problem with thousands of small jobs., Felipe Alvarez, 2015/02/06
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/06
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/06
- Re: Problem with thousands of small jobs., Rob Sargent, 2015/02/06
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/06
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/06
- Re: Problem with thousands of small jobs., Ole Tange, 2015/02/06
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/07