[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Problem with thousands of small jobs.
From: |
xmoon 2000 |
Subject: |
Re: Problem with thousands of small jobs. |
Date: |
Fri, 6 Feb 2015 17:44:10 +0000 |
2 x ( 8 core / 16 thread)
On 6 February 2015 at 16:40, Rob Sargent <robjsargent@gmail.com> wrote:
> What is your actual cpu configurations? eg 16 dual cores, or 8 quad core
> not hyperthreaded, or 8 dual core hyperthreaded or whatever.
>
> On 02/06/2015 08:39 AM, xmoon 2000 wrote:
>>
>> Do you think the arguments to the parallel command are OK?
>>
>> On 6 February 2015 at 14:58, Felipe Alvarez <felipe.alvarez@gmail.com>
>> wrote:
>>>
>>> echo $SHELL
>>>
>>> Bash is default in cygwin
>>>
>>>
>>> On Sat, 7 Feb 2015 00:56 xmoon 2000 <xmoon2000@googlemail.com> wrote:
>>>>
>>>> I am running this in cygwin. I call parallel from the standard
>>>> window/shell supplied by cygwin.
>>>>
>>>> What makes you think it is bash? (You may well be right)
>>>>
>>>> On 6 February 2015 at 14:43, Felipe Alvarez <felipe.alvarez@gmail.com>
>>>> wrote:
>>>>>
>>>>> Is there a"lighter" shell that you can use other than bash?
>>>>>
>>>>>
>>>>> On Fri, 6 Feb 2015 23:20 xmoon 2000 <xmoon2000@googlemail.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I need to run about 4,000 jobs that each take around 20 seconds to
>>>>>> complete
>>>>>>
>>>>>> using:
>>>>>>
>>>>>> cat /tmp/parList | parallel -j 28 --eta;
>>>>>>
>>>>>> On my 32 core machine works OK, BUT there is a "lull" in processing
>>>>>> every few seconds as new jobs are started, once the current crop have
>>>>>> completed. I assume this is due to an overhead in starting jobs that
>>>>>> is only noticable because my jobs are so short.
>>>>>>
>>>>>> Is there any way I could make this more efficient, so my cores are
>>>>>> fully utilised and getting through the whole process is faster?
>>>>>>
>>>>>> Moon
>>>>>>
>
>
- Problem with thousands of small jobs., xmoon 2000, 2015/02/06
- Re: Problem with thousands of small jobs., Felipe Alvarez, 2015/02/06
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/06
- Re: Problem with thousands of small jobs., Felipe Alvarez, 2015/02/06
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/06
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/06
- Re: Problem with thousands of small jobs., Rob Sargent, 2015/02/06
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/06
- Re: Problem with thousands of small jobs.,
xmoon 2000 <=
- Re: Problem with thousands of small jobs., Ole Tange, 2015/02/06
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/07
- Re: Problem with thousands of small jobs., Ole Tange, 2015/02/07
- Re: Problem with thousands of small jobs., xmoon 2000, 2015/02/10
- Re: Problem with thousands of small jobs., Felipe Alvarez, 2015/02/10
Re: Problem with thousands of small jobs., Ole Tange, 2015/02/11