RE: [EXTERNAL] Re: Linux kernel question, pre 2.6.23 vs later kernel versions

[Groom, George]

MCVE

Extract the 2 attachments in the same directory.  Execute milti-processor-examples shell script.  Needs sudo access to pull machine profile.  Change path to linux dictionary in script as needed.

Script is straight forward as it forks “grep ‘string’ dict” processes for all words in word_list.

 

Actual output from my 2 HP systems are below.  Note execution time differences between later and older kernels.  Runtimes are quicker on an older machine with older kernel (process scheduling?). Note runtimes of xargs and for-loop are much quicker!

 

Thank you for your time,

 

George Groom

 

 

 

Comparing runtime of serial and 3 parallel processing methods...parallel, xargs and a for-loop

 

Run mode: cat

 

Kernel

3.10.0-693.17.1.el7.x86_64

 

Machine profile

                HTT (Multi-threading)

        Version: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

        Core Count: 22

        Core Enabled: 22

        Thread Count: 44

                Multi-Core

                Hardware Thread

                HTT (Multi-threading)

        Version: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

        Core Count: 22

        Core Enabled: 22

        Thread Count: 44

                Multi-Core

                Hardware Thread

        Product Name: Synergy 480 Gen9

        Product Name: Synergy 480 Gen9 Compute Module

        String 2: Product ID: 732350-B21

 

 

serially... in this mode processing takes too long...skipping

 

parallel...

 

real    0m36.438s

user    2m40.102s

sys     0m29.745s

 

xargs...

 

real    0m28.718s

user    2m26.768s

sys     0m13.546s

 

for-loop...

 

real    0m29.858s

user    2m25.937s

sys     0m9.881s

 

 

 

 

Comparing runtime of serial and 3 parallel processing methods...parallel, xargs and a for-loop

 

Run mode: cat

 

Kernel

2.6.18-238.el5

 

Machine profile

        Product Name: ProLiant DL360 G7

                HTT (Multi-threading)

        Version: Intel(R) Xeon(R) CPU E5649 @ 2.53GHz           

        Core Count: 6

        Core Enabled: 6

        Thread Count: 12

                HTT (Multi-threading)

        Version: Intel(R) Xeon(R) CPU E5649 @ 2.53GHz           

        Core Count: 6

        Core Enabled: 6

        Thread Count: 12

        String 2: Product ID: 579237-B21     

 

 

serially... in this mode processing takes too long...skipping

 

parallel...

 

real    0m23.732s

user    1m33.362s

sys     0m34.111s

 

xargs...

 

real    0m4.950s

user    1m16.905s

sys     0m16.935s

 

for-loop...

 

real    0m3.667s

user    1m2.713s

sys     0m12.475s

 

 

 

 

 

 

 

 

 

-----Original Message-----
From: Ole Tange [mailto:ole@tange.dk]
Sent: Sunday, October 21, 2018 12:53 PM
To: Groom, George
Cc: parallel
Subject: [EXTERNAL] Re: Linux kernel question, pre 2.6.23 vs later kernel versions

 

On Wed, Oct 17, 2018 at 10:25 PM Groom, George <George_Groom@comcast.com> wrote:

 

> I understand that kernel process scheduling changed at 2.6.23… run queue based to Completely Fair Scheduling (CFS).  Has anyone experienced longer parallel script execute times on systems with later versions of the Linux kernel?

> 

> I’ve noticed longer execute times where I’m generating several thousands of processes on later kernel machines.  CFS seems to throttle my processes within a few seconds and then control execution until script completion.  Are there known workarounds for this behavior?  Depending on application needs the older method might be preferable?

 

I cannot reproduce this. Make an MCVE.

 

/Ole

multi-processor-examples
Description: multi-processor-examples

word_list
Description: word_list