Query regarding stackoverflow question (Run million of list in PBS with parallel tool)

[Rajiv Gandhi Govindaraj]

Dear Ole Tange,

Thanks for this great tool and for helping me out on this.

We, Computational chemist widely run long/large scale calculations on biological data and GNU parallel is very helpful in achieving it.

As per you suggestion I've introduced a simple function and tested in my local machine which has a 4 cpu for a list contains 10,000: 

top shows 4 running while using --pipepart --block -10k (option 2) but shows only 1-2 running in others ( option 1 & 3):

Here are the completed time in using/not-using function in script:

1. Introducing new function (example_10k.lst contains only 10,000)

job_script.sh

#!/bin/bash

dowork() {

export WDIR=/shared/TF_data/work_dir

cd $WDIR;  

parallel --wd $WDIR sh run_script.sh {}

}

export -f dowork

cat example_10k.lst | dowork

Completes the job in 

real 4m4.599s

user 2m29.206s

sys 0m30.267s

2. Intoducing the new function, --pipepart --block -10k instead of cat in job_script.sh

parallel -a example_10k.lst --pipepart --block -10 dowork

Completes the job in

real 4m44.067s

user 2m58.884s

sys 0m46.153s

3. With No new functions

#!/bin/bash

export WDIR=/shared/TF_data/work_dir

cd $WDIR;  

cat example_10k.lst | parallel --wd $WDIR sh run_script.sh {}

Completes the job in

real 4m5.139s

user 2m28.339s

sys 0m30.723s

Since i tested in 4 cpu local machine, wonder whether testing on more cpu (72 x 5 nodes) or full list (200k) can get a better cpu utilzation ?

Any suggestion is much appreciated. 

Best,

Rajiv