Running GNU parallel with PBS

[Erik Petigura]

Hello,

This is my first post, so let me start off by saying thanks for a great tool!  I am using GNU parallel to manage some data-parallel tasks on a cluster where the resources are managed by PBS.  

Here are my two questions:

1) How do I transfer my environment variables to a remote machine using the --slf option

2) How does parallel read in the config file.  I can't seem to set `--slf $PBS_NODEFILE' in the config file.

--

1)

My first issue is that my local environment variables are not getting transfered to remote nodes

local> echo $PYTHONPATH /global/home/users/petigura/Kepler//keppycode//

local> seq 0 15 | parallel -j8 --slf $PBS_NODEFILE 'echo {}; echo $PYTHON_PATH; echo $PWD'

/global/home/users/petigura

15

<snip>

/global/home/users/petigura

10

I can work around this by explicitly sourcing my bashrc file

local> seq 0 15 | parallel -j8 --slf $PBS_NODEFILE 'source ~/.bashrc;echo {}; echo $PYTHONPATH; echo $PWD'

8

/global/home/users/petigura/Kepler//keppycode//

/global/home/users/petigura

4

/global/home/users/petigura/Kepler//keppycode//

/global/home/users/petigura

2)

I'd like to use a config file to simplify my commands, but I can't get it to work.

local> cat ~/.parallel/nodes

#Config file.  Options to parallel when running on multiple nodes

--slf $PBS_NODEFILE

source $HOME/.bashrc;

local > seq 0 15 | parallel -j8 -Jnodes  --wd $PWD 'source ~/.bashrc;echo {}; echo $PYTHONPATH; echo $PWD'

Cannot open $PBS_NODEFILE

--

Thanks for your help!

Erik