Re: Running parallel from perl system() with redirects

[Ole Tange]

On Mon, Jan 12, 2015 at 11:37 PM, Josef wells
<Josefwells@alumni.utexas.net> wrote:

> I have a problem that did not happen with version:
> 20130822
> But does happen with:
> 20141122
>
> This works:
> parallel --joblog - ::: 'cat hello >& hello_log' > joblog 2>&1

Confirmed (example 1).

> But it doesn't work from inside a perl program.
> perl -e "system('parallel --joblog - ::: \'cat hello >& hello_log\' > joblog
> 2>&1');"
> joblog contains the error:
> /bin/sh: 1: Syntax error: Bad fd number

Confirmed (example 2).

Just like this does not work:

  (example 3)  perl -e "system('cat hello >& hello_log');"

but this does (when run from bash):

  (example 4)  perl -e "system('parallel',':::', 'cat hello >& hello_log');"

This is by design.

So let us just analyze what is happening. I assume you use bash. In
example 1 'parallel' is being started from bash. In example 2 you use
'system'. 'system' spawns '/bin/sh' if the command cannot be exec'ed.
If the first argument to system is a command, then exec is used
(example 4); otherwise '/bin/sh -c' is used to parse the argument
(example 3 and 2).

Up to 20140822 GNU Parallel used $SHELL to start commands. The reason
why this was changed was the following:

  #!/usr/bin/tcsh
  # (example 5) Note the tcsh in the #!-line

  setenv a 1; echo $a
  parallel 'setenv a {}; echo $a' ::: 2

What should that script do?

For GNU Parallel < 20140822: if your login $SHELL is /usr/bin/tcsh
then it will print 1 and 2; but if your $SHELL is /bin/bash you will
get 'setenv: command not found'. So the same script will give
different results depending on which login $SHELL a given user has.

For GNU Parallel >= 20140822: it will print 1 and 2 - no matter your
login $SHELL.

For people writing scripts it is unacceptable that different login
$SHELL would break their script: A tcsh user would then not be able to
give a script to a bash user and assume it worked. That is why GNU
Parallel now tries to figure out which shell was used to start
'parallel' and use the same shell to run its commands. It does that by
looking at parallel's (grand*)parent to find a shell. If none are
found $SHELL is used. In example 2 and 3 /bin/sh is found, but in
example 1 and 4 /bin/bash is found. In example 5 /usr/bin/tcsh is
found.

The solution does not solve every case:

  #!/usr/bin/perl
  # (example 6)

  system("parallel",'setenv a {}; echo $a',":::",2);

If that perl script is run from tcsh, it works great, but it fails if
it is run from bash. If you find a way to make this Do The Right
Thing, then I will be happy to hear your ideas.

You are hitting one of the inconveniences by the new practice, because
your $SHELL is bash. For people like you the environment variable
'PARALLEL_SHELL' was introduced:

  perl -e "\$ENV{PARALLEL_SHELL}='/bin/bash'; system('parallel :::
\'cat hello >& hello_log\' ');"
  PARALLEL_SHELL=/bin/bash perl -e "system('parallel ::: \'cat hello
>& hello_log\' ');"

This forces GNU Parallel to use $PARALLEL_SHELL as the shell for
running commands.


/Ole

PS: If you find this complex, then welcome to the world of trying to
please everybody without exasperating too many.