Re: Combining --pipe and --shebang options

[Ole Tange]

On Sun, Nov 18, 2012 at 3:13 PM, Ole Tange <ole@tange.dk> wrote:
> On Fri, Nov 16, 2012 at 2:13 PM, Michel Samia
> <michel.samia@firma.seznam.cz> wrote:
>
>> is it possible to combine --shebang and --pipe options?
>
> It ought to be. And your script shows a good reason why. But currently
> it seems you cannot.
>
> I have created this bugreport https://savannah.gnu.org/bugs/index.php?37766

It actually is possible to combine --shebang and --pipe - but not the
way you want.

--shebang considers the rest of the file as data as if it was received
on -a or through stdin.

#!/usr/local/bin/parallel --shebang --pipe -k -j24 cat
a
b
c
d

What you want is to pass the rest of the file to python and let
parallel chunk up stdin to the python script.

I really like your idea, but it is clearly not a bug that it does not
work currently.

Your idea would be useful for any script (Shell, Perl, Python) which
can either process only one file or process stuff from stdin. Using
GNU Parallel your script can suddenly process many files/blocks of
data and in parallel.

I do not see we can change the behaviour of --shebang, but we can
invent a new option.

So it could be something like:

#!/usr/bin/parallel --shebang-program --pipe -k -j24 /usr/bin/python

(Please come up with a better name than  --shebang-program)

This should accept data on stdin which should be chunked and passed to
the python program. So the program will be called like:

  cat foo bar | my_program

or:

  my_program foo bar

Without the --pipe:

#!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/python

should work like:

  parallel -k -j24 /usr/bin/python my_program {}

So:

  my_program foo bar
  (echo foo; echo bar) | my_program

should do the same as:

  parallel -k -j24 /usr/bin/python my_program {} ::: foo bar

We should allow for putting options on the command interpreter. E.g:

#!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p

Also without the --pipe the {} should work as expected:

#!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p {}
{.}.out > {.}.log

Are there things I am not covering? Are there use cases that this will
not cover?


/Ole