Re: Issues using --pipe

[Ole Tange]

On Tue, Dec 13, 2011 at 10:45 PM, Jay Hacker <jayqhacker@gmail.com> wrote:

> I'm not sure if this is my own misunderstanding of --pipe or bugs, but
> maybe somebody can help me.  I'm using version 20111122 on CentOS 5.6
> x86-64.
>
> First, if I forget the job, I get a weird message and parallel hangs:
>
> $ seq -w 10 | parallel --pipe
> Deep recursion on subroutine "CommandLineQueue::get" at
> /usr/bin/parallel line 4190.

With --pipe you need to give parallel a command: What command do you
want to pipe stuff into?

But it is a good point that the error message should be better, and
parallel should not just hang, but exit with a reasonable error. How
about:

parallel: --pipe requires a command to pipe data into (e.g. cat).

https://savannah.gnu.org/bugs/index.php?35069

> Second, it seems to be launching jobs for input blocks that don't
> exist; the number of times the 'EOF' marker gets printed below changes
> from run to run:

That is a known bug: http://savannah.gnu.org/bugs/?34241

The main problem here is that for efficiency GNU Parallel spawns a new
job as soon as an old job dies, thus it does so before it knows
whether there is more data to send. This way the initialization of the
job can be done in the background while data is being read.

Unfortunately this causes problems for commands that cannot run on
empty input (such as md5sum mentioned in the bug).

I have spend a little time to see how this could be changed but did
not find an easy fix.


/Ole