Re: newbie ?

[ed bierly]

thank you very much Ole

could you point me to where i can get this capability for ubuntu 16.10

GNU parallel 20141022

thanks

ed

On Thu, Apr 6, 2017 at 6:04 PM, Ole Tange <ole@tange.dk> wrote:

On Tue, Apr 4, 2017 at 10:22 PM, ed bierly <ebierly@gmail.com> wrote:

> my goal is to use this stdout so i think i need --pipe in order to tee out a
> single stream
> pbzip2 -cd 25Gzipped.bz2|parallel --pipe ...
>
> the awk program would now be like this
> {if(a==$1)print $0>$1}

What you are looking for is '--pipe --tee':

  seq 3 | parallel --tee --pipe awk -v a={} -f a.awk ::: 2 3

It is exactly made for your situation where the input is slow to generate.

--tee is very new and is in alpha test, so please give feedback.

There are some limitations:

Every job uses 4 file handles, so on a normal GNU/Linux system you
will be able to run 250 jobs in parallel. This can be raised with
`ulimit -n 65535` giving 16300 jobs.

All jobs will be run in parallel. This is due to the input not being
saved, but sent directly to the jobs.


/Ole