Re: GNU Parallel seems to drop

[Dirk Eddelbuettel]

Here is more concrete example:

$ wc -l data.txt 
1000000 data.txt
$ cat data.txt | awk -v path=dataSerial '{print $0 > (path "/" $1 ".txt")}'
$ cat dataSerial/*txt | wc -l
1000000
$ cat data.txt | parallel --pipe -- awk -v path=dataParallel -f script.awk
edd@max:/tmp/parallel$ cat dataParallel/*txt | wc -l
50040
edd@max:/tmp/parallel$ 
edd@max:/tmp/parallel$ cat script.awk 
{ 
    print $0 > (path "/" $1 ".txt")
}
edd@max:/tmp/parallel$ 

Now, am I being silly and just falling into the trap of a race condition 
where my data is being lost when written?

Dirk