Re: Processing files from a tar archive in parallel

[Jay Hacker]

Hans,

That is a great idea.  However, can I be sure the file is completely
written to disk before tar prints the filename?  It seems to print the
filename first.  Could that not lead to a race condition, or the
consumer reaching the "end" of the file before tar has finished
writing it?



On Tue, Mar 29, 2011 at 11:20 AM, Hans Schou <chlor@schou.dk> wrote:
> On Tue, 29 Mar 2011, Jay Hacker wrote:
>
>> I have a large gzipped tar archive containing many small files; just
>> untarring it takes a lot of time and space.  I'd like to be able to process
>> each file in the archive, ideally without untarring the whole thing first,
>> and I'd like to process several files in parallel.  Is there a recipe for
>> this with GNU Parallel?
>
> tar xvf big-file.tar.gz | parallel echo "Proc this file {}"
>
> Parallel will start when the first file is untared.
>
> /hans
> --
> Horsebakken 78, DK-2400 København NV
>