RE: Processing files from a tar archive in parallel

[Cook, Malcolm]

Ole - thanks for the education - Malcolm
 
 

> -----Original Message-----
> From: ole.tange@gmail.com [mailto:ole.tange@gmail.com] On 
> Behalf Of Ole Tange
> Sent: Tuesday, March 29, 2011 5:03 PM
> To: Cook, Malcolm
> Cc: Jay Hacker; parallel@gnu.org
> Subject: Re: Processing files from a tar archive in parallel
> 
> On Tue, Mar 29, 2011 at 11:41 PM, Cook, Malcolm 
> <MEC@stowers.org> wrote:
> > ooops, more like:
> >
> >        tar -t big-file.tar.gz  | parallel tar -O -x -f 
> big-file.tar.gz 
> > '|' someCommandThatReadsFromStdIn
> 
> You probably mean:
> 
>   tar -tf big-file.tar.gz  | parallel tar -O -x -f 
> big-file.tar.gz {} '|' someCommandThatReadsFromStdIn
> 
> I am afraid this is highly inefficient. If big-file.tar.gz 
> contains 1000 files, then to extract the last file the second 
> tar will have to read through the first 999 files (as you 
> cannot seek into tar.gz files). In total you will be reading 
> big-file.tar.gz 1000*999/2 times.
> 
> 
> /Ole
>