[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Processing files from a tar archive in parallel
From: |
Ole Tange |
Subject: |
Re: Processing files from a tar archive in parallel |
Date: |
Wed, 30 Mar 2011 00:02:31 +0200 |
On Tue, Mar 29, 2011 at 11:41 PM, Cook, Malcolm <MEC@stowers.org> wrote:
> ooops, more like:
>
> tar -t big-file.tar.gz | parallel tar -O -x -f big-file.tar.gz '|'
> someCommandThatReadsFromStdIn
You probably mean:
tar -tf big-file.tar.gz | parallel tar -O -x -f big-file.tar.gz {}
'|' someCommandThatReadsFromStdIn
I am afraid this is highly inefficient. If big-file.tar.gz contains
1000 files, then to extract the last file the second tar will have to
read through the first 999 files (as you cannot seek into tar.gz
files). In total you will be reading big-file.tar.gz 1000*999/2 times.
/Ole
- Processing files from a tar archive in parallel, Jay Hacker, 2011/03/29
- Re: Processing files from a tar archive in parallel, Hans Schou, 2011/03/29
- Re: Processing files from a tar archive in parallel, Jay Hacker, 2011/03/29
- Re: Processing files from a tar archive in parallel, Hans Schou, 2011/03/29
- Re: Processing files from a tar archive in parallel, Ole Tange, 2011/03/29
- RE: Processing files from a tar archive in parallel, Cook, Malcolm, 2011/03/29
- RE: Processing files from a tar archive in parallel, Cook, Malcolm, 2011/03/29
- Re: Processing files from a tar archive in parallel,
Ole Tange <=
- RE: Processing files from a tar archive in parallel, Cook, Malcolm, 2011/03/30
- Re: Processing files from a tar archive in parallel, Jay Hacker, 2011/03/30
- Re: Processing files from a tar archive in parallel, Hans Schou, 2011/03/29
- Re: Processing files from a tar archive in parallel, Ole Tange, 2011/03/29
- Re: Processing files from a tar archive in parallel, Benjamin R. Haskell, 2011/03/30
Re: Processing files from a tar archive in parallel, Benjamin R. Haskell, 2011/03/30
Re: Processing files from a tar archive in parallel, Benjamin R. Haskell, 2011/03/30