Re: Processing files from a tar archive in parallel

[Hans Schou]

On Tue, 29 Mar 2011, Benjamin R. Haskell wrote:

> On Tue, 29 Mar 2011, Hans Schou wrote:
> > tar xvf big-file.tar.gz | parallel echo "Proc this file {}"

> Hans, you left off the 'z' in 'tar zxvf':

No, its a new feature in tar (Ubuntu 10). If you specify a file, tar 
will test compression type it self and unzip it.

Try this at home:
 tar zcf htmls.tar.gz $(find /usr/share/doc -name '*.html')

And then do the thing:
 tar xvf htmls.tar.gz | parallel tidy -miq

> Jay, you probably also want to 'rm' the files as you go, since space 
> sounds like an issue.

And write a script which can handle one file, if you have a lot of 
commands to run on one file - or decitions.

/hans
--
Horsebakken 78, DK-2400 København NV