Re: Processing a big file using more CPUs

[Nio Wiklund]

Den 2019-02-11 kl. 23:54, skrev Ole Tange:
> On Mon, Feb 4, 2019 at 10:19 PM Nio Wiklund <nio.wiklund@gmail.com> wrote:
> :
> >     cat bigfile | parallel --pipe --recend '' -k gzip -9 > bigfile.gz
> :
> > The reason why I want this is that I often create compressed images of
> > the content of a drive, /dev/sdx, and I lose approximately half the
> > compression improvement from gzip to xz, when using parallel. The
> > improvement in speed is good, 2.5 times, but I think larger blocks would
> > give xz a chance to get a compression much closer to what it can get
> > without parallel.
> >
> > Is it possible with with the current code? In that case how?
>
> Since version 2016-07-22:
>
> parallel --pipepart -a bigfile --recend '' -k --block -1 xz > bigfile.xz
> parallel --pipepart -a /dev/sdx --recend '' -k --block -1 xz > bigfile.xz
>
> Unfortunately the size computation of block devices only works under GNU/Linux.
>
> (That said: pxz exists, and it may be more relevant to use here).
>
>
> /Ole

Thanks for this reply, Ole,

I will test how your suggested command lines work for me, and also look 
into parallel processing within xz.

Best regards
Nio