Re: [Bug-tar] Multi-threaded tar

[Ralph Corderoy]

Hi J.R.,

> I was trying to make some processes faster which use tar a bit on
> large archives.  I was able to use the command line options
> --use-compress-program pbzip2 and that helped a lot.

I think analysis of where the remaining delay is would be needed before
investing effort in removing it.  There's a pipe between tar that's
reading the files, and pbzip2 that's writing the compressed tar file.
tar write(2)s down it 10 KiB at a time.  

pipe(7) on Linux says the pipe's buffer is 64 KiB, but can be altered
with fcntl(2)'s F_SETPIPE_SZ.  That 64 KiB allows the pipe writer to run
ahead of the reader: tar ahead of the compressor, a bigger pipe buffer
might stop the pipe reader being starved, if that's what's happening.

I thought I'd look at the 25/50/75-percentile of time taken to write(2)
down the pipe compared with reading from it.

    $ LC_ALL=C strace -ffe %desc -tttT -o st \
          tar --use-compress-program 'lzip -9' -cf foo.tar.lz 
/boot/vmlinuz-linux
    $
    $ grep ' write(4, ' st.23321 |
    > egrep -o ' <[0-9.]+>$' |
    > tr -d ' <>' |
    > stats
       length      min   25%ile   median     mean   75%ile      max   stddev
          521 0.0000340 0.0000880 0.0000920 0.0001344 0.0000980 0.0197400 
0.0008624761
    $
    $ sed -n '/boot\/vmlinuz-linux/,$p' st.23322 |
    > grep 'read(0, ' |
    > egrep -o ' <[0-9.]+>$' |
    > tr -d ' <>' |
    > stats
       length      min   25%ile   median     mean   75%ile      max   stddev
          516 0.0000490 0.0002430 0.0002480 0.0002580 0.0002572 0.0017330 
6.939183e-05
    $

-- 
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy