[rdiff-backup-users] Re: [librsync-users] more info on 25gig files

[Donovan Baarda]

On Thu, 2005-05-05 at 11:52 -0600, Clint Silvester wrote:
> I tried rdiff last night instead of rdiff-backup on the same 25 gig
> files that rdiff-backup gave me an error 107 and rdiff worked fine on
> the files.  I got a full copy of the newer file (what I was trying to do
> with rdiff-backup).  This was just with standard options, no block size
> changes or anything.  Here are the timings I got:
[...]
> So that's over 4 and a half hours.  I'm thinking maybe I'll need to play
> with block sizes but that only mentions changing the block sizes for the
> signature file, so I don't know if it will make any difference for
> creating the delta.  Anyway, I'll try some more things, but it looks

The block size will have a significant impact on speed, as it "walks"
through the file faster when there are hits. I would recommend using a
blocksize that is the square-root of the file size... ie

1M file, 1K blocksize
1G file, 32K blocksize
4G file, 64K blocksize
16G file, 128K blocksize
64G file, 256K blocksize

(this also reduces the probablity of blocksum collisions, which though
very unlikely, will cause corruption).

However, if you know some characteristics of the file contents, you will
benefit by taking that into account.. if know it is full of fixed sized
records, you should pick a block size that is a multiple of the record
size (taking into account the recommended size above).

> like rdiff doesn't suffer from the same problem rdiff-backup does.  I'm
> still testing and it takes a while to run these.  I'll report back again.

I _think_ rdiff-backup uses an extension module that hooks into librsync
itself. It is possible the rdiff-backup extension module is not
correctly compiled with 64 bit support. I suspect the whole python
interpreter would need to be compiled with 64 bit support.


-- 
Donovan Baarda <address@hidden>
http://minkirri.apana.org.au/~abo/