Re: [rdiff-backup-users] Severe performance degradation

[Jakob Unterwurzacher]

Am 21.12.2010 18:25, schrieb Maarten J H van den Berg:
> 
> Hi there,
> 
> I'm looking for help with a very severe performance problem using
> rdiff-backup. Basically we've bought a new server with more and faster
> resources to replace a 4-year old one. However, rdiff-backup refuses to
> perform on the new server.
> 
> Various tests show that generic disk accesses are much faster on the new
> server, and it has more memory and faster CPU's.
> Nevertheless we see a very severe slowdown when rdiff-backup is making
> an incremental backup, up to four or even tenfold or more times slower
> than it used to be before on the old server.
> 
> I gathered some numbers but they differ wildly depending on the source
> material / dir. Maybe it is therefore better to leave specific numbers
> for what they are for now and focus on the big picture: our old server
> did an rdiff-backup of a remote storage server, worth some 300 GB, in
> typically under an hour or so. The new server running with the same
> source dataset typically starts at night and is still running the next
> morning, into the afternoon even(!).
> 
> When we do trials on a tiny subset of the data we get varying results.
> Some data takes eightfold the amount of time, some is within a +80%
> margin. So that is not very dependable, alas.
> Still, what is observable is that any initial backup run (with --force)
> runs significantly faster on the new server. Any differential run
> afterwards is slower than on the original server. I feel this proves
> there are no performance bottlenecks in the network, disks, filesystems
> etc of the server.
> 
> This is fully repeatable and a real time tail on the log file shows no
> one file is to blame, it is just the overall speed that's slow.
> 
> The new server runs rdiff-backup 1.2.8, the old one 1.0.5. Downgrading
> the new server to 1.0.5 makes things a bit interesting: that speeds it
> up a bit, but still a fair bit slower than the original.
> 
> During investigation we experimented with different filesystems, testing
> local versus remote backups, looking at compile flags and versions of
> librsync and python, but we have had no success there.
> All versions use librsync 0.9.7 All OS'es are Gentoo, 32 bit.
> 
> We did search for workarounds like spawning multiple parallel
> rdiff-backup processes dealing each with separate directories so as to
> fully use the eight CPU cores. Sadly even that speedup is still not
> resulting in an acceptable overall speed. We compared compilation flags,
> options and parameters but nothing obvious struck us in that regard.
> 
> 
> I'm basically out of ideas. I was tearing my hair out over this until a
> couple of days ago (yes, well, I'm bald now). I turn to this list as a
> last resort. Can anyone help debugging this strange problem please ?
> 
> Regards, thanks for listening,
> 
> Maarten



What about the kernel version? Probably barriers are now working as they
should and you pay the price for that. dpkg package management had huge
performance regressions due to this AFAIR.

Other than that, I'd try to find out where rdiff-backup is spending all
the the time. What does
iostat -dx 1
in the %utils column say while an incremental is running?
You could even fire up sysprof to see what it is waiting for.

Regards,
Jakob