[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [rdiff-backup-users] What are the bottlenecks in --verify? (Or how t

From: Thomas Harold
Subject: Re: [rdiff-backup-users] What are the bottlenecks in --verify? (Or how to speed up verification?)
Date: Thu, 03 Oct 2013 13:11:31 -0400
User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8

On 10/3/2013 9:52 AM, Dominic Raferd wrote:
There have been discussions about verification speeds and issues here
before. I think you are right that the CPU core is a bottleneck as
rdiff-backup only uses one core when running a verification.

Verifications do use temporary space and can use a lot of it even though
the temporary files never seem to be visible in the filesystem. I would
advise an explicit --tempdir setting, and a different spindle with lots
of space (and speed) would be ideal.

Yeah, I think the verifications definitely can benefit from having a spindle dedicated to /tmp (or --tempdir setting). We added a 2nd set of small fast spindles, moved /tmp over, and it seems to have dramatically sped up the verify speed. With "atop" we can see rates of 80-120 MB/s being written to the /tmp file system now that it is on its own spindle. So it's getting a lot of use.

From what I can tell, we're still limited by the spindle speeds of both the place where the rdiff-backup directory is stored, and also the speed of the /tmp (or --tempdir) location.

Our Opteron 6348 (12-core 2.8GHz) doesn't seem to have trouble calculating the SHA1 hashes that rdiff-backup uses.

I don't have hard numbers on rdiff-backup itself, but our script to rsync the rdiff-backup files off to an external USB3 drive now takes about 12 hours instead of 18 hours. It also does --verify of the source directory prior to the rsync, then does a --verify of the target directory after the rsync. I have to examine things again to see if I can cut that time window down to about 4-5 hours.

The next step for us will be to move the rdiff-backup directories to a PV with more and faster spindles.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]