[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[rdiff-backup-users] Wasted bandwidth and accounting for it

From: Chris Wilson
Subject: [rdiff-backup-users] Wasted bandwidth and accounting for it
Date: Wed, 12 May 2010 12:20:24 +0100 (BST)
User-agent: Alpine 1.00 (DEB 882 2007-12-20)

Hi all,

I have been a pretty happy user of rdiff-backup for many years. Currently all servers that I control use rdiff-backup 1.0.x for local and offsite backups. The reason for sticking with an ancient version is simply that I cannout upgrade my backup servers and all clients at the same time, and cross-version compatibility is difficult.

I recently discovered that one of my offsite hosts had been backing up an increasing amount of data each day:


(this host is shown in dark green). When I investigated, the session statistics showed that the change in the destination was small:

$ sudo cat rdiff-backup-data/session_statistics.2010-05-07T00:17:43+01:00.data
StartTime 1273187863.00 (Fri May  7 00:17:43 2010)
EndTime 1273229597.92 (Fri May  7 11:53:17 2010)
ElapsedTime 41734.92 (11 hours 35 minutes 34.92 seconds)
SourceFiles 144698
SourceFileSize 11994143167 (11.2 GB)
MirrorFiles 144697
MirrorFileSize 12043475927 (11.2 GB)
NewFiles 5
NewFileSize 12542425 (12.0 MB)
DeletedFiles 4
DeletedFileSize 61886561 (59.0 MB)
ChangedFiles 74
ChangedSourceSize 14252455 (13.6 MB)
ChangedMirrorSize 14241079 (13.6 MB)
IncrementFiles 84
IncrementFileSize 8256201 (7.87 MB)
TotalDestinationSizeChange -41076559 (-39.2 MB)
Errors 0

Even though this backup took 11.5 hours to finish, and in fact transferred 2.6 GB of data!

I looked into the error log, and this gave me the clue that some huge files (e.g. 2GB Argus packet logs) were being tranferred and then discarded because they were still changing on the sending side.

First of all, the files transferred and then discarded are not logged anywhere in the session statistics. I think it would be useful to track this as a metric. Could I request this as a feature, if 1.2.x doesn't already do it? (we will upgrade eventually).

I think in some cases, for example log files, I would prefer to keep the "corrupted" copy as part of the snapshot, as I know that the file will just be growing and I'd rather not discard and transfer it again every time. Could I request this as a feature for 1.2.x as well?

Perhaps it would even be possible, if the checksum fails, to compare the checksums of the first X bytes (the recorded length when the transfer started), and if these match, to truncate the file on the destination to that length? Could I request that as another feature for 1.2.x?

Cheers, Chris.
_ ___ __     _
 / __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |

reply via email to

[Prev in Thread] Current Thread [Next in Thread]