[rdiff-backup-users] Wasted bandwidth and accounting for it

[Chris Wilson]

Hi all,

I have been a pretty happy user of rdiff-backup for many years. Currently 
all servers that I control use rdiff-backup 1.0.x for local and offsite 
backups. The reason for sticking with an ancient version is simply that I 
cannout upgrade my backup servers and all clients at the same time, and 
cross-version compatibility is difficult.

I recently discovered that one of my offsite hosts had been backing up an 
increasing amount of data each day:

  http://i41.tinypic.com/2qajtyx.png

(this host is shown in dark green). When I investigated, the session 
statistics showed that the change in the destination was small:

$ sudo cat 
rdiff-backup-data/session_statistics.2010-05-07T00:17:43+01:00.data
StartTime 1273187863.00 (Fri May  7 00:17:43 2010)
EndTime 1273229597.92 (Fri May  7 11:53:17 2010)
ElapsedTime 41734.92 (11 hours 35 minutes 34.92 seconds)
SourceFiles 144698
SourceFileSize 11994143167 (11.2 GB)
MirrorFiles 144697
MirrorFileSize 12043475927 (11.2 GB)
NewFiles 5
NewFileSize 12542425 (12.0 MB)
DeletedFiles 4
DeletedFileSize 61886561 (59.0 MB)
ChangedFiles 74
ChangedSourceSize 14252455 (13.6 MB)
ChangedMirrorSize 14241079 (13.6 MB)
IncrementFiles 84
IncrementFileSize 8256201 (7.87 MB)
TotalDestinationSizeChange -41076559 (-39.2 MB)
Errors 0

Even though this backup took 11.5 hours to finish, and in fact transferred 
2.6 GB of data!

I looked into the error log, and this gave me the clue that some huge 
files (e.g. 2GB Argus packet logs) were being tranferred and then 
discarded because they were still changing on the sending side.

First of all, the files transferred and then discarded are not logged 
anywhere in the session statistics. I think it would be useful to track 
this as a metric. Could I request this as a feature, if 1.2.x doesn't 
already do it? (we will upgrade eventually).

I think in some cases, for example log files, I would prefer to keep the 
"corrupted" copy as part of the snapshot, as I know that the file will 
just be growing and I'd rather not discard and transfer it again every 
time. Could I request this as a feature for 1.2.x as well?

Perhaps it would even be possible, if the checksum fails, to compare the 
checksums of the first X bytes (the recorded length when the transfer 
started), and if these match, to truncate the file on the destination to 
that length? Could I request that as another feature for 1.2.x?

Cheers, Chris.
--
_ ___ __     _
 / __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |