[rdiff-backup-users] Incremental backup runs extremely long time

[Sai kee Wong]

Further to my recent report on 2012-10-17 about the fix to make rdiff-backup
works on Mac OS X.  A backup was started on an iMac for 210GB of data, so
far so good, the initial backup was 3 hrs and incr backup around 15 min
which is reasonable for normal home user.

A month later, I started another backup on another iMac for 300GB of data.

Have the following observations:

(1) Discovered that the --exclude doesn't work if the directory doesn't
    exist.

(2) Discovered that the --exclude doesn't work for wildcard like
       --exclude "/**.Spotlight-V100" \
       --exclude "/automount**" \
       --exclude "/cores**" \
       --exclude "/Network**" \
       --exclude "/private/tmp**" \
       --exclude "/private/var/vm**" \
       --exclude "/tmp**" \
       --exclude "/Volumes/**" \
    nor
       --exclude "/tmp*" \
    but only works for
        --exclude "/**/.Spotlight-V100" \
        --exclude "/**/Downloads" \
        --exclude "/**/.Trash" \
        --exclude "/**/Library/Caches" \
        --exclude "/**/tmp" \

(3) The initial backup was 4 hrs (reasonable) but every incr backup
    takes 10 hrs which is abnormal, and it runs at 100% of a CPU 
    core (the mac has quad core Intel CPU). As a normal home user,
    there is only less than 1GB changes everyday.  Following is
    the session statistics.

    StartTime 1355880519.00 (Wed Dec 19 09:28:39 2012)
    EndTime 1355917466.22 (Wed Dec 19 19:44:26 2012)
    ElapsedTime 36947.22 (10 hours 15 minutes 47.22 seconds)
    SourceFiles 1178810
    SourceFileSize 311932438922 (291 GB)
    MirrorFiles 1178805
    MirrorFileSize 311931930965 (291 GB)
    NewFiles 35
    NewFileSize 906566 (885 KB)
    DeletedFiles 30
    DeletedFileSize 144514 (141 KB)
    ChangedFiles 338
    ChangedSourceSize 160572782 (153 MB)
    ChangedMirrorSize 160826877 (153 MB)
    IncrementFiles 415
    IncrementFileSize 7228028 (6.89 MB)
    TotalDestinationSizeChange 7735985 (7.38 MB)
    Errors 0

(4) During the initial backup, it says about 200 files with error like:

    UpdateError Users/skwong/gtk/inst/bin/automake-1.10 Updated mirror temp 
file /Volumes/BK2/Users/skwong/gtk/inst/bin/rdiff-backup.tmp.889172 does not 
match source

    but I can open the source file without problem.  And the 
    statistics as shown above also says Errors 0.

    During the incremental backup, same messages appear for those
    files again.

(5) I rerun the whole backup from 0 again, then notice that it runs
    only at 11% of a CPU core but the inactive memory grow from 0
    to 10GB (until total of the 16GB are all used) within 10 min.

For (1) and (2), may be I mis-understand the operations about
rdiff-backup, and might not be a problem.  For (5), inactive memory
doesn't really matter as the system can reclaim it as needed, but
just wonder if that's normal.

For (3), that's really out of my expectation.  Not only it takes
2.5x more than the initial new backup time, if CPU usage is also
taking into consideration, then, that is 25x; but also that make
the backup not practical.  Also, as the backup on the other iMac
runs quite normal, I really doubt there may be problem in my files
which may trigger hidden bug.  Can anyone suggest on how to debug
the problem.

Also, don't know what's wrong with (4).

Thanks in advance.

SK