[Rdiff-backup-commits] rdiff-backup CHANGELOG rdiff_backup/backup.py

[Andrew Ferguson]

CVSROOT:        /sources/rdiff-backup
Module name:    rdiff-backup
Changes by:     Andrew Ferguson <owsla> 08/01/20 16:38:12

Modified files:
        .              : CHANGELOG 
        rdiff_backup   : backup.py 

Log message:
        Improve the handling of directories with many small files by flushing
        the pipeline more often. Fixes a bug where 
Globals.pipeline_max_length*4-1
        files are in a directory causes KeyError.

CVSWeb URLs:
http://cvs.savannah.gnu.org/viewcvs/rdiff-backup/CHANGELOG?cvsroot=rdiff-backup&r1=1.259&r2=1.260
http://cvs.savannah.gnu.org/viewcvs/rdiff-backup/rdiff_backup/backup.py?cvsroot=rdiff-backup&r1=1.41&r2=1.42

Patches:
Index: CHANGELOG
===================================================================
RCS file: /sources/rdiff-backup/rdiff-backup/CHANGELOG,v
retrieving revision 1.259
retrieving revision 1.260
diff -u -b -r1.259 -r1.260
--- CHANGELOG   9 Jan 2008 19:19:25 -0000       1.259
+++ CHANGELOG   20 Jan 2008 16:38:11 -0000      1.260
@@ -1,6 +1,10 @@
 New in v1.1.16 (????/??/??)
 ---------------------------
 
+Improve the handling of directories with many small files when backing-up
+over a network connection. Thanks to Austin Clements for the test case.
+(Andrew Ferguson)
+
 Change high-bit permissions test to check both files and directories.
 Improves rdiff-backup's support for AFS and closes Debian bug #450409.
 (Patch from Marc Horowitz)

Index: rdiff_backup/backup.py
===================================================================
RCS file: /sources/rdiff-backup/rdiff-backup/rdiff_backup/backup.py,v
retrieving revision 1.41
retrieving revision 1.42
diff -u -b -r1.41 -r1.42
--- rdiff_backup/backup.py      24 Dec 2007 22:39:47 -0000      1.41
+++ rdiff_backup/backup.py      20 Jan 2008 16:38:12 -0000      1.42
@@ -161,18 +161,18 @@
                every so often so it doesn't get congested on destination end.
 
                """
-               flush_threshold = int(Globals.pipeline_max_length/2)
-               num_rorps_skipped = 0
+               flush_threshold = Globals.pipeline_max_length - 2
+               num_rorps_seen = 0
                for src_rorp, dest_rorp in cls.CCPP:
-                       if (src_rorp and dest_rorp and src_rorp == dest_rorp and
+                       if (Globals.backup_reader is not Globals.backup_writer):
+                               num_rorps_seen += 1
+                               if (num_rorps_seen > flush_threshold):
+                                       num_rorps_seen = 0
+                                       yield iterfile.MiscIterFlushRepeat
+                       if not (src_rorp and dest_rorp and src_rorp == 
dest_rorp and
                                (not Globals.preserve_hardlinks or
                                 Hardlink.rorp_eq(src_rorp, dest_rorp))):
-                               num_rorps_skipped += 1
-                               if (Globals.backup_reader is not 
Globals.backup_writer and
-                                       num_rorps_skipped > flush_threshold):
-                                       num_rorps_skipped = 0
-                                       yield iterfile.MiscIterFlushRepeat
-                       else:
+
                                index = src_rorp and src_rorp.index or 
dest_rorp.index
                                sig = cls.get_one_sig(dest_base_rpath, index,
                                                                          
src_rorp, dest_rorp)