Re: [rdiff-backup-users] Reporting update bug for large files in rdiff-backup (>4Gb)

[David Kempe]

Gerard van Dijnsen wrote:

> This again takes a while and then produces the following output:
> python: ERROR: (rs_job_iter) internal error: job made no progress 
> [orig_in=23, orig_out=65536, final_in=23, final_out=65536]
> UpdateError largefile_4096_MB.dat librsync error 107 while in patch cycle
> python: ERROR: (rs_job_iter) internal error: job made no progress 
> [orig_in=17533, orig_out=65536, final_in=17533, final_out=65536]
> UpdateError largefile_5120_MB.dat librsync error 107 while in patch cycle
these errors look fairly similar to me.

> I have seen this behaviour on my servers on several occasion, and it 
> seems consistent. Also, this has nothing to do with ssh, because we 
> are making a local backup. However, the behaviour over ssh is exactly 
> the same. Now for good measure I used rdiff to see if this would 
> generate a similar error:
>
> $ rdiff signature tmp2/largefile_4096_MB.dat signature.dat
> $ rdiff delta signature.dat tmp/largefile_4096_MB.dat delta.dat
> $ rdiff patch tmp2/largefile_4096_MB.dat delta.dat largefile_4096_MB.dat
> rdiff: ERROR: (rs_job_iter) internal error: job made no progress 
> [orig_in=23, orig_out=16000, final_in=23, final_out=16000]
> rdiff: ERROR: library internal error

i thought if rdiff generated the same error, then its definitely a 
librsync bug. rdiff-backup doesn't use rdiff for anything, despite the name.

Although chatting to Martin Pool (librsync author) a while back, he 
reckons it needs a rewrite and it would take ages to do it properly. 

I tried the same test on a CVS version of rdiff-backup from today, with 
librsync 0.9.7 (ubuntu breezy)
I did not run out of temp space at all.
I believe its a librsync bug, and am pursuing the author to see where 
its up to....

Same problem:
rdiff-backup -v5 temp/ temp-backup
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
 Access control lists                         On
 Extended attributes                          On
 Case sensitivity                             On
 Mac OS X style resource forks                Off
 Mac OS X Finder information                  Off
-----------------------------------------------------------------
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
 Ownership changing                           On
 Hard linking                                 On
 fsync() directories                          On
 Directory inc permissions                    On
 High-bit permissions                         On
 Extended filenames                           On
 Access control lists                         On
 Extended attributes                          On
 Case sensitivity                             On
 Mac OS X style resource forks                Off
 Mac OS X Finder information                  Off
-----------------------------------------------------------------
Starting increment operation temp to temp-backup
Processing changed file .
Incrementing mirror file temp-backup
Processing changed file largefile_1024_MB.dat
Incrementing mirror file temp-backup/largefile_1024_MB.dat
Processing changed file largefile_2048_MB.dat
Incrementing mirror file temp-backup/largefile_2048_MB.dat
Processing changed file largefile_4096_MB.dat
python: ERROR: (rs_job_iter) internal error: job made no progress 
[orig_in=13, orig_out=65536, final_in=13, final_out=65536]
Exception 'librsync error 107 while in patch cycle' raised of class 
'rdiff_backup.librsync.librsyncError':
 File "/usr/lib/python2.4/site-packages/rdiff_backup/robust.py", line 
32, in check_common_error
   try: return function(*args)
 File "/usr/lib/python2.4/site-packages/rdiff_backup/Rdiff.py", line 
98, in patch_local
   if outrp: return outrp.write_from_fileobj(patchfile)
 File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 
1045, in write_from_fileobj
   copyfileobj(fp, outfp)
 File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 
58, in copyfileobj
   inbuf = inputfp.read(blocksize)
 File "/usr/lib/python2.4/site-packages/rdiff_backup/librsync.py", line 
77, in read
   self._add_to_outbuf_once()
 File "/usr/lib/python2.4/site-packages/rdiff_backup/librsync.py", line 
88, in _add_to_outbuf_once
   except _librsync.librsyncError, e: raise librsyncError(str(e))

UpdateError largefile_4096_MB.dat librsync error 107 while in patch cycle
Processing changed file largefile_5120_MB.dat
python: ERROR: (rs_job_iter) internal error: job made no progress 
[orig_in=69754, orig_out=65536, final_in=69754, final_out=65536]
Exception 'librsync error 107 while in patch cycle' raised of class 
'rdiff_backup.librsync.librsyncError':
 File "/usr/lib/python2.4/site-packages/rdiff_backup/robust.py", line 
32, in check_common_error
   try: return function(*args)
 File "/usr/lib/python2.4/site-packages/rdiff_backup/Rdiff.py", line 
98, in patch_local
   if outrp: return outrp.write_from_fileobj(patchfile)
 File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 
1045, in write_from_fileobj
   copyfileobj(fp, outfp)
 File "/usr/lib/python2.4/site-packages/rdiff_backup/rpath.py", line 
58, in copyfileobj
   inbuf = inputfp.read(blocksize)
 File "/usr/lib/python2.4/site-packages/rdiff_backup/librsync.py", line 
77, in read
   self._add_to_outbuf_once()
 File "/usr/lib/python2.4/site-packages/rdiff_backup/librsync.py", line 
88, in _add_to_outbuf_once
   except _librsync.librsyncError, e: raise librsyncError(str(e))

UpdateError largefile_5120_MB.dat librsync error 107 while in patch cycle
Processing changed file largefile_512_MB.dat
Incrementing mirror file temp-backup/largefile_512_MB.dat