[rdiff-backup-users] OverflowError reading get_sigs() response in very large tree

[Charles Duffy]

Running a backup with a very large number of files (about 150,000) on 
both ends, one of the RPC requests returns a response larger than 
PY_SSIZE_T_MAX:

Traceback (most recent call last):
 File "/usr/bin/rdiff-backup", line 23, in ?
   rdiff_backup.Main.Main(sys.argv[1:])
 File "/usr/lib64/python2.3/site-packages/rdiff_backup/Main.py", line 
286, in Main
   take_action(rps)
 File "/usr/lib64/python2.3/site-packages/rdiff_backup/Main.py", line 
256, in take_action
   elif action == "backup": Backup(rps[0], rps[1])
 File "/usr/lib64/python2.3/site-packages/rdiff_backup/Main.py", line 
306, in Backup
   backup.Mirror_and_increment(rpin, rpout, incdir)
 File "/usr/lib64/python2.3/site-packages/rdiff_backup/backup.py", line 
49, in Mirror_and_increment
   dest_sigiter = DestS.get_sigs(dest_rpath)
 File "/usr/lib64/python2.3/site-packages/rdiff_backup/connection.py", 
line 445, in __call__
   return apply(self.connection.reval, (self.name,) + args)
 File "/usr/lib64/python2.3/site-packages/rdiff_backup/connection.py", 
line 365, in reval
   result = self.get_response(req_num)
 File "/usr/lib64/python2.3/site-packages/rdiff_backup/connection.py", 
line 314, in get_response
   try: req_num, object = self._get()
 File "/usr/lib64/python2.3/site-packages/rdiff_backup/connection.py", 
line 239, in _get
   data = self._read(length)
 File "/usr/lib64/python2.3/site-packages/rdiff_backup/connection.py", 
line 208, in _read
   return self.inpipe.read(length)
OverflowError: requested number of bytes is more than a Python string 
can hold

This is, needless to say, a Bad Thing. There are certainly a few fixes 
possible (such as chunking the signatures -- perhaps by directory)... 
but if anyone can think of something better (or is familiar with 
upstream Python changes which might impact this issue, or otherwise has 
something to add/suggest), I'm very much all ears.