[Duplicity-talk] Restore crashes in robust.py

[Simon Blandford]

robust.py seems to be handling an error incorrectly by passing a string 
to errno.errorcode instead of an error code.

There are no errors reported when making the backup. Only with the 
restore. It's a new, full backup.

I am unable to restore a backup from S3 without a fatal crash. I 
urgently need help finding a solution because I effectively have very 
little usable backup right now. I can accept one or two files not being 
restored correctly but not with the restore process itself braking 
without recovery. I recommend anyone else making backups to S3 to check 
that they can actually fully restore the data from time to time, even if 
the backup reported no errors.

I am using the following version of Duplicity and dependencies.
duplicity-0.5.03
librsync-0.9.7
pexpect-2.3
GnuPGInterface-0.3.2
ncftp-3.2.1
python-boto-1.6a
gnupg-1.4.5

I have re-made the backup using the above package versions and tried to 
restore it to an EC2 machine (so that the restore is fast and cheap).

The crash always happens at the same point in a particular backup. In 
the latest case at about 6GB into it out of about 50GB. The backup 
includes some very large webcast video media files, up to a few 100MB each.

If I look at what robust.py is trying to do I find that an "if" 
statement is failing because a string is passed as an error code. I 
break just before the offending if statement and look at the variables.

1) The variable being tested...
(Pdb) p exc
<exceptions.IOError instance at 0x1e66758>

2) The first test in the "if" statement...
(Pdb) p not isinstance(exc, EnvironmentError)
False

3) The second test in the "if" statement
(Pdb) p p errno.errorcode[exc[0]] in ['EPERM', 'ENOENT', 'EACCES', 
'EBUSY', 'EEXIST', 'ENOTDIR', 'ENAMETOOLONG', 'EINTR', 'ENOTEMPTY', 
'EIO', 'ETXTBSY', 'ESRCH', 'EINVAL']
*** SyntaxError: <exceptions.SyntaxError instance at 0x1e66dd0>

4) The content of exc[0]
(Pdb) p exc[0]
'GnuPG exited non-zero, with code 131072'


I have invoked pdb at the point at which duplicity was crashing in 
robust.py with pdb.set_trace() like so...

import pdb

def check_common_error(error_handler, function, args = ()):
   """Apply function to args, if error, run error_handler on exception

   This only catches certain exceptions which seem innocent
   enough.

   """
   # todo: import here to avoid circular dependency issue
   import duplicity.path as path

   try: return function(*args)
   #except (EnvironmentError, SkipFileException, DSRPPermError,
   #       RPathException, Rdiff.RdiffException,
   #       librsync.librsyncError, C.UnknownFileTypeError), exc:
   #   TracebackArchive.add()
   except (EnvironmentError, librsync.librsyncError, 
path.PathException), exc:
       pdb.set_trace()
       if (not isinstance(exc, EnvironmentError) or
           (errno.errorcode[exc[0]] in
            ['EPERM', 'ENOENT', 'EACCES', 'EBUSY', 'EEXIST',
             'ENOTDIR', 'ENAMETOOLONG', 'EINTR', 'ENOTEMPTY',
             'EIO', 'ETXTBSY', 'ESRCH', 'EINVAL'])):
           #Log.exception()
           if error_handler: return error_handler(exc, *args)
       else:
           #Log.exception(1, 2)
           raise