[Duplicity-talk] Re: --num-retries doesn't work when fetching signature files

[Russell Clemings]

I see this regularly as well and can't figure it out. Sometimes it
works fine from a cron job; sometimes I have to run it manually, and
even then it can take several tries to succeed.

I have --timeout=120 and --num-retries=15 ... but it never gets that
high. Duplicity is v0.5.02 (yes, I know I need to upgrade); boto is
0.9d; python obviously 2.4.

Here's the most recent attempt:

Last full backup date: Sat Jan  3 13:10:25 2009
Using temporary directory /s3backuptmp/duplicity--hYU2R-tempdir
Downloading 
s3+http://mybucketname/duplicity-inc.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.manifest.gpg
gpg: encrypted with 2048-bit ELG-E key, ID E07FG162, created 2007-11-15
     "Myusername (Key for duplicity) <address@hidden>"
gpg: Signature made Wed Jan  7 00:00:26 2009 EST using DSA key ID 9683957A
gpg: Good signature from "Myusername (Key for duplicity) <address@hidden>"
Downloading 
s3+http://mybucketname/duplicity-full-signatures.2009-01-03T13:10:25-05:00.sigtar.gpg
Downloading 
s3+http://mybucketname/duplicity-new-signatures.2009-01-03T13:10:25-05:00.to.2009-01-04T00:46:09-05:00.sigtar.gpg
Downloading 
s3+http://mybucketname/duplicity-new-signatures.2009-01-04T00:46:09-05:00.to.2009-01-05T00:06:17-05:00.sigtar.gpg
Downloading 
s3+http://mybucketname/duplicity-new-signatures.2009-01-05T00:06:17-05:00.to.2009-01-06T00:22:46-05:00.sigtar.gpg
Downloading 
s3+http://mybucketname/duplicity-new-signatures.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.sigtar.gpg
Download 
s3+http://mybucketname/duplicity-new-signatures.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.sigtar.gpg
failed (attempt #1)
Downloading 
s3+http://mybucketname/duplicity-new-signatures.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.sigtar.gpg
Download 
s3+http://mybucketname/duplicity-new-signatures.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.sigtar.gpg
failed (attempt #2)
Downloading 
s3+http://mybucketname/duplicity-new-signatures.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.sigtar.gpg
Download 
s3+http://mybucketname/duplicity-new-signatures.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.sigtar.gpg
failed (attempt #3)
Downloading 
s3+http://mybucketname/duplicity-new-signatures.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.sigtar.gpg
Download 
s3+http://mybucketname/duplicity-new-signatures.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.sigtar.gpg
failed (attempt #4)
Downloading 
s3+http://mybucketname/duplicity-new-signatures.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.sigtar.gpg
Download 
s3+http://mybucketname/duplicity-new-signatures.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.sigtar.gpg
failed (attempt #5)
Downloading 
s3+http://mybucketname/duplicity-new-signatures.2009-01-06T00:22:46-05:00.to.2009-01-06T23:51:02-05:00.sigtar.gpg
AsyncScheduler: instantiating at concurrency 0
Traceback (most recent call last):
 File "/usr/local/bin/duplicity", line 522, in ?
   with_tempdir(main)
 File "/usr/local/bin/duplicity", line 516, in with_tempdir
   fn()
 File "/usr/local/bin/duplicity", line 509, in main
   incremental_backup(sig_chain)
 File "/usr/local/bin/duplicity", line 240, in incremental_backup
   bytes_written = write_multivol("inc", tarblock_iter, globals.backend)
 File "/usr/local/bin/duplicity", line 131, in write_multivol
   globals.gpg_profile, globals.volsize)
 File "/usr/local/lib/python2.4/site-packages/duplicity/gpg.py", line
231, in GPGWriteFile
   data = block_iter.next(bytes_to_go).data
 File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py",
line 456, in next
   result = self.process(self.input_iter.next(), size)
 File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py",
line 297, in get_delta_iter_w_sig
   for new_path, sig_path in collated:
 File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py",
line 202, in collate2iters
   relem2 = riter2.next()
 File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py",
line 263, in combine_path_iters
   range(len(path_iter_list))))
 File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py",
line 241, in get_triple
   path = path_iter_list[iter_index].next()
 File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py",
line 176, in sigtar2path_iter
   sigtarobj.close()
 File "/usr/local/lib/python2.4/site-packages/duplicity/dup_temp.py",
line 115, in close
   assert not self.fileobj.close()
 File "/usr/local/lib/python2.4/site-packages/duplicity/gpg.py", line
152, in close
   self.gpg_process.wait()
 File "/usr/local/lib/python2.4/site-packages/GnuPGInterface.py", line
639, in wait
   raise IOError, "GnuPG exited non-zero, with code %d" % (e << 8)
IOError: GnuPG exited non-zero, with code 131072

rac

> Date: Tue, 06 Jan 2009 08:51:20 +0000
> From: Simon Blandford <address@hidden>
> Subject: [Duplicity-talk] --num-retries doesn't work when fetching
>        signature       files
> To: Discussion of the backup program duplicity
>        <address@hidden>
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> I set --num-retries to 50 but sometimes Duplicity gives up when trying
> to fetch signature files after 5 attempts. This causes some backups or
> restores to fail.
>
> Shouldn't all access to S3 be retried --num-retries number of times?
>