Re: [Duplicity-talk] [patch] Use --num-retries option for Amazon S3

[Kenneth Loafman]

Thanks.  Will get it out in the next release.

...Ken

Greg Hewgill wrote:
> The following patch implements the --num-retries option for the Boto
> Amazon S3 backend.
> 
> I successfully backed up 30 GB of data using this patch. Previously, S3
> was failing often enough to prevent me from completing a full backup.
> 
> Greg Hewgill
> http://hewgill.com
> 
> 
> ------------------------------------------------------------------------
> 
> Only in duplicity-0.4.4.RC3.orig: build
> diff -ur duplicity-0.4.4.RC3.orig/src/backends.py 
> duplicity-0.4.4.RC3/src/backends.py
> --- duplicity-0.4.4.RC3.orig/src/backends.py  Tue Oct  2 12:06:51 2007
> +++ duplicity-0.4.4.RC3/src/backends.py       Sun Nov 18 03:55:09 2007
> @@ -701,18 +701,36 @@
>       def put(self, source_path, remote_filename=None):
>               if not remote_filename:
>                       remote_filename = source_path.get_filename()
> -             log.Log("Uploading %s to Amazon S3" % remote_filename, 5)
>               key = self.key_class(self.bucket)
>               key.key = remote_filename
> -             key.set_contents_from_filename(source_path.name, 
> -                             {'Content-Type': 'application/octet-stream'})
> +             for n in range(1, globals.num_retries+1):
> +                     log.Log("Uploading %s to Amazon S3 (attempt #%d)" % 
> (remote_filename, n), 5)
> +                     try:
> +                             
> key.set_contents_from_filename(source_path.name, 
> +                                     {'Content-Type': 
> 'application/octet-stream'})
> +                             return
> +                     except:
> +                             pass
> +                     log.Log("Uploading %s failed (attempt #%d)" % 
> (remote_filename, n), 1)
> +                     time.sleep(30)
> +             log.Log("Giving up trying to upload %s after %d attempts" % 
> (remote_filename, globals.num_retries), 1)
> +             raise BackendException("Error uploading %s" % remote_filename)
>       
>       def get(self, remote_filename, local_path):
> -             log.Log("Downloading %s from Amazon S3" % remote_filename, 5)
>               key = self.key_class(self.bucket)
>               key.key = remote_filename
> -             key.get_contents_to_filename(local_path.name)
> -             local_path.setdata()
> +             for n in range(1, globals.num_retries+1):
> +                     log.Log("Downloading %s from Amazon S3 (attempt #%d)" % 
> (remote_filename, n), 5)
> +                     try:
> +                             key.get_contents_to_filename(local_path.name)
> +                             local_path.setdata()
> +                             return
> +                     except:
> +                             pass
> +                     log.Log("Downloading %s failed (attempt #%d)" % 
> (remote_filename, n), 1)
> +                     time.sleep(30)
> +             log.Log("Giving up trying to download %s after %d attempts" % 
> (remote_filename, globals.num_retries), 1)
> +             raise BackendException("Error downloading %s" % remote_filename)
>  
>       def list(self):
>               filename_list = [k.key for k in self.bucket]
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk

signature.asc
Description: OpenPGP digital signature