Re: [Duplicity-talk] performance issue

[Peter Schuller]

> I've been using duplicity to back up to amazon S3 for about 3 months. 
> I'm backing up nearly an entire RHEL5 server, excluding a few things. It 
> used to take 70 minutes or so, and suddenly one day it jumped to taking 
> 4 or 5 hours. The size of the files has only gone up from 32GB to about 
> 36. Is there a way to see which part is taking long (ie. is it the 
> amazon interaction taking so long, or is it collecting the information 
> on the files, or is it the taring that's taking so long)? Now that I 
> look at the output and back in my logs, it looks like it catches a 
> socket error 3 times each run. Any other ideas what the problem might be?

Unfortunately there is no direct timing of "local cpu/disk"
vs. "transfer", etc to give an immediate indication of where the
bottleneck is. However, if you are trying to figure it out I sugest
running at -v5 or above and observation the time it takes to upload a
file - also in conjunction with e.g. 'nload' or some other traffic
monitoring tool to show you the bandwidth usage.

In the case of S3 though, I can definitely say that I have had some
inconsistent performance in the past. There was one particular period
of a few weeks where I would get *lots* of socket errors,
nececcitating sending each volume several times. I would also see very
bursty bandwidth utilization. This problem, when I had it, was not
specific to duplicity or the boto S3 library that duplicity uses
(there is much traffic on this from a few months back in the ML
archives).

I think it's pretty likely that the increase in time has to do with
the backend - that is, either your connectivity, an S3 issue, or
something in between. Try timing a backup of something reasonable -
say 10 MB. Extrapolate the time it took to upload the file (just time
it manually when running with -v5) and extrapolate.

How long has the performance been subpar? If only for a few days you
might just try waiting a week ;)

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <address@hidden>'
Key retrieval: Send an E-Mail to address@hidden
E-Mail: address@hidden Web: http://www.scode.org

pgptjeHECFnpu.pgp
Description: PGP signature