[Duplicity-talk] performance issue

[Chris Chatelain]

I've been using duplicity to back up to amazon S3 for about 3 months. 
I'm backing up nearly an entire RHEL5 server, excluding a few things. It 
used to take 70 minutes or so, and suddenly one day it jumped to taking 
4 or 5 hours. The size of the files has only gone up from 32GB to about 
36. Is there a way to see which part is taking long (ie. is it the 
amazon interaction taking so long, or is it collecting the information 
on the files, or is it the taring that's taking so long)? Now that I 
look at the output and back in my logs, it looks like it catches a 
socket error 3 times each run. Any other ideas what the problem might be?

My script looks like this:
export AWS_ACCESS_KEY_ID=#(insert your key_id)
export AWS_SECRET_ACCESS_KEY=#(insert access key)
date
echo "Backing up filesystem to Amazon S3"
time duplicity --exclude-device-files --exclude /sys --exclude 
/var/named/chroot/var/run --exclude /var/named/chroot/var/tmp --exclude 
/var/named/chroot/proc --exclude /var/named/chroot/dev --exclude /proc 
--exclude /tmp --no-encryption --allow-source-mismatch / 
s3+http://thewebshop-black-full
echo "Backup complete"
date
export AWS_ACCESS_KEY_ID= #clear out your id and key
export AWS_SECRET_ACCESS_KEY=

and this is the output:


Thu Jul 31 05:31:15 CDT 2008

Backing up filesystem to Amazon S3
Caught a socket error, trying to recover
Caught a socket error, trying to recover
--------------[ Backup Statistics ]--------------
StartTime 1217501754.04 (Thu Jul 31 05:55:54 2008)
EndTime 1217511035.29 (Thu Jul 31 08:30:35 2008)
ElapsedTime 9281.25 (2 hours 34 minutes 41.25 seconds)
SourceFiles 859713
SourceFileSize 62674401579 (58.4 GB)
NewFiles 2509
NewFileSize 237279065 (226 MB)
DeletedFiles 1798
ChangedFiles 6062
ChangedFileSize 36115011699 (33.6 GB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 10369
RawDeltaSize 38133832 (36.4 MB)
TotalDestinationSizeChange 630377063 (601 MB)
Errors 0
-------------------------------------------------


real    179m44.075s
user    85m47.365s
sys     56m5.727s
Backup complete
Thu Jul 31 08:30:59 CDT 2008