Re: [Duplicity-talk] Over Uploading and Status Bar Not Making Sense

[Andrew Claus]

Anyone have any experience with the --s3-use-multiprocessing option?

It seems to be what was causing my problems with the progress showing it well over-uploaded, even though the files were not actually uploaded.

I used a couple sub folders to test it with various options and while including multiprocessing did not always cause a failure, removing it always resulted in a successful upload. I then tried it with the full directory and it worked.

Andrew Claus

On Wed, May 11, 2016 at 4:46 PM, Andrew Claus <address@hidden> wrote:

Follow-up: it finally stopped at 6.9GB! But only 2 volumes were actually uploaded to S3...

6.9GB 01:13:58 [195.4KB/s] [=======================================> ] 99% ETA < 45sec

Thread-1: Uploaded 26042368/26214400 bytes

Thread-1: Uploaded 26181632/26214400 bytes

Thread-3: Uploaded 24788992/26214400 bytes

Thread-1: Uploaded 26214400/26214400 bytes

6.9GB 01:14:01 [303.2KB/s] [=======================================> ] 99% ETA < 45sec

Thread-1: Uploaded chunk 8 at roughly 26083.8271029 bytes/second

Thread-1: Upload of chunk 8 complete

6.9GB 01:14:04 [212.2KB/s] [=======================================> ] 99% ETA < 45sec

Thread-3: Uploaded 24928256/26214400 bytes

Thread-3: Uploaded 25067520/26214400 bytes

Thread-3: Uploaded 25206784/26214400 bytes

6.9GB 01:14:07 [175.7KB/s] [=======================================> ] 99% ETA < 45sec

Thread-3: Uploaded 25346048/26214400 bytes

Thread-3: Uploaded 25485312/26214400 bytes

6.9GB 01:14:10 [150.2KB/s] [=======================================> ] 99% ETA < 45sec

Thread-3: Uploaded 25624576/26214400 bytes

Thread-3: Uploaded 25763840/26214400 bytes

6.9GB 01:14:13 [132.3KB/s] [=======================================> ] 99% ETA < 45sec

Thread-3: Uploaded 25903104/26214400 bytes

Thread-3: Uploaded 26042368/26214400 bytes

6.9GB 01:14:16 [119.8KB/s] [=======================================> ] 99% ETA < 45sec

6.9GB 01:14:19 [83.8KB/s] [=======================================> ] 99% ETA < 45sec

6.9GB 01:14:22 [0.0B/s] [=======================================> ] 99% ETA Stalled!

Thread-3: Uploaded 26181632/26214400 bytes

Thread-3: Uploaded 26214400/26214400 bytes

6.9GB 01:14:25 [75.5KB/s] [=======================================> ] 99% ETA < 45sec

Thread-3: Uploaded chunk 6 at roughly 22478.6619143 bytes/second

Thread-3: Upload of chunk 6 complete

Done waiting for the pool to finish processing

Uploaded s3://s3.amazonaws.com/PRIVATE/duplicity/SurfacePro4Documents/duplicity-full.20160511T182940Z.vol2.difftar.gpg to STANDARD Storage at roughly 93857.529269 bytes/second

Deleting /tmp/duplicity-HTVBro-tempdir/mktemp-9a2Ll7-3

Forgetting temporary file /tmp/duplicity-HTVBro-tempdir/mktemp-9a2Ll7-3

AsyncScheduler: task execution done (success: True)

AsyncScheduler: active workers = 0

Releasing lockfile /home/andrew/duplicity_hashes/SurfacePro4Documents/lockfile.lock

Removing still remembered temporary file /tmp/duplicity-HTVBro-tempdir/mkstemp-6YdDqP-1

Removing still remembered temporary file /tmp/duplicity-HTVBro-tempdir/mktemp-fiG4h1-4

INT intercepted...exiting.

Andrew Claus

On Wed, May 11, 2016 at 2:20 PM, Andrew Claus <address@hidden> wrote:

Hello All,

Cygwin, Duplicity 0.7.07.1, Python 2.7, GnuPG 1.4.20, boto3

I am trying to backup 1.7 GB to S3, but Duplicity seems to be running in a loop and keeping on uploading more than is necessary, but not of the right volumes. Right now the progress shows the below, and has been similar for a while. The temp directory only has files ending in 1,3,4. S3 only shows volume 1 uploaded.

4.0GB 00:47:00 [1.4MB/s] [=======================================> ] 99% ETA < 30sec

Thread-1: Uploaded 13508608/26214400 bytes

4.0GB 00:47:03 [990.6KB/s] [=======================================> ] 99% ETA < 30sec

Thread-2: Uploaded 18104320/26214400 bytes

Thread-4: Uploaded 15597568/26214400 bytes

Thread-3: Uploaded 14065664/26214400 bytes

4.1GB 00:47:06 [1.1MB/s] [=======================================> ] 99% ETA < 30sec

Thread-1: Uploaded 13647872/26214400 bytes

Thread-2: Uploaded 18243584/26214400 bytes

4.1GB 00:47:09 [1.2MB/s] [=======================================> ] 99% ETA < 30sec

Thread-1: Uploaded 13787136/26214400 bytes

Thread-2: Uploaded 18382848/26214400 bytes

Thread-3: Uploaded 14204928/26214400 bytes

4.1GB 00:47:12 [1.3MB/s] [=======================================> ] 99% ETA < 30sec

Thread-1: Uploaded 13926400/26214400 bytes

Thread-2: Uploaded 18522112/26214400 bytes

4.1GB 00:47:15 [1.3MB/s] [=======================================> ] 99% ETA < 30sec

Thread-4: Uploaded 15736832/26214400 bytes

Thread-1: Uploaded 14065664/26214400 bytes

Thread-3: Uploaded 14344192/26214400 bytes

Thread-2: Uploaded 18661376/26214400 bytes

4.1GB 00:47:18 [1.4MB/s] [=======================================> ] 99% ETA < 30sec

Thread-1: Uploaded 14204928/26214400 bytes

My options are as follows:

--timeout=500  

--asynchronous-upload 

--progress 

--s3-use-new-style 

--s3-use-multiprocessing 

--s3-use-server-side-encryption

--volsize 200 

--archive-dir /home/andrew/duplicity_hashes/

-vDEBUG

I tried putting it to a flash drive instead of S3 and could see each of the volumes being created in the temp directory and then appear in the flash drive in real time. It showed a correct total transfer amount, but oddly the status bar stayed at 0% and stalled while files were being transferred, and then suddenly jumped to 99% and then done.

I appreciate any help.

Thanks,

Andrew