[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Duplicity-talk] Weird error message after incremental backup of large d

From: Jakob Bohm
Subject: [Duplicity-talk] Weird error message after incremental backup of large drive with a bunch of changed files
Date: Tue, 14 Mar 2023 21:54:12 +0100
User-agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:5.2) Goanna/20221030 Epyrus/2.0.0

Dear group,

I have set up some scripts to do various parts of a full system backup
via duplicity to a geographically close S3 bucket (AWS Stockholm),
however for the largest drive, I occasionally experience hangs/errors
near the end of each backup, with the progress display wobbling between
"stalled" and less than 30 minutes left (this time I observed as low as
16 minutes at one point).

Then a day into the stall/short time phase, I received to following
error message (redacted bucket name, mountpoints etc.):

Attempt of put Nr. 1 failed. S3UploadFailedError: Failed to upload
/duplicitypart/tmp/commonprefix-partname/duplicity-g3cvly63-tempdir/duplicity_temp.1/commonprefix-partname_duplicity-inc.20230212T043018Z.to.20230311T203857Z.vol1501.difftar.gpg.vol000+200.par2 to bucketname/partname/commonprefix-partname_duplicity-inc.20230212T043018Z.to.20230311T203857Z.vol1501.difftar.gpg.vol000+200.par2: An error occurred (RequestTimeTooSkewed) when calling the CreateMultipartUpload operation: The difference between the request time and the current time is too large.

After the message there were some small increases in the GB counter in the progress bar

ls report after killing duplicity:

-rw-r--r-- 1 root root 21336776 Mar 13 18:33 /duplicitypart/tmp/commonprefix-partname/duplicity-g3cvly63-tempdir/duplicity_temp.1/commonprefix-partname_duplicity-inc.20230212T043018Z.to.20230311T203857Z.vol1501.difftar.gpg.vol000+200.par2

Currently invoking duplicity 1.2.1 (patched) with command line

duplicity incremental --name commonprefix-partname \
  --archive-dir /duplicitypart/archive/commonprefix-partname \
  --asynchronous-upload \
  --file-prefix commonprefix-partname_ \
  --tempdir /duplicitypart/tmp/commonprefix-partname \
  --verbosity notice \
  --progress \
  --log-file /duplicitypart/tmp/commonprefix-partname/log_20230311T20_38_52.log \
  --gpg-options '--homedir /configdir/.gnupg --compress-algo=bzip2' \
  --encrypt-secret-keyring /configdir/.gnupg/secret.gpg \
  --encrypt-key 1234567890ABCDEF1234567890ABCDEF12345678 \
  --sign-key 1234567890ABCDEF1234567890ABCDEF12345678 \
  --hidden-encrypt-key 1234567890ABCDEF1234567890ABCDEF12345678 \
  --exclude-other-filesystems \
  --full-if-older-than 3M \
  --s3-use-multiprocessing \
  --numeric-owner \
  /partname \

OS: Debian GNU/Linux 11.7 (bullseye) with Python 3.9.2, python-boto3 version 1.13.14-1

I suspect a relationship with issue #254, and as you see, I have incorporated some of the workarounds into the command line.

What are the appropriate troubleshooting steps?


Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

reply via email to

[Prev in Thread] Current Thread [Next in Thread]