Re: [Duplicity-talk] Hanging just before completion

[Kenneth Loafman]

Hi Cody,

I'm having trouble reproducing this.

What kind of target schema do you use?  Is it the file:// schema or something else?  Please list the full command line.

I'd like a log file, please. This is going big and I only need a portion. Please run with -v9, redirect to a log file, then run:

$ head -n 200 < log > log.head
$ tail -n 200 < log > log.tail

Attach, don't copy/paste, log.head and log.tail to your reply.

...Thanks,
...Ken

On Thu, Oct 15, 2020 at 11:56 AM Cody Ashe-McNalley via Duplicity-talk <duplicity-talk@nongnu.org> wrote:

Hi Ken,

The hanging seems to happen before upload would take place.  The sigtar files are huge, sometimes almost a terabyte.  But they will sit there for weeks with no network or process activity.  It only takes me a few hours to rename, gzip, and upload them.

It seems related to finishing the async process forking loop, but perhaps that does finish and it hangs immediately after.  The final difftar is uploaded.

Many thanks,

Cody

On Thu, Oct 15, 2020 at 6:36 AM Kenneth Loafman via Duplicity-talk <duplicity-talk@nongnu.org> wrote:

Hi Cody,

What are the sizes of the manifest and sigtar files?  They both get uploaded last and take a while.  Check for network activity.

...Thanks,

...Ken

 

On Wed, Oct 14, 2020 at 4:42 PM Cody Ashe-McNalley via Duplicity-talk <duplicity-talk@nongnu.org> wrote:

Hi All,

Duplicity is a great program and I appreciate this community and the development process.

Several duplicity jobs have hung after putting the final archive file up using the rclone backend.  The total archive size has been >=30TB.  The manifest and sigtar files are complete, but are left in the archive-dir with the .part suffix.  I've tried to look into this with gdb but the process forking makes that difficult.

Debug logging shows this as the point where it hangs:

DEBUG 1

. AsyncScheduler: active workers = 0

Centos Linux tethys.atmos.ucla.edu 3.10.0-1127.19.1.el7.x86_64
python 3.6.8 (virtual env)
duplicity-0.8.16
Options: --asynchronous-upload --no-compression --no-encryption --volsize 20000

I've been able to save the jobs by manually renaming and transferring the manifest and sigtar.  

Best regards,
Cody

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk@nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk@nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk@nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk