Re: [Duplicity-talk] Increasing number of asynchronously created temp files?

[Kenneth Loafman]

For now, the limit is one async thread.  We've not had time to take it further.

...Ken

On Sun, Jan 28, 2018 at 4:21 AM, Tapio Sokura via Duplicity-talk <address@hidden> wrote:

Hello,

I just started using duplicity (0.7.15 from epel) and it seems to be working fine. I'm using the asynchronous-upload option to make better use of the CPU and network resources. I can see the two temporary gpg files of volsize being used.

The stuff I'm backing up varies, there are collections of large files, but also some areas of the file system have a lot of small files. When the backup process comes across areas of small files, the upload "pipeline" stalls. Meaning that the archive file upload is happening faster than the archive file creation. Thus changing the backup process from being mostly network speed bound to CPU(/disk) bound when it hits these small files.

I tried to look for a setting in the duplicity man page for increasing the number of temporary archive files to use from two, but couldn't find one. Would it be possible to add such a user-tunable feature? This way when duplicity is going through sections of the filesystem containing large files it could build up a bigger queue of upload work. This queue could then keep the upload process better occupied when the archiving part runs into those areas of smaller files.

  Tapio

_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk