[Duplicity-talk] PATCH [1/1] - Add progress bar estimation for full & incremental backup

[Juan A. Moya Vicén]

From time ago, there are people asking for a progress bar estimation in 
duplicity.
There is even a script that circumvents the issue, getting info from the 
log so as to estimate the progress status
( https://github.com/quentin/Duplicity-progress 
<https://github.com/quentin/Duplicity-progress> ) but does not give 
enough feedback and the estimation is rather plain.

I have developed a set of heuristics that gather information from the 
deltas and the transfer ratios of the backend so as to forecast % of 
progress, estimation of remaining time and average speed, for both full 
and incremental backup uploads.

The current implementation works for boto backend, but to port the other 
backends to use this feature would be quite easy (we can discuss the 
details if interested).

The algorithm is activated by the --progress command line flag, and will 
perform a first-pass dry-run to collect evidence for all the deltas. 
Next it will trigger the real upload, while a thread statistically 
estimates the ratio of changes and compression for the data in/out, and 
uses these ratios to forecast time remaining and % of completion.

The progress data will be logged periodically each 3 seconds, or the 
--progress-rate flag.

This patch is also available in launchpad as a merge proposition, at: 
https://code.launchpad.net/~juan-f/duplicity/progress

Enjoy!


== FAQ ==

Q: (Michael Terry)
    Hm. Deja Dup has provided a progress count for years based on 
watching how
    much of the *source data* we have covered during a --dry-run, then 
using
    that as a total when we do the real run. This does not keep track of
    uploads at all though.

    Is this similar, except that it does uploads too?

A: Yes sort of,

    What it does is to provide realtime feedback by watching how much 
of the source data has to pack and send, estimating the real amount of 
that raw data really changed.

    The difference comes in how it forecasts the % of progress and how 
it self-adapts during the upload, forecasting also the time of finish 
(like bittorrent programs do). It should feel smoother and more 
progressive than other approaches.

progress-estimation.patch
Description: Text Data