Re: [Duplicity-talk] Sliding window backup strategy?

[Lars Marowsky-Bree]

On 2008-10-05T16:14:23, Colin Ryan <address@hidden> wrote:

> Sidenote: I believe - please correct me - that duplicity uses the 
> signatures and manifest files to  do incrementals from the last backup not 
> from the last full, so does this not avoid the incremental approaching the 
> size of a full dilemma?

Yes, it does, but it worsens the effect of lengthening dependency chains
of course.

> this statement does not duplicity essentially handle this as it checks the 
> backup chain, i.e. incomplete backup chain found forcing full??? Maybe I'm 
> smoking something on that last point.

It handles this if it detects it by regenerating a complete full backup,
yes.  Again, this is not optimal - a sub-set of the full backup (to fix
the chain) would be sufficient and reduce bandwidth needs.

> An open thought on this discussion is the balance of function versus
> simplicity, would not more advanced techniques as you - and I in the
> past - have requested incur a penalty on the download (thought I agree
> there is more bandwidth there).

A valid point. Complexity kills - either in the form of too long
dependency chains, or in the form of too buggy code or download needs.
However, my idea was aimed to both reduce server side storage _and_
transfer bandwidth needs.

> I've the more recursive/parity/whatever checking you have to do on the
> backup chain to accomplish goal "x", the more data must also be
> additionally pulled from the back-end in addition to pushed.

I'm not sure. The manifest could easily carry the depth of the backup
chain for any given file - an additional int wouldn't hurt - and then be
immediately able to figure out whether a check-point should be inserted.

The rest of the dependency tracking seems already there - duplicity can
already identify chunks which are no longer in the dependency tree for
a full restore, no? remove-* use that information.

As such, I would venture that it should also be "reasonably easy" to
identify chunks which have become not completely, but only "mostly"
obsolete, and then insert check-points into the incremental to be able
to free those chunks. (Side note: if those were sorted by mtime, it
seems very likely that over time, chunks would grow stable.)

Or does it only identify whether whole sets have become "free"? I
thought it did this to the volume level.


Regards,
    Lars

-- 
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde