[Duplicity-talk] Sliding window backup strategy?

[Lars Marowsky-Bree]

Hi all,

having grown sick of my home-grown scripts - which essentially
implemented rdiff-backup + hard-linked directories for storing the old
revisions on top of encrypted mounts -, I investigated new backup
software, because I wanted to have less dependencies on needing to trust
the remote servers etc. duplicity so far fits my needs best - opaque
chunks at the remote server, no need to trust the remote site etc. I had
basically come up with most of the same ideas and then decided I didn't
need to reimplement duplicity ;-) Excellent tool!

However, one idea I did have (maybe it is not original, but I couldn't
readily find it elsewhere) was what I call "sliding window
full-backups". 

The closest I found in the archives here is
http://lists.gnu.org/archive/html/duplicity-talk/2008-08/msg00075.html
(a discussion between Rik v. A and Colin Ryan), but merging changes back
into encrypted chunks is of course hard.

My idea is slightly different.

Goals: I never want to do another full backup (after of course the first
one). Bandwidth on the run is expensive, as is the time needed and
battery power required. Yet, I also don't want to need to restore a full
backup + a couple thousand incrementals.

Hence: "sliding full backup". With every incremental backup, I also
would be willing to perform n% of a full backup.

Another name for this might also be "log structured" backup, but I think
sliding window captures it better.

For example, if I set n = 5%, I would only require the last 20d daily
backups to restore a full snapshot. And every incremental run would be
bounded and about the same time - I'd never have to pay the penalty of a
full sync.

(n% might vary between runs, and might not always be known in advance,
as the connection might get interrupted during the run - so it'd be nice
if full backups were check-pointed and could resume.)

And then, I might want to keep incrementals around for M days - possibly
longer than the last full backup.

Am I making sense to anyone else?

I have not yet fully thought everything out in the above scheme, but
maybe, if others have similar needs, we could explore them.


Regards,
    Lars

PS: Another topic I'd like to explore one day is efficient handling of
backups to several sites. Paranoid that I am, I have backups to my USB
stick, my home file server, and a co-lo site - running duplicity
multiple times is rather inefficient, so I know do rsync from the USB
stick to the additional sites, but this doesn't do justice to the fact
that the sites have significantly different storage capacities and could
store more revisions than the USB stick does ... But one thing after the
other.


-- 
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde