Re: [Duplicity-talk] Duplicity Maintenance

[Kenneth Loafman]

Travis H. wrote:
> On Mon, Apr 23, 2007 at 08:27:32PM -0500, Kenneth Loafman wrote:
> > I'll be assuming the reins of duplicity maintenance.   Hopefully, I'll 
> > have a lot of help from those of you that use it.
> >
> > Thanks to rsync.net for sponsoring the project.  Thanks to Ben Escoto 
> > for initially authoring duplicity and thanks to all the rest that have 
> > submitted improvements and administered the project in the past.
>
> Out of curiousity; is this a paid position?

I wish.  There is a bounty once I get the work done, the first part of 
which I hope to finish this weekend.

> I was sort of interested in it but have bills to pay.

Ditto, plus having to start up a new company since Team 2 came in to my 
last startup company and kicked out Team 1.  I expected it.  I've pretty 
much always been Team 1 all my life, so Team 2 can have the customer 
support and other nightmares associated with their change from a 
web-based service on Linux to a Windows based desktop product.

> (Alas, my landlord does not take "kick ass software" as payment)
>
> > I plan on getting a new version of duplicity released fairly soon with 
> > the current patches (that work), plus some bug fixes.  Feature requests 
> > that weren't covered in the patches will be considered for a subsequent 
> > release.  The plans are being firmed up now and I should have more to 
> > tell you in the next week or so.  I'm still settling in.
>
> Some thoughts I've collected;
>
> http://www.securityfocus.com/infocus/1650
>
> See the VFS stack stuff in here and think of reading stuff inside
> an archive of some kind.

The very nature of the process 'tar | gzip | gpg' precludes random 
reading of the archive.  Any change to another format would be a rather 
major development effort.

> Unix filter for adding redundancy (to tolerate errors in files)
> like .par but evenly distributed throughout the file (so no
> distinction between redundant data and plaintext; it's all
> welded together) to be used with tar, duplicity, etc.
> Basically a pre and post-processor for dealing with bitrot

After 35+ years working this field, I'm still not a believer in bitrot, 
except on certain popular OSes that tend to eat their young.  I would 
think a good set of checksums should be able to check for corruption, 
but that would require executing something on the remote server, 
somewhat like a P2P client does.

Besides, as you said, it could be a filter, applied after duplicity. If 
someone were to supply such a filter, then a backend for that filter 
would be easy.  I think that filter should probably be its own project.

> use bz2 library in python (bzip2 compression)

Good idea.  bzip2 is slower than gzip, but this is a backup program and 
does not need to be a speed demon.  Would save some space.

> --no-deltas - don't transfer deltas, transfer entire file (the same as
> rsync -W i believe)

What you want is the equivalent of an incremental backup.  That's 
possible, but if if you plan your backup strategy to do full backups on 
a reasonable basis, then there should be no file out of date by more 
than a week or so, even if an incremental is corrupted somehow.

> --dry-run - to see what would be done but not act on it (rsync -n, and
> not actually compress/encrypt anything either, just report what's
> changed)

Hmmm, not a bad idea.  Let me think about it for a bit.

> --max-size to tell duplicity how big of archive files to create
> (basically the "filesize" variable in the GPGWriteFile and Gzfile or
> whatever stuff)

Possible.  Is there a real need for such a change?

...Ken