Re: [Duplicity-talk] incremental size

[Ben Escoto]

>>>>> "SF" == sfarrell  <address@hidden>
>>>>> wrote the following on Sun, 9 Mar 2003 00:06:25 +1000

  SF> Ben, I am interest in trying to make the incremental file size a
  SF> little larger.

  SF> Currently its fixed to 5Mb as far as I can see.

  SF> You end up with so many files, they are hard to manage, fsck,
  SF> deleteing is a problem sometimes also.

  SF> I was thinking if you could pass a variable/switch to request it
  SF> to be say 650Mb or something (to fit on a CD) , or even 100mb
  SF> would make 20 times less files.

I can see how having thousands of volumes could be a problem, but they
are short for one main reason:  When restoring, duplicity integrates
information from each session simultaneously.  For instance, suppose
that a file was changed 20 times and then deleted in the last
incremental backup.  When restoring, duplicity will read the 21
different file versions from various volumes and then just not write
the file because it sees that it was finally deleted.  On the other
hand, if the file was a regular file and had just changed 20 times,
duplicity would write out the initial snapshot, and then apply the
diffs sequentially.

    So the upshot is that duplicity reads from all the backup sets at
the same time, and the least of one backup set that it can download at
one time is one volume's worth.  So, if duplicity is applying 100
incremental diffs, this would require 5MB * 100 = 500MB free disk
space.  If you increased the size to 100MB, it would take 10GB free
space.  This is the reason I didn't make the volume size configurable
from the command line - I didn't want people not to appreciate this
issue and make backups they couldn't restore.

  SF> I had also thought using a sub directory for each incremental
  SF> run, using the date stamp or serial number just once. You end up
  SF> with a sort of tree (just 1 level) and you dont get out of
  SF> control directory sizes. This would make analyzing/cleaning up
  SF> much easier. It might even enable you to keep filename lengths a
  SF> little more tidy, and it may even work with a samba share at
  SF> some stage (which I think the long filenames of the archives
  SF> breaks currently).

Yes, maybe that would be a good idea.  At first I wanted duplicity to
be able to use the simplest possible interface, but maybe every
interface that can list, put, and delete also supports directories?

About the filename lengths, I see that I failed to put this in the man
page (just fixed), but I had a similar filename length problem backing
up to MacOS and added the --short-filenames options.  The shorter
filenames are about 30 characters long, which helps with MacOS, dunno
about samba.


-- 
Ben Escoto

pgpUs38Bsnn5b.pgp
Description: PGP signature