Re: [Duplicity-talk] Why do I have a 19GB ~/.cache/duplicity folder?

[Olivier Berger]

Hi.

Has a ticket indeed been filed for this ?

Is it fixed in latest release ?

Thanks in advance.

Regards,

Le mercredi 10 février 2010 à 18:35 +0100, address@hidden a écrit :
> Yeah, well done .. please add a ticket at launchpad.. ede
> 
> On 10.02.2010 17:22, Kenneth Loafman wrote:
> > Tim,
> >
> > Good testing!
> >
> > Please report this bug at https://bugs.launchpad.net/duplicity
> >
> > ...Thanks,
> > ...Ken
> >
> > Tim Riemenschneider wrote:
> >> Timothee Besset schrieb:
> >>> I'm finding that some systems that were backed up by duplicity are
> >>> running out of disk space after I upgraded the duplicity version. More
> >>> specifically, I am seeing 19GB worth of data in a ~/.cache/duplicity 
> >>> folder?
> >>>
> >>> I can't afford to have 20GB of disk occupied by my backup software. I
> >>> haven't found much information on the nature of that directory, except
> >>> that it's an addition in 0.6? Should it really be growing that big? Is
> >>> there a limit to how much it'll grow? Is it a case of backup biting it's
> >>> own tail like we had backups of /tmp trying to backup duplicity's own
> >>> temporary files at one point?
> >>>
> >>>
> >> The problem seems to be, that duplicity does not cleanup signature-files
> >> when removing old backups, so the signatures occupy more and more space.
> >>
> >> To reproduce, I did some backups (locally), some with a forced full
> >> backup and some incementals. Afterwards the backup looks like this:
> >>
> >> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> >> Local and Remote metadata are synchronized, no sync needed.
> >> Last full backup date: Wed Feb 10 16:51:46 2010
> >> Collection Status
> >> -----------------
> >> Connecting with backend: LocalBackend
> >> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
> >>
> >> Found 2 secondary backup chains.
> >> Secondary chain 1 of 2:
> >> -------------------------
> >> Chain start time: Wed Feb 10 16:50:16 2010
> >> Chain end time: Wed Feb 10 16:50:44 2010
> >> Number of contained backup sets: 4
> >> Total number of contained volumes: 4
> >>   Type of backup set:                            Time:      Num volumes:
> >>                  Full         Wed Feb 10 16:50:16 2010                 1
> >>           Incremental         Wed Feb 10 16:50:36 2010                 1
> >>           Incremental         Wed Feb 10 16:50:41 2010                 1
> >>           Incremental         Wed Feb 10 16:50:44 2010                 1
> >> -------------------------
> >>
> >> Secondary chain 2 of 2:
> >> -------------------------
> >> Chain start time: Wed Feb 10 16:51:01 2010
> >> Chain end time: Wed Feb 10 16:51:39 2010
> >> Number of contained backup sets: 4
> >> Total number of contained volumes: 4
> >>   Type of backup set:                            Time:      Num volumes:
> >>                  Full         Wed Feb 10 16:51:01 2010                 1
> >>           Incremental         Wed Feb 10 16:51:36 2010                 1
> >>           Incremental         Wed Feb 10 16:51:38 2010                 1
> >>           Incremental         Wed Feb 10 16:51:39 2010                 1
> >> -------------------------
> >>
> >>
> >> Found primary backup chain with matching signature chain:
> >> -------------------------
> >> Chain start time: Wed Feb 10 16:51:46 2010
> >> Chain end time: Wed Feb 10 16:52:04 2010
> >> Number of contained backup sets: 2
> >> Total number of contained volumes: 2
> >>   Type of backup set:                            Time:      Num volumes:
> >>                  Full         Wed Feb 10 16:51:46 2010                 1
> >>           Incremental         Wed Feb 10 16:52:04 2010                 1
> >> -------------------------
> >> No orphaned or incomplete backup sets found.
> >> $ ls /tmp/duptest/
> >> duplicity-full.20100210T155016Z.manifest.gpg
> >> duplicity-full.20100210T155016Z.vol1.difftar.gpg
> >> duplicity-full.20100210T155101Z.manifest.gpg
> >> duplicity-full.20100210T155101Z.vol1.difftar.gpg
> >> duplicity-full.20100210T155146Z.manifest.gpg
> >> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> >> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> >> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> >> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> >> duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
> >> duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
> >> duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
> >> duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
> >> duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
> >> duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
> >> duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> >> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
> >>
> >> Now I remove the old backups:
> >>
> >> $ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
> >> file:///tmp/duptest
> >> Local and Remote metadata are synchronized, no sync needed.
> >> Last full backup date: Wed Feb 10 16:51:46 2010
> >> Deleting backup sets at times:
> >> Wed Feb 10 16:50:16 2010
> >> Wed Feb 10 16:50:36 2010
> >> Wed Feb 10 16:50:41 2010
> >> Wed Feb 10 16:50:44 2010
> >> Wed Feb 10 16:51:01 2010
> >> Wed Feb 10 16:51:36 2010
> >> Wed Feb 10 16:51:38 2010
> >> Wed Feb 10 16:51:39 2010
> >> Warning, found the following local orphaned signature files:
> >> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
> >> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
> >> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
> >> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
> >> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
> >> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
> >>
> >> Note the "locally orphaned files".
> >> When I look at the status:
> >>
> >> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> >> Synchronizing remote metadata to local cache...
> >> Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
> >> Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
> >> Last full backup date: Wed Feb 10 16:51:46 2010
> >> Collection Status
> >> -----------------
> >> Connecting with backend: LocalBackend
> >> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
> >>
> >> Found 0 secondary backup chains.
> >>
> >> Found primary backup chain with matching signature chain:
> >> -------------------------
> >> Chain start time: Wed Feb 10 16:51:46 2010
> >> Chain end time: Wed Feb 10 16:52:04 2010
> >> Number of contained backup sets: 2
> >> Total number of contained volumes: 2
> >>   Type of backup set:                            Time:      Num volumes:
> >>                  Full         Wed Feb 10 16:51:46 2010                 1
> >>           Incremental         Wed Feb 10 16:52:04 2010                 1
> >> -------------------------
> >> No orphaned or incomplete backup sets found.
> >>
> >> Note that duplicity did fetch the signatures of the 2 old (deleted!)
> >> full-backups
> >>
> >> When looking at the target, it becomes clear that "duplicity remove..."
> >> did only remove the data (*vol*difftar-files), not the obsoleted 
> >> signatures:
> >>
> >> $ ls /tmp/duptest/
> >> duplicity-full.20100210T155146Z.manifest.gpg
> >> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> >> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> >> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> >> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> >> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
> >>
> >> These files are also present (in decrypted form) in the cache-dir. And
> >> they stay there (or are refetched from the backup-target).
> >> Only workaround: delete the unneeded signature-files on the target, then
> >> duplicity also removes them from the cache (but be carefull to only
> >> remove old and unneeded files):
> >> $ cd /tmp/duptest/
> >> :/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> >> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T1550*
> >> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T15513*
> >> :/tmp/duptest$ ls
> >> duplicity-full.20100210T155146Z.manifest.gpg
> >> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> >> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> >> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
> >> :/tmp/duptest$ cd -
> >> ~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> >> Synchronizing remote metadata to local cache...
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
> >> (not authoritative at backend).
> >> Last full backup date: Wed Feb 10 16:51:46 2010
> >> Collection Status
> >> -----------------
> >> Connecting with backend: LocalBackend
> >> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
> >>
> >> Found 0 secondary backup chains.
> >>
> >> Found primary backup chain with matching signature chain:
> >> -------------------------
> >> Chain start time: Wed Feb 10 16:51:46 2010
> >> Chain end time: Wed Feb 10 16:52:04 2010
> >> Number of contained backup sets: 2
> >> Total number of contained volumes: 2
> >>   Type of backup set:                            Time:      Num volumes:
> >>                  Full         Wed Feb 10 16:51:46 2010                 1
> >>           Incremental         Wed Feb 10 16:52:04 2010                 1
> >> -------------------------
> >> No orphaned or incomplete backup sets found.
> >>
> >> (Of course, one has to do full-backups from time to time in the first
> >> place, when you only ever do incrementals, ALL signature-files are still
> >> needed)
> >>
> >> cu
> >> Tim
> >>
> >>
> >>
> >> _______________________________________________
> >> Duplicity-talk mailing list
> >> address@hidden
> >> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >>
> >
> >
> >
> >
> > _______________________________________________
> > Duplicity-talk mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk

-- 
Olivier BERGER <address@hidden>
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingénieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)