Re: [Duplicity-talk] Why do I have a 19GB ~/.cache/duplicity folder?

[Tim Riemenschneider]

Timothee Besset schrieb:
> I'm finding that some systems that were backed up by duplicity are
> running out of disk space after I upgraded the duplicity version. More
> specifically, I am seeing 19GB worth of data in a ~/.cache/duplicity folder?
>
> I can't afford to have 20GB of disk occupied by my backup software. I
> haven't found much information on the nature of that directory, except
> that it's an addition in 0.6? Should it really be growing that big? Is
> there a limit to how much it'll grow? Is it a case of backup biting it's
> own tail like we had backups of /tmp trying to backup duplicity's own
> temporary files at one point?
>
>   
The problem seems to be, that duplicity does not cleanup signature-files
when removing old backups, so the signatures occupy more and more space.

To reproduce, I did some backups (locally), some with a forced full
backup and some incementals. Afterwards the backup looks like this:

$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 2 secondary backup chains.
Secondary chain 1 of 2:
-------------------------
Chain start time: Wed Feb 10 16:50:16 2010
Chain end time: Wed Feb 10 16:50:44 2010
Number of contained backup sets: 4
Total number of contained volumes: 4
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb 10 16:50:16 2010                 1
         Incremental         Wed Feb 10 16:50:36 2010                 1
         Incremental         Wed Feb 10 16:50:41 2010                 1
         Incremental         Wed Feb 10 16:50:44 2010                 1
-------------------------

Secondary chain 2 of 2:
-------------------------
Chain start time: Wed Feb 10 16:51:01 2010
Chain end time: Wed Feb 10 16:51:39 2010
Number of contained backup sets: 4
Total number of contained volumes: 4
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb 10 16:51:01 2010                 1
         Incremental         Wed Feb 10 16:51:36 2010                 1
         Incremental         Wed Feb 10 16:51:38 2010                 1
         Incremental         Wed Feb 10 16:51:39 2010                 1
-------------------------


Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb 10 16:51:46 2010                 1
         Incremental         Wed Feb 10 16:52:04 2010                 1
-------------------------
No orphaned or incomplete backup sets found.
$ ls /tmp/duptest/
duplicity-full.20100210T155016Z.manifest.gpg
duplicity-full.20100210T155016Z.vol1.difftar.gpg
duplicity-full.20100210T155101Z.manifest.gpg
duplicity-full.20100210T155101Z.vol1.difftar.gpg
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg

Now I remove the old backups:

$ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
file:///tmp/duptest
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Feb 10 16:51:46 2010
Deleting backup sets at times:
Wed Feb 10 16:50:16 2010
Wed Feb 10 16:50:36 2010
Wed Feb 10 16:50:41 2010
Wed Feb 10 16:50:44 2010
Wed Feb 10 16:51:01 2010
Wed Feb 10 16:51:36 2010
Wed Feb 10 16:51:38 2010
Wed Feb 10 16:51:39 2010
Warning, found the following local orphaned signature files:
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz

Note the "locally orphaned files".
When I look at the status:

$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb 10 16:51:46 2010                 1
         Incremental         Wed Feb 10 16:52:04 2010                 1
-------------------------
No orphaned or incomplete backup sets found.

Note that duplicity did fetch the signatures of the 2 old (deleted!)
full-backups

When looking at the target, it becomes clear that "duplicity remove..."
did only remove the data (*vol*difftar-files), not the obsoleted signatures:

$ ls /tmp/duptest/
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg

These files are also present (in decrypted form) in the cache-dir. And
they stay there (or are refetched from the backup-target).
Only workaround: delete the unneeded signature-files on the target, then
duplicity also removes them from the cache (but be carefull to only
remove old and unneeded files):
$ cd /tmp/duptest/
:/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-new-signatures.20100210T1550*
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T15513*
:/tmp/duptest$ ls
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
:/tmp/duptest$ cd -
~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Synchronizing remote metadata to local cache...
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
(not authoritative at backend).
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb 10 16:51:46 2010                 1
         Incremental         Wed Feb 10 16:52:04 2010                 1
-------------------------
No orphaned or incomplete backup sets found.

(Of course, one has to do full-backups from time to time in the first
place, when you only ever do incrementals, ALL signature-files are still
needed)

cu
Tim