Re: [Duplicity-talk] Why do I have a 19GB ~/.cache/duplicity folder?

[Kenneth Loafman]

The OP never did file a bug report and its not been "fixed".

The large size is in the nature of the beast that is incremental backup.
 We try to keep backup sizes small, so we use small chunks for signature
comparison.  This produces more signatures on large files, and a minimal
backup.  Larger chunks would produce smaller signatures and a smaller
backup, so the only "fix" is to allow the user to select signature size
thresholds on their own.

I've not had much time to chase this, but the code that determines the
chunk size is simple, so it may not take much to make the change.

Plus, my suspicion is that these large metadata stores are a direct
result of long strings of incremental backups with no full backup at
regular intervals.  Since this is bad practice in general, I've not
really been inclined to fix duplicity to encourage it.

...Ken

Olivier Berger wrote:
> Hi.
> 
> Has a ticket indeed been filed for this ?
> 
> Is it fixed in latest release ?
> 
> Thanks in advance.
> 
> Regards,
> 
> Le mercredi 10 février 2010 à 18:35 +0100, address@hidden a écrit :
>> Yeah, well done .. please add a ticket at launchpad.. ede
>>
>> On 10.02.2010 17:22, Kenneth Loafman wrote:
>>> Tim,
>>>
>>> Good testing!
>>>
>>> Please report this bug at https://bugs.launchpad.net/duplicity
>>>
>>> ...Thanks,
>>> ...Ken
>>>
>>> Tim Riemenschneider wrote:
>>>> Timothee Besset schrieb:
>>>>> I'm finding that some systems that were backed up by duplicity are
>>>>> running out of disk space after I upgraded the duplicity version. More
>>>>> specifically, I am seeing 19GB worth of data in a ~/.cache/duplicity 
>>>>> folder?
>>>>>
>>>>> I can't afford to have 20GB of disk occupied by my backup software. I
>>>>> haven't found much information on the nature of that directory, except
>>>>> that it's an addition in 0.6? Should it really be growing that big? Is
>>>>> there a limit to how much it'll grow? Is it a case of backup biting it's
>>>>> own tail like we had backups of /tmp trying to backup duplicity's own
>>>>> temporary files at one point?
>>>>>
>>>>>
>>>> The problem seems to be, that duplicity does not cleanup signature-files
>>>> when removing old backups, so the signatures occupy more and more space.
>>>>
>>>> To reproduce, I did some backups (locally), some with a forced full
>>>> backup and some incementals. Afterwards the backup looks like this:
>>>>
>>>> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
>>>> Local and Remote metadata are synchronized, no sync needed.
>>>> Last full backup date: Wed Feb 10 16:51:46 2010
>>>> Collection Status
>>>> -----------------
>>>> Connecting with backend: LocalBackend
>>>> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>>>>
>>>> Found 2 secondary backup chains.
>>>> Secondary chain 1 of 2:
>>>> -------------------------
>>>> Chain start time: Wed Feb 10 16:50:16 2010
>>>> Chain end time: Wed Feb 10 16:50:44 2010
>>>> Number of contained backup sets: 4
>>>> Total number of contained volumes: 4
>>>>   Type of backup set:                            Time:      Num volumes:
>>>>                  Full         Wed Feb 10 16:50:16 2010                 1
>>>>           Incremental         Wed Feb 10 16:50:36 2010                 1
>>>>           Incremental         Wed Feb 10 16:50:41 2010                 1
>>>>           Incremental         Wed Feb 10 16:50:44 2010                 1
>>>> -------------------------
>>>>
>>>> Secondary chain 2 of 2:
>>>> -------------------------
>>>> Chain start time: Wed Feb 10 16:51:01 2010
>>>> Chain end time: Wed Feb 10 16:51:39 2010
>>>> Number of contained backup sets: 4
>>>> Total number of contained volumes: 4
>>>>   Type of backup set:                            Time:      Num volumes:
>>>>                  Full         Wed Feb 10 16:51:01 2010                 1
>>>>           Incremental         Wed Feb 10 16:51:36 2010                 1
>>>>           Incremental         Wed Feb 10 16:51:38 2010                 1
>>>>           Incremental         Wed Feb 10 16:51:39 2010                 1
>>>> -------------------------
>>>>
>>>>
>>>> Found primary backup chain with matching signature chain:
>>>> -------------------------
>>>> Chain start time: Wed Feb 10 16:51:46 2010
>>>> Chain end time: Wed Feb 10 16:52:04 2010
>>>> Number of contained backup sets: 2
>>>> Total number of contained volumes: 2
>>>>   Type of backup set:                            Time:      Num volumes:
>>>>                  Full         Wed Feb 10 16:51:46 2010                 1
>>>>           Incremental         Wed Feb 10 16:52:04 2010                 1
>>>> -------------------------
>>>> No orphaned or incomplete backup sets found.
>>>> $ ls /tmp/duptest/
>>>> duplicity-full.20100210T155016Z.manifest.gpg
>>>> duplicity-full.20100210T155016Z.vol1.difftar.gpg
>>>> duplicity-full.20100210T155101Z.manifest.gpg
>>>> duplicity-full.20100210T155101Z.vol1.difftar.gpg
>>>> duplicity-full.20100210T155146Z.manifest.gpg
>>>> duplicity-full.20100210T155146Z.vol1.difftar.gpg
>>>> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
>>>> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
>>>> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
>>>> duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
>>>> duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
>>>> duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
>>>> duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
>>>> duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
>>>> duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
>>>> duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
>>>> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>>>>
>>>> Now I remove the old backups:
>>>>
>>>> $ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
>>>> file:///tmp/duptest
>>>> Local and Remote metadata are synchronized, no sync needed.
>>>> Last full backup date: Wed Feb 10 16:51:46 2010
>>>> Deleting backup sets at times:
>>>> Wed Feb 10 16:50:16 2010
>>>> Wed Feb 10 16:50:36 2010
>>>> Wed Feb 10 16:50:41 2010
>>>> Wed Feb 10 16:50:44 2010
>>>> Wed Feb 10 16:51:01 2010
>>>> Wed Feb 10 16:51:36 2010
>>>> Wed Feb 10 16:51:38 2010
>>>> Wed Feb 10 16:51:39 2010
>>>> Warning, found the following local orphaned signature files:
>>>> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
>>>> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
>>>> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
>>>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
>>>> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
>>>> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
>>>>
>>>> Note the "locally orphaned files".
>>>> When I look at the status:
>>>>
>>>> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
>>>> Synchronizing remote metadata to local cache...
>>>> Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
>>>> Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
>>>> Last full backup date: Wed Feb 10 16:51:46 2010
>>>> Collection Status
>>>> -----------------
>>>> Connecting with backend: LocalBackend
>>>> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>>>>
>>>> Found 0 secondary backup chains.
>>>>
>>>> Found primary backup chain with matching signature chain:
>>>> -------------------------
>>>> Chain start time: Wed Feb 10 16:51:46 2010
>>>> Chain end time: Wed Feb 10 16:52:04 2010
>>>> Number of contained backup sets: 2
>>>> Total number of contained volumes: 2
>>>>   Type of backup set:                            Time:      Num volumes:
>>>>                  Full         Wed Feb 10 16:51:46 2010                 1
>>>>           Incremental         Wed Feb 10 16:52:04 2010                 1
>>>> -------------------------
>>>> No orphaned or incomplete backup sets found.
>>>>
>>>> Note that duplicity did fetch the signatures of the 2 old (deleted!)
>>>> full-backups
>>>>
>>>> When looking at the target, it becomes clear that "duplicity remove..."
>>>> did only remove the data (*vol*difftar-files), not the obsoleted 
>>>> signatures:
>>>>
>>>> $ ls /tmp/duptest/
>>>> duplicity-full.20100210T155146Z.manifest.gpg
>>>> duplicity-full.20100210T155146Z.vol1.difftar.gpg
>>>> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
>>>> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
>>>> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
>>>> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>>>>
>>>> These files are also present (in decrypted form) in the cache-dir. And
>>>> they stay there (or are refetched from the backup-target).
>>>> Only workaround: delete the unneeded signature-files on the target, then
>>>> duplicity also removes them from the cache (but be carefull to only
>>>> remove old and unneeded files):
>>>> $ cd /tmp/duptest/
>>>> :/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
>>>> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T1550*
>>>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T15513*
>>>> :/tmp/duptest$ ls
>>>> duplicity-full.20100210T155146Z.manifest.gpg
>>>> duplicity-full.20100210T155146Z.vol1.difftar.gpg
>>>> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
>>>> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>>>> :/tmp/duptest$ cd -
>>>> ~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
>>>> Synchronizing remote metadata to local cache...
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Last full backup date: Wed Feb 10 16:51:46 2010
>>>> Collection Status
>>>> -----------------
>>>> Connecting with backend: LocalBackend
>>>> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>>>>
>>>> Found 0 secondary backup chains.
>>>>
>>>> Found primary backup chain with matching signature chain:
>>>> -------------------------
>>>> Chain start time: Wed Feb 10 16:51:46 2010
>>>> Chain end time: Wed Feb 10 16:52:04 2010
>>>> Number of contained backup sets: 2
>>>> Total number of contained volumes: 2
>>>>   Type of backup set:                            Time:      Num volumes:
>>>>                  Full         Wed Feb 10 16:51:46 2010                 1
>>>>           Incremental         Wed Feb 10 16:52:04 2010                 1
>>>> -------------------------
>>>> No orphaned or incomplete backup sets found.
>>>>
>>>> (Of course, one has to do full-backups from time to time in the first
>>>> place, when you only ever do incrementals, ALL signature-files are still
>>>> needed)
>>>>
>>>> cu
>>>> Tim
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Duplicity-talk mailing list
>>>> address@hidden
>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

signature.asc
Description: OpenPGP digital signature