[Duplicity-talk] Files left behind when deleting full backups sets

[Elvis Stansvik]

Good evening,

I have some strange behavior I can't explain. Let me show with a very
simple test case:

address@hidden ~/dup_test]$ rm -rf ~/.cache
address@hidden ~/dup_test]$ mkdir source target
address@hidden ~/dup_test]$ touch source/testfile

So we have our source and target. Let's create the first backup set:

address@hidden ~/dup_test]$ duplicity --no-encryption
--no-print-statistics source file://target
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.

Since this was the first time, Duplicity made a full set. Let's make
an incremental as well:

address@hidden ~/dup_test]$ duplicity --no-encryption
--no-print-statistics source file://target
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Oct 21 18:37:17 2012

Let's do the same thing again, and create another full backup, with an
incremental on top:

address@hidden ~/dup_test]$ duplicity full --no-encryption
--no-print-statistics source file://target
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Oct 21 18:37:17 2012
address@hidden ~/dup_test]$ duplicity --no-encryption
--no-print-statistics source file://target
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Oct 21 18:37:41 2012

Now let's see what it looks like:

address@hidden ~/dup_test]$ duplicity --no-encryption collection-status
file://target
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Oct 21 18:37:41 2012
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/dup_test/.cache/duplicity/2334326747ca7273e1b6291b0ce4d508

Found 1 secondary backup chain.
Secondary chain 1 of 1:
-------------------------
Chain start time: Sun Oct 21 18:37:17 2012
Chain end time: Sun Oct 21 18:37:25 2012
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set:                            Time:      Num volumes:
                Full         Sun Oct 21 18:37:17 2012                 1
         Incremental         Sun Oct 21 18:37:25 2012                 1
-------------------------


Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Sun Oct 21 18:37:41 2012
Chain end time: Sun Oct 21 18:37:52 2012
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set:                            Time:      Num volumes:
                Full         Sun Oct 21 18:37:41 2012                 1
         Incremental         Sun Oct 21 18:37:52 2012                 1
-------------------------
No orphaned or incomplete backup sets found.

Looks very good. We have two chains, each with one full backup set and
one incremental set.

Let's have a look at the backup files:

address@hidden ~/dup_test]$ ls -1 target/
duplicity-full-signatures.20121021T163717Z.sigtar.gz
duplicity-full-signatures.20121021T163741Z.sigtar.gz
duplicity-full.20121021T163717Z.manifest
duplicity-full.20121021T163717Z.vol1.difftar.gz
duplicity-full.20121021T163741Z.manifest
duplicity-full.20121021T163741Z.vol1.difftar.gz
duplicity-inc.20121021T163717Z.to.20121021T163725Z.manifest
duplicity-inc.20121021T163717Z.to.20121021T163725Z.vol1.difftar.gz
duplicity-inc.20121021T163741Z.to.20121021T163752Z.manifest
duplicity-inc.20121021T163741Z.to.20121021T163752Z.vol1.difftar.gz
duplicity-new-signatures.20121021T163717Z.to.20121021T163725Z.sigtar.gz
duplicity-new-signatures.20121021T163741Z.to.20121021T163752Z.sigtar.gz

This looks fine right? First comes the sigtars, difftars and manifests
for the two full backups. Then comes the the same for the two
incrementals.

Now let's try to remove the older of the two full backup sets. Here's
where the strange behavior begins!

address@hidden ~/dup_test]$ duplicity remove-all-but-n-full 1 --force
--no-encryption file://target
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Oct 21 18:37:41 2012
Deleting backup sets at times:
Sun Oct 21 18:37:17 2012
Sun Oct 21 18:37:25 2012
Deleting set inc Sun Oct 21 18:37:25 2012
Deleting set full Sun Oct 21 18:37:17 2012
Warning, found the following local orphaned signature file:
duplicity-new-signatures.20121021T163717Z.to.20121021T163725Z.sigtar.gz

It looks good to begin with. It correctly identified the full backup
set from 18:37:17 and its associated incremental backups set from
18:37:25, and it says it deleted them.

(What about that warning though? Why can't it simply remove the
orphaned signature from the cache?)

But what is even stranger is when we look at the backup files again:

address@hidden ~/dup_test]$ ls -1 target/
duplicity-full-signatures.20121021T163717Z.sigtar.gz     <-- ??
duplicity-full-signatures.20121021T163741Z.sigtar.gz
duplicity-full.20121021T163741Z.manifest
duplicity-full.20121021T163741Z.vol1.difftar.gz
duplicity-inc.20121021T163741Z.to.20121021T163752Z.manifest
duplicity-inc.20121021T163741Z.to.20121021T163752Z.vol1.difftar.gz
duplicity-new-signatures.20121021T163717Z.to.20121021T163725Z.sigtar.gz
    <-- ??
duplicity-new-signatures.20121021T163741Z.to.20121021T163752Z.sigtar.gz

It didn't delete the sets fully! There's still the sigtars of the full
and incremental set left (indicated above).

Looking at the cache directory, we see the same problem, but here only
the sigtar of the (deleted) incremental set is still there:

address@hidden ~/dup_test]$ ls -1
~/.cache/duplicity/2334326747ca7273e1b6291b0ce4d508/
duplicity-full-signatures.20121021T163741Z.sigtar.gz
duplicity-full.20121021T163741Z.manifest
duplicity-inc.20121021T163741Z.to.20121021T163752Z.manifest
duplicity-new-signatures.20121021T163717Z.to.20121021T163725Z.sigtar.gz
   <-- ??
duplicity-new-signatures.20121021T163741Z.to.20121021T163752Z.sigtar.gz

Why is it that Duplicity leaves these files behind?

For my purposes, I want a backup routine like:

Mon - Full backup + Remove everything but the last 2 full backups
Tue - Incremental
Wed - Incremental
Thu - Incremental
Fri - Incremental
Sat - Incremental
Sun - Incremental

Which means I'd always have the option of restoring from any day in
the last 1-2 weeks.

But if Duplicity leaves files behind like this when deleting sets,
it's just going to pile up, and I really don't want that. I want all
files associated with those sets to be deleted, both from the backup
location and the cache.

So, is this a bug or some intended behavior of some kind?

Best regards,
Elvis Stansvik