[Duplicity-talk] Corrupted back-up chain

[Tomaž Šolc]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi

I've been successfully using Duplicity for quite some time to keep
backups of several machines. Today I've tried to restore some lost
files, in the same way I've done many times before, and found that I can't.

Right now it looks like I'll be able to get most of my data back by
picking apart the difftar files by hand, so this isn't a big problem. It
would be much worse if I would have to restore the complete system.

Since I can't reproduce this I know that chances of finding the cause
are slim. If someone has any idea how this situation might have happened
I would be grateful for any tips. I would very much like to trust my
backups again.

Here are the details:


For some directories when running "duplicity restore --file-to-restore
xxx ..." I get an error "xxx not found in archive, no files restored."
even though "duplicity list-current-files" lists the directory xxx as
included in the archive. For others it works correctly. For some, I get
the following stack trace:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1251, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1244, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1198, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 538, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line
520, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line
493, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line
463, in patch_seq2ropath
    assert len( patch_seq ) == 1, len( patch_seq )
AssertionError: 2

As I tried to find the cause of the failed assertion I've found that
many files in the backup chain have a patch_sequence like this (see
attached patch that adds some print statements):

before normalize_ps():
(
        (('.git', 'config') reg difftype=snapshot), # 1
        (('.git', 'config') None difftype=deleted), # 2
        None, # 3
        None, # 4
        None, # 5
        None, # 6
        None, # 7
        (('.git', 'config') reg difftype=diff) # 8
)

So #8 is a diff that should apply to a file that was deleted in #2. A
diff can only be applied to a file snapshot, so this situation shouldn't
happen. This invalid chain then correctly fails an assertion check in
patch_seq2ropath().

I've made a shell script that tries to reproduce the sequence above, but
in that case it works correctly (set #8 contains a snapshot, not a diff).

Files that duplicity reports as not present have a chain that looks like
this:

before normalize_ps():
(
        (('style',) sym difftype=snapshot),
        (('style',) None difftype=deleted),
        None,
        None,
        None,
        None,
        None,
        None
)

As far as I can see, these data structures correctly reflect what is
actually written in the difftar files. It looks like for some reason a
lot of files have been marked as deleted in set #2 and then this
information has been ignored when creating later sets.


A word about my setup:

I'm running Debian Squeeze on amd64. The backup has been created using
the version of duplicity shipped with Squeeze (0.6.08b-1+b1), while I
tried to restore with both the Debian version and the latest 0.6.15 from
the website.

Backups are made to a mounted (local) hard drive by periodically running
a shell script that does:

duplicity remove-older-than 6M --force "file://$dest"
duplicity incremental --exclude-filelist duplicity_exclude --volsize 256
- --full-if-older-than 2M "$src" "file://$dest"


Thanks for your help
Tomaž
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFOjdMRsAlAlRhL9q8RAvjuAKCXpz28tUaRuv91CkSXWDzMTXcSzwCgyx5u
/SWQvImCai2GA+vtgwHt2kw=
=eYaf
-----END PGP SIGNATURE-----

duplicity-debug.patch
Description: Text Data

repro.sh
Description: Bourne shell script