No. A verify run does not detect the data corruption. Also I've
simplified the reproduction of this bug to rule out anything related to
the unessential specifics of the usage scenario I was testing.
First backup::
duplicity --volsize 5 /data file:///tmp/backup/uninterrupted
Second backup::
duplicity --volsize 5 /data file:///tmp/backup/interrupted
Interrupted repeatedly with CTRL-C.
Now restore the interrupted and uninterrupted backups and compare::
duplicity file:///tmp/backup/uninterrupted /tmp/restore/uninterrupted
duplicity file:///tmp/backup/interrupted /tmp/restore/interrupted
cd /tmp/restore/uninterrupted
find -type f | xargs md5sum> md5sums
cd /tmp/restore/interrupted
find -type f|xargs md5sum> md5sums
cd /tmp/restore
# one corrupted file for every time we use checkpoint/restore
diff -u {interrupted,uninterrupted}/md5sums
Now that we've confirmed data corruption has happened let's try to
detect it via a verify run::
duplicity verify file:///tmp/backup/interrupted /data
Verify complete: 1864 files compared, 0 differences found.
No dice. Silent and deadly. If you try restoring the full backup YOU
WILL suffer data loss.
Just in case you're hoping the next incremental backup will smooth
things over, no such luck. In fact it gets worse. The next incremental
backup will corrupt the backup to the point where it won't restore at
all:
duplicity file:///tmp/backup/interrupted /tmp/restore/interrupted
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Fri Jul 30 04:44:24 2010
Traceback (most recent call last):
File "/usr/bin/duplicity", line 1251, in<module>
with_tempdir(main)
File "/usr/bin/duplicity", line 1244, in with_tempdir
fn()
File "/usr/bin/duplicity", line 1198, in main
restore(col_stats)
File "/usr/bin/duplicity", line 538, in restore
restore_get_patched_rop_iter(col_stats)):
File "/usr/lib/python2.5/site-packages/duplicity/patchdir.py",
line 522, in Write_ROPaths
ITR( ropath.index, ropath )
File "/usr/lib/python2.5/site-packages/duplicity/lazy.py", line
320, in __call__
self.process_w_branch(index, self.root_branch, args)
File "/usr/lib/python2.5/site-packages/duplicity/lazy.py", line
293, in process_w_branch
branch.start_process, args)
File "/usr/lib/python2.5/site-packages/duplicity/robust.py", line
37, in check_common_error
return function(*args)
File "/usr/lib/python2.5/site-packages/duplicity/patchdir.py",
line 544, in start_process
assert ropath.index == (), ropath.index
The good news is that if you run verify now it will at least detect
something is terribly wrong:
duplicity verify file:///tmp/backup/interrupted /data
...
verify complete: 1864 files compared, 781 differences found.
Closer inspections shows that it looks like the first incremental backup
deleted the contents of all volumes which were corrupted in the first
full backup.
The second incremental backup restores the deleted contents. This makes
the latest backup restorable again. Now there are fewer data corruptions
but they still exist. I didn't test what happens if I try to use
checkpoint/restore in the incremental backups. I very much doubt it
would work.
Bug report filed here: https://bugs.launchpad.net/duplicity/+bug/613244
Cheers,
Liraz
address@hidden wrote:
Does a verify run detect them? .. ede
On 03.08.2010 04:16, Liraz Siri wrote:
I've found a much more serious bug with checkpoint/restore that leads to
silent data corruption. I'm using duplicity 0.6.09 installed
on a 32-bit Ubuntu system, backing up to the local filesystem.
I discovered this by running two full backups with the same parameters.
duplicity --archive-dir /var/cache/duplicity --volsize 10
--include-filelist /tmp/filelist --exclude '**' /
file:///var/tmp/duplicity
The first backup I allowed to complete without interruption.
The second backup I repeatedly stopped/resumed after the first volume
had been created.
I then restored the backups to /tmp/restore.broken and /tmp/restore and
compared them as follows:
cd /tmp/restore
find |xargs stat -c "%n %U %G %A %s"> statlist
cd /tmp/restore.broken
find |xargs stat -c "%n %U %G %A %s"> statlist
cd ../
diff -u /tmp/restore/{statlist,statlist.broken}
I discovered what looks like one corrupt file for each time I
CTRL-C/resumed the backup. I'm pretty sure these are the files duplicity
resumed from.
I would recommend that until this is fixed nobody use the
checkpoint/restore feature but unfortunately there doesn't seem to be
any way to disable it (in duplicity 0.6).
If you've already used checkpoint/restore in a version of duplicity that
has this bug it would be wise to recreate your backups immediately. Just
make sure the backup completes.
Cheers,
Liraz
_______________________________________________
Duplicity-talk mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
_______________________________________________
Duplicity-talk mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/duplicity-talk