[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Duplicity-talk] Some questions and observations about duplicity backups

From: Roger James
Subject: [Duplicity-talk] Some questions and observations about duplicity backups
Date: Thu, 22 Oct 2009 18:59:53 +0100
User-agent: Thunderbird (Windows/20090812)

About two years ago I configured my subversion (svn) server (debian) to backup up its repository to rsync.net using backupninja. I configured it to run the following duplicity commands once a month.

duplicity cleanup -v9 --no-print-statistics --scp-command 'scp ' --sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111 scp://address@hidden/gatekeeper-svn-backup duplicity remove-older-than 60D -v9 --no-print-statistics --scp-command 'scp ' --sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111 scp://address@hidden/gatekeeper-svn-backup duplicity full -v9 --no-print-statistics --scp-command 'scp ' --sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111 --include '/work/svn' --exclude '**' / scp://address@hidden/gatekeeper-svn-backup

and the following commands every day but after the above commands if they had been run

duplicity cleanup -v9 --no-print-statistics --scp-command 'scp ' --sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111 scp://address@hidden/gatekeeper-svn-backup Debug: duplicity remove-older-than 60D -v9 --no-print-statistics --scp-command 'scp ' --sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111 scp://address@hidden/gatekeeper-svn-backup Debug: duplicity -v9 --no-print-statistics --scp-command 'scp ' --sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111 --include '/work/svn' --exclude '**' / scp://address@hidden/gatekeeper-svn-backup

I thought these commands would give a situation where I had a full backup every month and daily incremental backups to the last full backup.

A few days ago I had the opportunity to test this all out as we had a power outage and for some reason when the power came back the hard drive with the svn respository on it was toasted!

The first thing I noticed when doing the restore was that duplicity only found one full backup (the very first one) and a huge number of subsequent increments.

Looking at the backup files on rsync.net confimed this

sftp> ls
duplicity-full-signatures.2008-11-07T14:27:09+01:00.sigtar.gpg duplicity-full.2008-11-07T14:27:09+01:00.manifest.gpg duplicity-full.2008-11-07T14:27:09+01:00.vol1.difftar.gpg and so on up to duplicity-full.2008-11-07T14:27:09+01:00.vol262.difftar.gpg

and so on up to (with some increments having more than one volume)

and so on up to

So my first question is. How do I get duplicity do what I want? (Ignoring backupninja for the time being!). That is, a full backup every month followed by up to a months worth of increments, with backups more than 60 days old discarded.

Now for the interesting bit, experiences trying to actually do the restore. The first thing I found out that it was just about impossible to get the backup files from rsync.net using the inbuilt scp process in duplicity I could never get the job to run to completion due to what appeared to be network failures detected in gpg/ssh. The most common were "authentication timeouts" and "MAC errors". I eventually gave up and used sftp to transfer the files to a local directory and ran duplicity on them there.

Has anyone else had this kind of experience with scp?

The next problem I encountered was that duplicity (gpg) reported that one of the backup files had been "tampered with". I ran gpg standalone on all the files that I had downloaded and found out that three of the files decrypted fine but reported digital signature mismatches. One was a volume in the middle of the initial backup, and the other two were volumes in the middle of an incremental backup (both in the same increment).

Does anyone know how this could occur? My experience of encrypted and signed files, is that data loss in transit usually means that files will not even decrypt! I wonder if this is something to do with these files being encrypted with an older version of gpg. The server is now running debian lenny, it was probably running sarge when these files were signed. But what about about the hundred or so other files with good signatures?

Because I had nothing to lose I decrypted the bad files and then re-signed and encrypted them manually. It was at this point that I discovered that I would also need to change the signatures in the duplicity manifests for these files. So I decrypted, edited and then re-encrypted and signed the relevant manifests. Success - the restore ran through to the end. So the big question was, was my svn respository OK. Looking at the decrypted tar archive files I could see the at the bad volumes contained lumps of subversion "rev" files. The subversion database maintains its own robust checksums (message digests) for these files. So I ran the subversion verification tool and this reported that all was well. This means that the backup data was good but checksums generated at the time were different from the checksums generated on the same data now!

Here is the gpg output on one of the bad files.

gpg: encrypted with 2048-bit ELG-E key, ID 82088CFB, created 2007-12-02
"Roger James (Duplicity encryption key) <address@hidden>"
gpg: Signature made Fri 07 Nov 2008 18:28:39 GMT using DSA key ID B885BD48
gpg: Good signature from "Roger James (Duplicity encryption key) <address@hidden>"
gpg: WARNING: encrypted message has been manipulated!

Comments anyone, how can I avoid this happening again, suggestions gratefully received.?


reply via email to

[Prev in Thread] Current Thread [Next in Thread]