[Duplicity-talk] Some questions and observations about duplicity backups

[Roger James]

About two years ago I configured my subversion (svn) server (debian) to 
backup up its repository to rsync.net using backupninja. I configured it 
to run the following duplicity commands once a month.

duplicity cleanup -v9 --no-print-statistics --scp-command 'scp ' 
--sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111 
scp://address@hidden/gatekeeper-svn-backup
duplicity remove-older-than 60D -v9 --no-print-statistics --scp-command 
'scp ' --sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111 
scp://address@hidden/gatekeeper-svn-backup
duplicity full -v9 --no-print-statistics --scp-command 'scp ' 
--sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111  
--include '/work/svn' --exclude '**' / 
scp://address@hidden/gatekeeper-svn-backup

and the following commands every day but after the above commands if 
they had been run

duplicity cleanup -v9 --no-print-statistics --scp-command 'scp ' 
--sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111 
scp://address@hidden/gatekeeper-svn-backup
Debug:  duplicity remove-older-than 60D -v9 --no-print-statistics 
--scp-command 'scp ' --sftp-command 'sftp ' --encrypt-key 11111111 
--sign-key 11111111 scp://address@hidden/gatekeeper-svn-backup
Debug:  duplicity  -v9 --no-print-statistics --scp-command 'scp ' 
--sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111  
--include '/work/svn' --exclude '**' / 
scp://address@hidden/gatekeeper-svn-backup

I thought these commands would give a situation where I had a full 
backup every month and daily incremental backups to the last full backup.

A few days ago I had the opportunity to test this all out as we had a 
power outage and for some reason when the power came back the hard drive 
with the svn respository on it was toasted!

The first thing I noticed when doing the restore was that duplicity only 
found one full backup (the very first one) and a huge number of 
subsequent increments.

Looking at the backup files on rsync.net confimed this

sftp> ls
duplicity-full-signatures.2008-11-07T14:27:09+01:00.sigtar.gpg                  

duplicity-full.2008-11-07T14:27:09+01:00.manifest.gpg                           

duplicity-full.2008-11-07T14:27:09+01:00.vol1.difftar.gpg 
and so on up to                     
duplicity-full.2008-11-07T14:27:09+01:00.vol262.difftar.gpg                     


duplicity-inc.2008-11-07T14:27:09+01:00.to.2008-11-08T01:00:05+01:00.manifest.gpg
duplicity-inc.2008-11-07T14:27:09+01:00.to.2008-11-08T01:00:05+01:00.vol1.difftar.gpg
and so on up to (with some increments having more than one volume)
duplicity-inc.2009-10-17T01:01:40+01:00.to.2009-10-18T01:02:01+01:00.manifest.gpg
duplicity-inc.2009-10-17T01:01:40+01:00.to.2009-10-18T01:02:01+01:00.vol1.difftar.gpg

duplicity-new-signatures.2008-11-07T14:27:09+01:00.to.2008-11-08T01:00:05+01:00.sigtar.gpg
and so on up to
duplicity-new-signatures.2009-10-17T01:01:40+01:00.to.2009-10-18T01:02:01+01:00.sigtar.gpg

So my first question is. How do I get duplicity do what I want? 
(Ignoring backupninja for the time being!). That is, a full backup every 
month followed by up to a months worth of increments, with backups more 
than 60 days old discarded.

Now for the interesting bit, experiences trying to actually do the 
restore. The first thing I found out that it was just about impossible 
to get the backup files from rsync.net using the inbuilt scp process in 
duplicity I could never get the job to run to completion due to what 
appeared to be network failures detected in gpg/ssh. The most common 
were "authentication timeouts" and "MAC errors". I eventually gave up 
and used sftp to transfer the files to a local directory and ran 
duplicity on them there.

Has anyone else had this kind of experience with scp?

The next problem I encountered was that duplicity (gpg) reported that 
one of the backup files had been "tampered with". I ran gpg standalone 
on all the files that I had downloaded and found out that three of the 
files decrypted fine but reported digital signature mismatches. One was 
a volume in the middle of the initial backup, and the other two were 
volumes in the middle of an incremental backup (both in the same increment).

Does anyone know how this could occur? My experience of encrypted and 
signed files, is that data loss in transit usually means that files will 
not even decrypt! I wonder if this is something to do with these files 
being encrypted with an older version of gpg. The server is now running 
debian lenny, it was probably running sarge when these files were 
signed. But what about about the hundred or so other files with good 
signatures?

Because I had nothing to lose I decrypted the bad files and then 
re-signed and encrypted them manually. It was at this point that I 
discovered that I would also need to change the signatures in the 
duplicity manifests for these files. So I decrypted, edited and then 
re-encrypted and signed the relevant manifests. Success - the restore 
ran through to the end. So the big question was, was my svn respository 
OK. Looking at the decrypted tar archive files I could see the at the 
bad volumes contained lumps of subversion "rev" files. The subversion 
database maintains its own robust checksums (message digests) for these 
files. So I ran the subversion verification tool and this reported that 
all was well. This means that the backup data was good but checksums 
generated at the time were different from the checksums generated on the 
same data now!

Here is the gpg output on one of the bad files.

gpg: encrypted with 2048-bit ELG-E key, ID 82088CFB, created 2007-12-02
     "Roger James (Duplicity encryption key) 
<address@hidden>"
gpg: Signature made Fri 07 Nov 2008 18:28:39 GMT using DSA key ID B885BD48
gpg: Good signature from "Roger James (Duplicity encryption key) 
<address@hidden>"
gpg: WARNING: encrypted message has been manipulated!

Comments anyone, how can I avoid this happening again, suggestions 
gratefully received.?

Roger