[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Duplicity-talk] Some questions and observations about duplicity backups
From: |
Roger James |
Subject: |
[Duplicity-talk] Some questions and observations about duplicity backups |
Date: |
Thu, 22 Oct 2009 18:59:53 +0100 |
User-agent: |
Thunderbird 2.0.0.23 (Windows/20090812) |
About two years ago I configured my subversion (svn) server (debian) to
backup up its repository to rsync.net using backupninja. I configured it
to run the following duplicity commands once a month.
duplicity cleanup -v9 --no-print-statistics --scp-command 'scp '
--sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111
scp://address@hidden/gatekeeper-svn-backup
duplicity remove-older-than 60D -v9 --no-print-statistics --scp-command
'scp ' --sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111
scp://address@hidden/gatekeeper-svn-backup
duplicity full -v9 --no-print-statistics --scp-command 'scp '
--sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111
--include '/work/svn' --exclude '**' /
scp://address@hidden/gatekeeper-svn-backup
and the following commands every day but after the above commands if
they had been run
duplicity cleanup -v9 --no-print-statistics --scp-command 'scp '
--sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111
scp://address@hidden/gatekeeper-svn-backup
Debug: duplicity remove-older-than 60D -v9 --no-print-statistics
--scp-command 'scp ' --sftp-command 'sftp ' --encrypt-key 11111111
--sign-key 11111111 scp://address@hidden/gatekeeper-svn-backup
Debug: duplicity -v9 --no-print-statistics --scp-command 'scp '
--sftp-command 'sftp ' --encrypt-key 11111111 --sign-key 11111111
--include '/work/svn' --exclude '**' /
scp://address@hidden/gatekeeper-svn-backup
I thought these commands would give a situation where I had a full
backup every month and daily incremental backups to the last full backup.
A few days ago I had the opportunity to test this all out as we had a
power outage and for some reason when the power came back the hard drive
with the svn respository on it was toasted!
The first thing I noticed when doing the restore was that duplicity only
found one full backup (the very first one) and a huge number of
subsequent increments.
Looking at the backup files on rsync.net confimed this
sftp> ls
duplicity-full-signatures.2008-11-07T14:27:09+01:00.sigtar.gpg
duplicity-full.2008-11-07T14:27:09+01:00.manifest.gpg
duplicity-full.2008-11-07T14:27:09+01:00.vol1.difftar.gpg
and so on up to
duplicity-full.2008-11-07T14:27:09+01:00.vol262.difftar.gpg
duplicity-inc.2008-11-07T14:27:09+01:00.to.2008-11-08T01:00:05+01:00.manifest.gpg
duplicity-inc.2008-11-07T14:27:09+01:00.to.2008-11-08T01:00:05+01:00.vol1.difftar.gpg
and so on up to (with some increments having more than one volume)
duplicity-inc.2009-10-17T01:01:40+01:00.to.2009-10-18T01:02:01+01:00.manifest.gpg
duplicity-inc.2009-10-17T01:01:40+01:00.to.2009-10-18T01:02:01+01:00.vol1.difftar.gpg
duplicity-new-signatures.2008-11-07T14:27:09+01:00.to.2008-11-08T01:00:05+01:00.sigtar.gpg
and so on up to
duplicity-new-signatures.2009-10-17T01:01:40+01:00.to.2009-10-18T01:02:01+01:00.sigtar.gpg
So my first question is. How do I get duplicity do what I want?
(Ignoring backupninja for the time being!). That is, a full backup every
month followed by up to a months worth of increments, with backups more
than 60 days old discarded.
Now for the interesting bit, experiences trying to actually do the
restore. The first thing I found out that it was just about impossible
to get the backup files from rsync.net using the inbuilt scp process in
duplicity I could never get the job to run to completion due to what
appeared to be network failures detected in gpg/ssh. The most common
were "authentication timeouts" and "MAC errors". I eventually gave up
and used sftp to transfer the files to a local directory and ran
duplicity on them there.
Has anyone else had this kind of experience with scp?
The next problem I encountered was that duplicity (gpg) reported that
one of the backup files had been "tampered with". I ran gpg standalone
on all the files that I had downloaded and found out that three of the
files decrypted fine but reported digital signature mismatches. One was
a volume in the middle of the initial backup, and the other two were
volumes in the middle of an incremental backup (both in the same increment).
Does anyone know how this could occur? My experience of encrypted and
signed files, is that data loss in transit usually means that files will
not even decrypt! I wonder if this is something to do with these files
being encrypted with an older version of gpg. The server is now running
debian lenny, it was probably running sarge when these files were
signed. But what about about the hundred or so other files with good
signatures?
Because I had nothing to lose I decrypted the bad files and then
re-signed and encrypted them manually. It was at this point that I
discovered that I would also need to change the signatures in the
duplicity manifests for these files. So I decrypted, edited and then
re-encrypted and signed the relevant manifests. Success - the restore
ran through to the end. So the big question was, was my svn respository
OK. Looking at the decrypted tar archive files I could see the at the
bad volumes contained lumps of subversion "rev" files. The subversion
database maintains its own robust checksums (message digests) for these
files. So I ran the subversion verification tool and this reported that
all was well. This means that the backup data was good but checksums
generated at the time were different from the checksums generated on the
same data now!
Here is the gpg output on one of the bad files.
gpg: encrypted with 2048-bit ELG-E key, ID 82088CFB, created 2007-12-02
"Roger James (Duplicity encryption key)
<address@hidden>"
gpg: Signature made Fri 07 Nov 2008 18:28:39 GMT using DSA key ID B885BD48
gpg: Good signature from "Roger James (Duplicity encryption key)
<address@hidden>"
gpg: WARNING: encrypted message has been manipulated!
Comments anyone, how can I avoid this happening again, suggestions
gratefully received.?
Roger
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Duplicity-talk] Some questions and observations about duplicity backups,
Roger James <=