[Savannah-hackers-public] Re: address@hidden: Re: [Savannah-users] backup frequency]

[Sylvain Beucler]

> Sylvain - do you have or remember any details on the last savannah
> disaster? Someone had mentioned there had been a RAID corruption that
> caused downtime before, perhaps around 2007.

Yes, double disk failure, so the RAID 5 could not be recovered by
definition.

2.5 days downtime, delay mostly due to buying new disks which weren't
available in stock, and bringing+copying the backup at the
colo. Restored from weekly backup + partial (because interrupted
during the crash) rsync backup.

The backup was copied to the new disk from a rescue livecd, but
without the --numeric-ids option, resulting in some uids swaps,
fortunately fixable.

https://savannah.gnu.org/forum/forum.php?forum_id=4792


> Also I believe Sylvain has set up rsyncs with another site for another
> redundant level of backup protection.

That was a punctual action during the recovery, but not anymore.

The next goal is to install DRBD with another box at GNAPS (we tried
to do it with 2 external boxes but there were bandwidth issues).


> Sylvain - what rsync options are you using for those? We are now using:
[snip]
> --delete-after
> (instead of --delete which is the same as --delete-before)

Bad idea if there's no additional checks! E.g. just chmod 0000 a
directory and no file will be deleted, resulting in inconsistent
backups.

Also note that the filesystem corruption made files and dirs just
"disappear", so rsync would not necessarily hit any error.


In general I use
  rsync -qaHS --numeric-ids --delete-excluded --exclude...
+ manual checks (cronjob mails), or I rely on backuppc to filter
criticial errors (which doesn't use rsync directly but uses Perl
bindings).

-- 
Sylvain