Re: [adr-devel] Amanda Disaster Recovery Scripts.

[Alex Owen]

On Wed, 1 Sep 2004, Stefan Gregor Weichinger wrote:
> AO> Use OS features to make a snapshot of filesystems
>
> I do such snapshots via shell-script (find/tar).
> Is this feature available for Linux or is it Solaris-specific?

You have not been using LVM then!

I currently run LVM1 on Linux2.4 as that has snapshot support.
I believe LVM2 on Linux2.4 also has snapshot support but have not had time
to test this.
I also read that LVM2 on Linux2.6 has just got snapshot support... This
may be debian specific though.

But yes using LVM on Linux one can take a file system snapshot that works
at the block level using a copy-on-write system and creates a "virtual"
block device which "contains" the snapshot.

Solaris has an fssnap feature which does something simmilar but I think it
lies more at the FS level that the block level.

I don't know about IRIX but I think they have something simmilar to LVM -
XLV I think they call it. I'm guessing that some ind of snap shotting is
availabe ther also.

> AO> I also have a cron job that runs at say 10 to midnight which collects
> AO> important metadata ie partition layout etc and save this under
> AO> /etc/diskinfo/
>
> Selfmade script?

Yes: /usr/local/sbin/cron.save-diskinfo
---8<---
#!/bin/bash
export PATH="/usr/sbin:/usr/bin:/sbin:/bin"
SAVEDIR=/etc/diskinfo
cat /proc/partitions | ( while read major minor blocks name junk ; do
    if [ "$major" = "58" ] ; then
        # this is LVM so ignore here and get info from
        # /proc/lvm/* later
        :
    elif [ "$minor" = "0" ] ; then
        fname=$(echo $name | tr '/' '_')
        mv ${SAVEDIR}/${fname} ${SAVEDIR}/${fname}.old
        sfdisk -d /dev/${name} >${SAVEDIR}/${fname}
    fi
done)

if [ -d /proc/lvm ] ; then
        mv ${SAVEDIR}/proc_lvm_global ${SAVEDIR}/proc_lvm_global.old
        cat /proc/lvm/global >${SAVEDIR}/proc_lvm_global
fi

mv ${SAVEDIR}/swapon-s ${SAVEDIR}/swapon-s.old
swapon -s >${SAVEDIR}/swapon-s
---8<---

This could be enhanced to catch most norman situations and also then do a
run-parts over say /usr/local/sbin/cron.save-diskinfo.d which could save
local metainfo into /etc/diskinfo too (so admin can enhace for local
requorements or so the LVM stuff could be extracted and put into the
debian LVM package etc). Anyway some kind of hook which could prove
usefull in future!

> AO> I net-boot the failed server from the Amanda server and restore from tape
> AO> directly onto disk. There is a sun "blueprint" which shows you how to
> AO> modify a Solaris installation netboot image to be a generic readonly NFS
> AO> root image. I have also adapted this idea for Debian/Linux.
>
> Could you provide more details on this?
> I would love to be able to generate such an image every month or so,
> automated, if possible.

I currently have a woody based chroot exported via NFS.
I intend to upgrade this to a sarge based chroot some time soon.
I can take a copy and upgrade it by "chroot ./want-to-be-sarge apt-get
dist-upgrade" or something similar I hope... But yes Ultimatly I'd like to
script the build.

I started by debootstraping a woody chroot... I then to a bind mount of
the tftpdirectory to the chroot /boot. Then install some custom scripts
into $chroot/etc/mkinitrd/scripts/ which make the initrd created support
dhcp and NFS root. (requires a V. small patch to dhcp client) Then I
install debian kernel which also creates initrd.

I modify /etc/inittab to include the lines:
  1:2345:respawn:/sbin/getty -n -l /sbin/rootshell 38400 tty1
  T0:23:respawn:/sbin/serialrootshell

and have
--8<-- (/sbin/rootshell)
#!/bin/sh
exec /bin/login -f root
--8<--

--8<-- (/sbin/serialrootshell)
#!/bin/sh
speed=`stty speed --file=/dev/ttyS0`
exec /sbin/getty -n -l /sbin/rootshell -L ttyS0 $speed vt100
--8<--

This ensures a root shell on the console and serial port


Also the clever bit:

--8<--
# /etc/fstab: static file system information.
#
# file system    mount point   type    options                  dump pass
/dev/nfsroot     /             nfs     defaults                 0    0
proc             /proc         proc    defaults                 0    0
/dev/fd0         /floppy       auto    noauto,rw,sync,user,exec 0    0
/dev/cdrom       /cdrom        iso9660 noauto,ro,user,exec      0    0

tmpfs            /tmp          tmpfs   defaults                 0    0
tmpfs            /.etc_rw      tmpfs   defaults                 0    0
tmpfs            /.var_rw      tmpfs   defaults                 0    0
--8<--

--8<-- (/etc/init.d/mountfix)
#! /bin/sh

case "$1" in
    'start')
mount -r --bind /var /.var_ro
        cd /var
        tar -cf - --exclude='./cache*' --exclude='./lib/apt*' \
        --exclude='./lib/dpkg*' ./ | (cd /.var_rw ; tar -xpf -)
        ln -s /.var_ro/cache /.var_rw/cache
        ln -s /.var_ro/lib/apt /.var_rw/lib/apt
        ln -s /.var_ro/lib/dpkg /.var_rw/lib/dpkg
mount --bind /.var_rw /var

mount -r --bind /etc /.etc_ro
        cd /etc
        tar -cf - ./ | (cd /.etc_rw ; tar -xpf -)
        #fixup with symlinks if any filkes excluded from tar
mount --bind /.etc_rw /etc
    ;;
    'stop')
    ;;
    'restart')
        /etc/init.d/mountfix start
    ;;
    'force-reload')
        /etc/init.d/mountfix start
    ;;
    *)
    ;;
esac
--8<--

fstab mounts a tmpfs file system at /.var_rw
then in the mountfix sctipt we essentially bind mount /var to  /.var_ro
and copy /var to /.var_rw : at this point we can access the ro var fs via
/var and /.var_ro and a writabe copy at /.var_rw.
Then we bind mount  /.var_rw at /var...
now we have the original readonly NFS var fs available at /.var_ro and a
vritable copy at /var and /.var_rw.

We do the same with /etc/ which also needs to be writable!

> Any reason why you use rsh? Should be ssh these days ...

Fair point but...

I did not want the CPU overhead of encryption that ssh brings. My
environment is a LAN in a machine room... so that is how we deal with
security. Amanda passed the backup date in the clear so why bother
enrypting during restore?

I disable rsh access (via iptables firewall) to the internet at large. I
only punch a hole in that for the duration of the restore and only to the
machine being restored.


> Yes. Good plan ...
>
> AO> Debian has a package bootcd which allows me to make copy of my debian
> AO> server onto a bootable live CD. I can therefore boot this CD and use this
> AO> to restore the latest Amanda server image from tape.
>
> AO> The amanda database/config will be about one run out of date so I expect
> AO> that we would need to "amadmin import" an "amadmin export" file. Currently
> AO> I e-mail these off the amanda server at the end of the run. I guess I
> AO> should append this info to the end of that days amanda tape.
>
> There have been ways of doing this. I know of a solution to just
> generate that export-file after the amdump, position the tape after
> the last dump and append to it. So the tape would continue up-to-date
> metadata.
>
> I think this should go into AMANDA itself sooner or later.
> This would improve the disaster-recovery-usability ...
>
> AO> ----------------------------
> AO> Here are a couple of bash functions that I am developing to help speed up
> AO> the manual process... Perhaps they could grow to form the start of
> AO> ADR-0.0.1 ?
>
> Have to look at them ...
>
> AO> do_snap_restore should be extended to take the tape and file number info
> AO> found using find_snap_tape. It should then ensure the tape is loaded and
> AO> "mt asf" to the correct file on tape.
>
> AO> find_snap_tape should be changed to use "amadmin info" rather than
> AO> "amadmin find" as at present!
>
> AO> find_snap_tape(){ #fs=$1
> AO>     rsh  amanda.server -l backup /usr/sbin/amadmin Daily find
> AO> --sort dl   failed.server /snapshot${1}\\\$
> AO> }
>
> Shouldn't failed.server get something like $1 and fs $2 ?
What you mean something like:

export ADR_SERVER="amanda.server.com"

find_snap_tape(){
if [ $# -eq 1 ] ; then
        server=$(hostname)
        fs=$1
elif [  $# -eq 2 ] ; then
        server=$1
        fs=$2
else
        echo bad usage
fi
rsh  ${ADR_SERVER} -l backup /usr/sbin/amadmin Daily find --sort dl ${server} 
/snapshot${fs}\\\$
}