Re: [rdiff-backup-users] Simple nightly script?

[David Kempe]

Hi Greg,
After much experimentation with various scripts, I am happy with this 
latest rdiff-backup parser we have written. Script is written by 
Christian Marie, who also wrote the rdiff-backup nagios plugin on 
nagiosexchange.org.

The script is designed to be passed on stdin the output of an 
rdiff-backup job or multiple jobs with -v5 or greater and 
--print-statistics both turned on. It can deal with many jobs passed in 
ie: backup.sh | parse-rdiff-backup
where backup.sh might run three jobs.
check it out - you get a nicely formatted email that says things like 
Backup(s) OK 3/3 in the subject, and a summary of the job like so:

Status of backup(s) follow

Backup completed successfully
        Direction: /home/sol1backup/cookie-data to /var/backupdisk/cookieman
        Backup started: Tue Oct 17 23:00:06 2006
        Source files: 61248
        Source files size: 22290479056 (20.8 GB)
        New files size: 6224482 (5.94 MB)
        Deleted files size: 308229 (301 KB)
        Destination size change: 18.4 MB
        Errors: 79


Anway, Here is the script and config file. Read the source if you want 
more info.... :)

Script:
--------------------------------

#!/usr/bin/env ruby
require 'yaml'
require 'tempfile'

class Parser
  attr_reader :status
  DEFAULT_CONFIG = File::SEPARATOR +
                   File.join('etc', 'parse-rdiff-backup.conf')
  BACKUPOKCORRECT = /TotalDestinationSizeChange/

  def initialize data, config = DEFAULT_CONFIG
    begin
      @log = data
      @config = YAML::load_file(config)['parser']
      @statistics_output = []
      @statistics = []
      @config['statistics'].each do |key,value|
        @statistics << SessionStatistic.new(key, value)
      end
      parse!
    rescue Exception => exception
      @status = BackupStatus::ERROR
      @error = "Error: #{exception.class} (#{exception.message})\n"
      @error += "Backtrace:\n"
      @error += exception.backtrace.join "\n"
    end
  end

  def parse!
    @status = BackupStatus::BROKEN
    @log.each do |line|
      @statistics.each do |statistic|
        if line =~ /#{statistic.regex}/
          if $1
            @statistics_output << statistic.message + $1
          end
        end
      end
      if line =~ BACKUPOKCORRECT
        @status = BackupStatus::OK
      end
    end
  end

  def statistics_output
      return @statistics_output.join("\n").gsub(/^/, "\t")
  end

  def error
    return @error.gsub(/^/, "\t")
  end

  private
  class SessionStatistic
    attr_accessor :message, :regex
    def initialize message, regex
      @regex = regex
      @message = message + ': '
    end
  end
end

class BackupStatus
  OK = 0
  BROKEN = 1
  ERROR = 2
end

class Message
  attr_accessor :subject, :message, :to, :cc, :log
  DEFAULT_CONFIG = File::SEPARATOR +
    File.join('etc', 'parse-rdiff-backup.conf')

  def initialize config=DEFAULT_CONFIG
    @config = YAML::load_file(config)['message']
    @subject = 'Unknown status'
    @message = "Status of backup(s) follow\n\n"
    @to = @config['main_contact']
    @cc = @config['cc_contact']
  end

  def send
    temp = Tempfile.new "rdiff_backup_log"
    temp.puts @log
    temp.close

    zipped_log = "#{temp.path}.zip"

    `zip #{zipped_log} #{temp.path}`

    IO.popen(
      'nail ' +
      "-a '#{zipped_log}' " +
      "-c 'address@hidden' " +
      "-s 'address@hidden' 'address@hidden'",
      'w'
    ) do |nail|
      nail.puts @message
    end

    File.unlink zipped_log
  end
end

def split_log log
  marks = []
  logs = []

  log.each_with_index do |line, index|
    if line =~ /Starting mirror|increment\ operation (.*) to (.*)/
      marks << index
    end
  end

  marks.size.times do |index|
    first =  marks[index]
    last = marks[index+1] || log.size

    logs << log[first..last].to_s
  end
  return logs
end

if __FILE__ == $0
  parsers = []

  input = ARGF.readlines

  split_log(input).each do |log|
    parsers << Parser.new(log)
  end

  ok_backups = 0
  message = Message.new

  parsers.each do |parser|
    case parser.status
    when BackupStatus::OK
      ok_backups += 1
      message.message += "Backup completed successfully\n"
      message.message += parser.statistics_output
      message.message += "\n\n"
    when BackupStatus::BROKEN
      message.message +="Backup did not complete\n"
      message.message += parser.statistics_output
      message.message += "\n\n"
    when BackupStatus::ERROR
      message.message += "Error running parsing script\n"
      message.message += parser.error
      message.message += "\n\n"
    end
  end

  if parsers.size == 0
    message.subject = 'Backup(s) unknown'
    message.message = 'Status unknown, did not get a valid log.'
  elsif ok_backups == parsers.size
    message.subject = 'Backup(s) OK'
  else
    message.subject = 'Backup(s) failed'
  end
  message.subject += " (#{ok_backups}/#{parsers.size})"
  message.log = input

  message.send
end

--------------------------
Config file:
-------------------------
cat /etc/parse-rdiff-backup.conf
message:
  main_contact: address@hidden
  cc_contact: address@hidden

parser:
  statistics:
    Direction: Starting mirror|increment\ operation (.*)
    Backup started: ^StartTime [\d\.]+ \((.*)\)$
    #EndTime: ^EndTime (.*)$
    #ElapsedTime: ^ElapsedTime (.*)$
    Source files: ^SourceFiles (.*)$
    Source files size: ^SourceFileSize (.*)$
    #MirrorFiles: ^MirrorFiles (.*)$
    #MirrorFileSize: ^MirrorFileSize (.*)$
    #NewFiles: ^NewFiles (.*)$
    New files size: ^NewFileSize (.*)$
    #DeletedFiles: ^DeletedFiles (.*)$
    Deleted files size: ^DeletedFileSize (.*)$
    #ChangedFiles: ^ChangedFiles (.*)$
    #Changed files size: ^ChangedFileSize (.*)$
    #ChangedMirrorSize: ^ChangedMirrorSize (.*)$
    #IncrementFiles: ^IncrementFiles (.*)$
    #IncrementFileSize: ^IncrementFileSize (.*)$
    Destination size change: ^TotalDestinationSizeChange \d+ \((.*)\)$
    Errors: ^Errors (.*)$


Script is licensed under the GNU GPLv2 and Author is Christian Marie 
while working for Solutions First.

thanks

dave





Greg Freemyer wrote:
> All,
>
> I've looked at the examples at 
> http://www.nongnu.org/rdiff-backup/examples.html, but none of them seem 
> to address automated nightly scripts and error handling.
>
> Are there some more complex examples available?
>
> === Details
> I haven't used rdiff-backup for a couple of years and I never did have 
> it integrated into a nightly cron script for production use.
>
> My needs have changed and I want to give it another shot.
>
> I'm backing up to a local directory on a dedicated backup disk so it is 
> easy enough to add "rdiff-backup /src /backup" to my backup script.
>
> But what about catching errors?
>
> Seems like I should be sending output to a log file, grepping thru it 
> and e-mailing it to myself if anything goes wrong.
>
> I've looked at the examples at 
> http://www.nongnu.org/rdiff-backup/examples.html, but none of them seem 
> to address this common need.
>
> Thanks
> Greg
> --
> Greg Freemyer
> The Norcross Group
> Forensics for the 21st Century
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> rdiff-backup-users mailing list at address@hidden
> http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki