Re: [rdiff-backup-users] rdiff-backup features?

[Joe Beda]

On 7/11/07, address@hidden <address@hidden> wrote:
> On Wed, 11 Jul 2007, Joe Beda wrote:
>
> > (Having code to move the mirror forward and backward in time might be
> > interesting too at some point -- that way you could undo bad backups
> > by moving the mirror back in time and removing forward diffs).
>
> I rather like this strange option.  Can anyone think of a real-life use of
> this feature?  Could backups be made while the mirror was back in time,
> but had forward increments several day (weeks, months...) ahead?  It would
> be kind of cool to move the mirror forward and back in time.  If this is a
> side effect of your implementation, please do include an option for it!
>
> I like Andrew's suggestion of "The user could, optionally, "tie off" that
> backup as "incomplete" and move forward." for when a backup
> fails, but you need to apply the forward increments and move on without
> completing that particular backup (for some reason).

Yeah -- I like that too.  If there are multiple forward increments
before we get a full snapshot, we could create backward increments for
each of those.

> > The biggest challenge will be making sure that we deal with the case
> > where we have an increment but we don't have the metadata for that
> > increment.  That is the only unforeseen case that we'll want to be
> > able to clean up.
> >
> > So -- the backup process would change to look like this:
> > 1) Look for the presence of an update_mirror.<time> file.  If it
> > exists, delete any forward_mirror file and skip to step 6 and finish
> > the current_mirror update.
>
> Is this part of the recovery if #6 is interrupted?

Exactly -- I'm trying to think through the recovery cases.

> > After the initial implementation, I would also want to change the
> > metadata system to flush after so many bytes are transferred instead
> > of after so many files have been written.  This way we don't loose as
> > much on interruption when the files are very large.  (I backup up my
> > CDs as very large flac files and can only write a few a night)
>
> Good idea.  Graceful recovery after out-of-disk and kill-9 (power failure)
> conditions would be nice too; perhaps this your forward increment idea
> would allow for this by deleting the forward increments and keeping a
> consistent backup tree at all times if this were to happen.  Currently,
> rdiff-backup does not always recover from a power failure/kill-9.

If we want to be really careful, we'll probably want to make sure we
flush/sync when we are in a consistent state and, in other cases, know
when we aren't.  We shouldn't assume that just because something was
handed off to the OS that it was written to disk.  I'm not sure what
the best way to do that would be -- perhaps recognize when we had a
less than graceful termination and verify the hash in the metadata?
(Perhaps apply the forward diffs and compute a sha1 hash to compare
against the metadata?  We'd want to do this in a streaming way and not
have to write temp files to disk.)

Joe