Re: [rdiff-backup-users] rdiff-backup features?

[Joe Beda]

I'm still forming a plan (and busy at work so real hacking will have
to wait for the weekend) but my thinking was to write forward diffs
into the increment directory directly.   I would change the way that
current_mirror gets written and the diffing would have to take into
account the mirror+forward diffs.  After a forward increment is
complete, the mirror would be moved up.

(Having code to move the mirror forward and backward in time might be
interesting too at some point -- that way you could undo bad backups
by moving the mirror back in time and removing forward diffs).

The biggest challenge will be making sure that we deal with the case
where we have an increment but we don't have the metadata for that
increment.  That is the only unforeseen case that we'll want to be
able to clean up.

So -- the backup process would change to look like this:
1) Look for the presence of an update_mirror.<time> file.  If it
exists, delete any forward_mirror file and skip to step 6 and finish
the current_mirror update.
2) Look for the presence of a forward_mirror.<time> file.  If this
exists, run a cleanup pass.  This would consist of loading metadata
and checking forward increments.  Remove any forward increments that
aren't present in the metadata.
3) Write out the forward_mirror.<time> file.
4) Walk the source and dest.  Compare the source against the dest
taking into account the latest increment/mirror at dest.  Write out
increments for any changes that exist forward of the mirror time.
5) Write out the new update_mirror.<time> file and delete the
forward_mirror.<time> file.
6) Apply all of the forward increments.  For each file update the
mirror, write a backward increment and delete the forward increment.
We would need to take care to make sure we can recover if this process
is interrupted.  (I haven't thought through the exact steps yet).
7) Write out the new current_mirror.<time> file and delete the
update_mirror.<time> file.

Thoughts?

After the initial implementation, I would also want to change the
metadata system to flush after so many bytes are transferred instead
of after so many files have been written.  This way we don't loose as
much on interruption when the files are very large.  (I backup up my
CDs as very large flac files and can only write a few a night)

Joe

On 7/11/07, address@hidden <address@hidden> wrote:
> On Mon, 9 Jul 2007, Andrew Ferguson wrote:
> >> Step A, Backup Sync:
> >>    1. Write changes as diffs against the current repository into
> >> something like $REPO/rdiff-backup-data/scratch/.
> >>    2. Simultaneously, write changes as rdiffs against the will-be new
> >> version for placement in $REPO/rdiff-backup-data/increments/.
> >
> > What do you do when the link goes down in the middle of Step A? The
> > file's you've already written scratch changes for could have changed, so
> > you'll have to recompare anyway.
>
> True, but the new data wouldn't need to be retransfered.
>
> > Maybe the verdict is we should keep NEW files in a 'scratch area' in
> > case the link goes down? Then, in the backup after a failed backup,
> > rdiff-backup could inspect the scratch area for already transferred new
> > files? Those could then be the basis for a diff transfer.
>
> Yes, but in the scratch area we should hold diffs to the changed files,
> not the changed files themselves for space saving concerns.
>
> -Eric
>
>
> _______________________________________________
> rdiff-backup-users mailing list at address@hidden
> http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki