|
From: | Joe Beda |
Subject: | Re: [rdiff-backup-users] rdiff-backup features? |
Date: | Wed, 11 Jul 2007 15:22:19 -0700 |
I'm still forming a plan (and busy at work so real hacking will have to wait for the weekend) but my thinking was to write forward diffs into the increment directory directly. I would change the way that current_mirror gets written and the diffing would have to take into account the mirror+forward diffs. After a forward increment is complete, the mirror would be moved up. (Having code to move the mirror forward and backward in time might be interesting too at some point -- that way you could undo bad backups by moving the mirror back in time and removing forward diffs). The biggest challenge will be making sure that we deal with the case where we have an increment but we don't have the metadata for that increment. That is the only unforeseen case that we'll want to be able to clean up. So -- the backup process would change to look like this: 1) Look for the presence of an update_mirror.<time> file. If it exists, delete any forward_mirror file and skip to step 6 and finish the current_mirror update. 2) Look for the presence of a forward_mirror.<time> file. If this exists, run a cleanup pass. This would consist of loading metadata and checking forward increments. Remove any forward increments that aren't present in the metadata. 3) Write out the forward_mirror.<time> file. 4) Walk the source and dest. Compare the source against the dest taking into account the latest increment/mirror at dest. Write out increments for any changes that exist forward of the mirror time. 5) Write out the new update_mirror.<time> file and delete the forward_mirror.<time> file. 6) Apply all of the forward increments. For each file update the mirror, write a backward increment and delete the forward increment. We would need to take care to make sure we can recover if this process is interrupted. (I haven't thought through the exact steps yet). 7) Write out the new current_mirror.<time> file and delete the update_mirror.<time> file. Thoughts? After the initial implementation, I would also want to change the metadata system to flush after so many bytes are transferred instead of after so many files have been written. This way we don't loose as much on interruption when the files are very large. (I backup up my CDs as very large flac files and can only write a few a night) Joe On 7/11/07, address@hidden <address@hidden> wrote:
On Mon, 9 Jul 2007, Andrew Ferguson wrote: >> Step A, Backup Sync: >> 1. Write changes as diffs against the current repository into >> something like $REPO/rdiff-backup-data/scratch/. >> 2. Simultaneously, write changes as rdiffs against the will-be new >> version for placement in $REPO/rdiff-backup-data/increments/. > > What do you do when the link goes down in the middle of Step A? The > file's you've already written scratch changes for could have changed, so > you'll have to recompare anyway. True, but the new data wouldn't need to be retransfered. > Maybe the verdict is we should keep NEW files in a 'scratch area' in > case the link goes down? Then, in the backup after a failed backup, > rdiff-backup could inspect the scratch area for already transferred new > files? Those could then be the basis for a diff transfer. Yes, but in the scratch area we should hold diffs to the changed files, not the changed files themselves for space saving concerns. -Eric _______________________________________________ rdiff-backup-users mailing list at address@hidden http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
[Prev in Thread] | Current Thread | [Next in Thread] |