Re: [rdiff-backup-users] Tar replacement - format proposal

[Greg Freemyer]

On Fri, 2003-09-26 at 21:10, Ben Escoto wrote:
> On 26 Sep 2003 10:33:01 +0100 Kevin Spicer <address@hidden> wrote:
> > Interesting ideas.  You seem very focused on backup to disk, have you
> > considered what happens if someone wants to backup to tape.  Since the
> > index is at the end of the file they would need to read the entire file
> > to get the index, then rewind and read (possibily) the entire file again
> > to extract what they want.
> 
> I think this got cleared up later in your discussion with John
> Goerzen.  The entire file wouldn't need to be read to find the index.
> I don't have much personal experience with tapes (and none with
> industrial tape usage), but from reading the thread I get the
> impression that the format would be tape-writing friendly, in that it
> could be written to tape in a single pass, and not that tape
> unfriendly when it comes to reading.
> 
> Tar will be faster for restoring a whole archive from tape since no
> winding will be required, but this format might be quicker to restore,
> even from tape, a part of an archive, because the tape can wind there
> directly instead of needing to wind/read pass every file as tar
> typically requires.
> 
Ben,

I realize your not really talking about handling tape yet, but as an FYI
for the future:

Commercial backup software often writes "file marks" on the tape every
once in a while.  (Linux can do this with the mt command.  Also tape
file marks have nothing to do with traditional on disk files, but are a
unique tape concept.)  

I think a file mark every GB of data or so is not uncommon.  Actually
lots of more complex schemes could also be used, but every GB is a nice
simple way to look at it.

Then tape drives have the ability to move between file marks extremely
quickly.  Apparently they are physically very large and easy for the
tape drive to register even at very high speed.

So if you do this, then on the restore you can "seek" to the file mark
preceeding the data (often within 30 seconds or less), then advance the
correct number of blocks/records to the start of the actual data and
start reading.

As you say, tar's inability to take advantage of this type of tape
handling causes it to be extremely slow for partial restores.

Greg
-- 
Greg Freemyer