Re: [Gnu-arch-users] Avoiding ancestor scan during get with revision library

[Aaron Bentley]

Tom Lord wrote:
>     > From: Aaron Bentley <address@hidden>
>     > Agreed.  It's too damn slow.  But for the moment, it seems the latency 
>     > issues are inevitable.  Using pipelining, parallel downloads, a smart 
>     > server, or changing the archive format could improve matters.
>
> It can be solved more simply than any of that.

By "it", I mean the archive scanning.  Once we know where the cacherevs 
are, sure we can use 'em.  But I'd like to see the scanning sped up.  On 
DSL, you can still wait minutes before it even *starts* downloading.

> In this case, given a very recent cacherev and no recent ancestor in
> the revlib, tla should give up on hard-linking fairly quickly and just
> build a non-shared revlib tree.   The exact point at which to give up
> is something we can pick a default for and let people tweak with a
> library-config parameter.   

Actually, what I'd like to do is replace almost all of library_add() 
with build_revision()*.  The reason I haven't done that yet is that I'd 
like to get the backbuilder to use the changes --link functionality to 
hardlink anything that isn't already hardlinked, so long as it has some 
kind of relation in the revision library.  Once that's in place, we 
won't have to trade storage space for download speed.

> It's also something we can choose
> automagically if we start recording file sizes in archives (and this
> is like the N+1th reason we have for doing so).

So actually, it looks like the Nth reason (speeding up revision 
building) to me, but definitely desirable.

Something that would be quite valuable in this would be non-volatile 
ancestry data in the tree.  Patchlogs are almost right (and I've got 
tools that use 'em), but since patchlogs can be removed (e.g. by replay 
--reverse, or by your good self), we can't rely on them in every case.

Local ancestry data would make it easier to
1. make revlibs extremely efficient
2. build revisions from their siblings (build foo from bar, when both 
foo and bar are derived from baz)
3. make star-merge cross tag boundaries when seeking a common ancestor

* okay, not actually arch_build_revision(), but a close relation that 
does hardlinking when it can.

Aaron

--
Aaron Bentley
Director of Technology
Panometrics, Inc.