[Monotone-devel] Re: How to apply "automate get_revision" data

[Brian Downing]

Nathaniel Smith wrote:
> On Sun, Nov 11, 2007 at 11:39:55AM -0600, Brian Downing wrote:
> > I was wondering if there is a reliable way to apply the data from 
> > "automate get_revision" to transform one revision to the next.  I'm 
> > toying with a monotone to git converter, which is pretty simple,
> > but "automate get_manifest_of" is painfully, painfully slow.  (Around 
> > 3/4 second for the OpenEmbedded repository on my machine.)
>
> You might want to try using 'automate stdio'; if you do
> get_manifest_of through automate stdio, then the successive
> invocations can share caches.  Depending on what order you're fetching
> the manifests in, this may turn an O(n) operation into an O(1) one
> (where n is history size -- delta reconstruction has to walk the
> history graph).

I am actually using automate stdio for the conversion (or rather, I was 
using the new Monotone::AutomateStdio Perl module).  It still took quite 
some time per manifest; enough so that I figured there was no caching 
going on.  I'm fetching them in whatever order toposort returns them in.

Let me cook up a quick benchmark for the OpenEmbedded database...

Actually, it's much worse than I remembered:

manifest took 4.795 seconds
manifest took 4.711 seconds
manifest took 4.663 seconds
manifest took 4.716 seconds
manifest took 4.858 seconds
manifest took 4.920 seconds
manifest took 4.857 seconds
manifest took 4.853 seconds
manifest took 4.867 seconds
manifest took 4.890 seconds
manifest took 4.906 seconds
manifest took 4.991 seconds
manifest took 5.007 seconds
manifest took 4.946 seconds
manifest took 4.936 seconds

The first manifest up there had 6226 files.  Actually, seeing those 
again, I'm finding it hard to believe it could possibly be that bad.

I checked and the mtn process is using nearly 100 percent of the CPU, so 
it's not Perl that's being inefficient.

This is with:
monotone 0.37 (base revision: c21eefc002b8f9c430e9f4cc16c4af7b852f54ec)

> In general, there is no order you can apply them in; they are to be
> applied "all at once".  It's more like a declarative description of
> how two trees map to each other than it is a procedural description
> of how to produce one tree from another.  The algorithm for working
> with such things is a little subtle; the best description I know of
> is:
>   
> http://mebentley.blogspot.com/2005/12/tree-transforms-on-posix-filesystems.html
> (This is Aaron Bentley talking about bzr, but mtn and bzr
> independently invented identical algorithms here, so it still
> applies.)

Interesting.  I'll see if that'll be too complex for me to implement.

-bcd