[Monotone-devel] Re: slowness of simple operations like "add"

[graydon hoare]

Georg-W. Koltermann wrote:

>      1. It seems 51% of the time is spent in calculate_base_manifest(),
>         index [8] in the profiling data.  Why do we need to calculate
>         the base manifest at all?  Couldn't we cache that in MT/revision
>         or some other file? 

we could, but we don't. I prefer not to have extra files in the MT 
directory, since they must be cross-checked for consistency. probably 
it'll be sufficient to store the revision->manifest mapping in a 
separate table, rather than re-reading the revision each time we want to 
look that up.

>      2. Most time of the above is spent in parse_edge() [19] and
>         parse_change_set() [29].  Why does that take so long when I have
>         only ONE revision in the database???

the single revision you have is large: it involved 5000 additions. 
unfortunately our current code involves *reading* the base revision (the 
one before the one you're committing) in order to work out the base 
manifest. this is unnecessary, but it's what we currently do. we could 
change this to speed it up, as I mention above, with a separate table.

>      3. Then 43% of time is spent in build_addition() [15] with major
>         contributions by apply_path_rearrangement() [25] and
>         concatenate_rearrangements() [20].  This is an area where I
>         cannot even guess what it does.

well, we have to check that what you're trying to do makes sense. to do 
that we must compare your request (adding a file) to the existing 
manifest, subject to any other not-yet-committed rearrangement stored in 
the MT/work file. to do this we build a path set representing your 
manifest, then rearrange it under the existing MT/work rearrangement, 
then use that rearranged path set to check the validity of your request. 
this involves, among other things, an intermediate "scratch revision" 
which adds all 5000 files in your manifest to a rearrangement.

there are other ways we could check this which involve fewer, and less 
expensive, intermediate forms. it is also possible to modify the 
internal consistency checking of rearrangements, such that they are 
cheaper to check, so the intermediate forms don't cost as much. there 
are a few things we could try.

> Ideally I would think that the "monotone add" could be just a quick
> check in the database if the file is already present and then a write to
> the work file.  But maybe I am too naive here.

that is, in broad brush strokes, what it's doing. unfortunately you have 
a lot of files in your manifest, and a lot of additions in your base 
revision, and both those turn into heavy factors on the intermediate 
things we need to calculate. clearly we need to work on making those 
things cheaper. thanks for the feedback.

-graydon