Re: [Gnu-arch-users] Slow inventories on large source trees

[Aaron Bentley]

Tom Lord wrote:
>     > From: Mike Hearn <address@hidden>
>
>     > On the WineHQ source tree, with an in-tree build, doing tla
>     > inventory --ids >/dev/null takes approximately 20 seconds on my
>     > athlon 1200 with 256mb of ram.
>
>     > This makes certain operations extremely slow, and this is
>     > problematic for us (the Wine developers thinking of adopting
>     > arch).
>
>     > The total source tree when a complete build has been done is
>     > approximately 560mb of data.
>
>     > I haven't tested with an out-of-tree build - while I suspect
>     > performance would be better, I'm really not sure we want to ask
>     > everyone to do out-of-tree builds to work around this slowness
>     > in arch.
>
>     > Are there any ideas for how to optimize this? The CVS gateway
>     > tree is using names tagging, though a real switch to arch would
>     > be using explicit tags in future.
>
>
> Names tagging, eh?
>
> Could you please (in no particular order and paying attention to and
> taking steps to mitigate effects of OS-level FS caching):
>
> 1) measure the speed of GNU `find' on this tree with 
>
>         find . '!' -uid 0
>
>    (I'm assuming that root does not own any files in this tree.  The
>     find expression is to force `find' to stat files.)

Is that necessary?  With names tagging, it shouldn't need to stat 
anything, should it?

> 2) report user/system/real times for `inventory'

On a P3 500:
real    0m11.019s
user    0m9.450s
sys     0m0.160s


> 3) build tla from scratch with profiling (you can set CFLAGS= on the 
>    `make' command line) and summarize where most time is being spent
>
> 4) (especially if it should turn out that most time is being spent in
>    re* or rx* functions) let us know (a) have you customized
>    =tagging-method and (b) do you use many .arch-inventory files?

233 .arch-inventory files

Note that my last email profiled "changes", not "inventory".  And it was 
using a library that already had the revision.

Aaron

--
Aaron Bentley
Director of Technology
Panometrics, Inc.