[Monotone-devel] Re: [PATCH] pre-cache database file to improve response time

[Joe Wilson]

Hi Benoît,

Let's put this in perspective. Just compiling any *one* of 
monotone's C++/STL/boost-using source files with -O3 will also
evict 300MB of pages from the cache - several times the size of 
the 70MB monotone database repository, or twice the size of
your 180MB repository. Surely you compile something or 
run other programs between monotone operations, don't you?

If your machine only has 100MB cache+buffer, then no strategy - 
mmap or otherwise will improve the speed of monotone on your system.
You can't buy a new machine with less than 512MB these days.

>> BenoÃt Dejean wrote:
>i think this is irrelevant. sqlite does random writes/reads. moreover,
>if you cat'ed your dvd iso before burning it, that wouldn't speed at all
>the burning process. but you get the point, if you're running out of
>ram, the kernel destroys cache pages that have just been read by your
>'preload' :)

The DVD image (which is several times the size of most machine's 
RAM) had to be read into memory in the first place in order to be 
burned to the DVD. Pages will be evicted and replaced as result 
(all be it at a slower pace). But your OS still manages to recover 
somehow.

>on small databases, i get exactly the same performance boost -- on
>startup -- for small operations -- if i do twice the same commands.

No argument here. Hence the suggested patch.
But how often do you run the same command over and over again
in succession?

>my database is 180MB, trust me when i say that cat'ing it is useless.
>linux cache+buffer on my desktop is about 100MB. reading the whole file
>is slow, and monotone is unlikely to access every single byte of the
>database. so it just trashes the cache.
>
>i've tested on net.venge.monotone with cold cache :
>- monotone log --last 1 : 4.3s
>- cat + monotone status : 3s + 2s
>
>but of course, with hot cache : 1.5s
>
>swapping back to evolution to finish this email : about 4s ...

Let's be real here. Your machine would have to swap the pages 
out to get evolution into your RAM-constrained machine's memory 
after doing any significant operation anyway. That program is not 
exactly small.

$50 of RAM would improve your computer's speed a great deal.
Monotone is not a light-weight program.

>> Until SQLite uses mmap() instead of read(), I don't see another way to 
>> speed up monotone disk read access nearly as easily and effectively.
>> 
>
>Again, i don't think this is a scalable solution. this may only give a
>short boost on startup with small databases. otherwise, it may just slow
>the whole startup. I'm happy that not every single app on my computer do
>this. ram is a precious resource.
>my main concern is CPU consumption. this is the very limiting factor of
>monotone on my computers.

I thought people with a decent amount of RAM could benefit from 
the patch. A 3 second database pre-load (if the db file is cold) 
can save 20 seconds of wall clock time on a checkout (for example). 
If the file is already in cache, then only 1/10th of second is 
required for the pre-cache operation, so basically it's a no-op.

My proposal may not be popular, but I seriously question whether a 
more textbook solution would gain anything over this simple measure.
If you've got the RAM, you might as well put it to use instead of
waiting for monotone. SQLite will evict roughly the same number 
pages as a matter of course as it runs anyway.

All I can say is just try the patch out for a few hours before 
jumping to conclusions.


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com