[Monotone-devel] roster cache sizing

[Daniel Carosone]

When working with big trees (such as the netbsd repository), lots of
things are slow. One of the reasons for this is that the rosters are
very large, and therefore very few (ie, only the current one) fit in
the roster cache.

Quite some time ago, I made a quick hack (below) to try and get a feel
for the impact of this. Some operations are amazingly slow and use
enormous amounts of cpu, and I theorised it might be recreating older
rosters repeatedly.  Note that this tree is from cvs_import, and
therefore linear.

I had an old cvs_import done with 0.28.  I compared that to a new
cvs_import done with the new binary, and I compare a
regenerate_rosters of that old db done with stock 0.30 vs the altered
binary.  Both were notably faster, though the figure from the first
import is approximate.

                cvs_import      regenerate_rosters
default         ~528h (22d)     714:43:17
300m            347:18:49       185:19:34

Based on this, I think the size should be increased, or be tweakable
from a hook.  Especially when running as a server it could make a
serious difference.

--
Dan.

# 
# old_revision [1126fee8b8730b23a93922d4e4955a825788a150]
# 
# patch "constants.cc"
#  from [aa685613113f5ba800e3c64cc209e93072f86e7b]
#    to [6f2ea69798a45958413c4b9f80723a2d5eb0091d]
# 
============================================================
--- constants.cc        aa685613113f5ba800e3c64cc209e93072f86e7b
+++ constants.cc        6f2ea69798a45958413c4b9f80723a2d5eb0091d
@@ -37,11 +37,11 @@ namespace constants
   // the value of 7 MB was determined as the optimal point after timing
   // various values with a pull of the monotone repository - it could
   // be tweaked further.
-  size_t const db_version_cache_sz = 7 * (1 << 20);
+  size_t const db_version_cache_sz = 20 * (1 << 20);
 
   // the value of 7 MB was determined by blindly copying the line above and
   // not doing any testing at all - it could be tweaked further.
-  size_t const db_roster_cache_sz = 7 * (1 << 20);
+  size_t const db_roster_cache_sz = 300 * (1 << 20);
 
   // this value is very much an estimate.  the calculation is:
   //   -- 40 bytes content hash

pgptET6Lu8H7_.pgp
Description: PGP signature