[Monotone-devel] Large repo performance (again)

[Jeff Rizzo]

I haven't had a chance to attempt an import of the NetBSD repository in 
a while, but recent[1] successes the NetBSD folks have had cleaning up 
some of the weirder issues the repo and importing it into fossil and git 
made me want to give it a try under mtn again. Here's a brief 
rundown/report of my success.

Monotone version:  1.0:
domU22:riz  /usr/src> mtn --version
monotone 1.0 (base revision: a7c3a1d9de1ba7a62c9dd9efee17252234bb502c)

The CVS repo of "src" is almost 9GB:

domU22:riz  /usr/src> sudo du -sh /cvsroot/src
8.7G    /cvsroot/src

...and it takes approximately 19 hours on a Xen DomU with 10GB RAM and 4 
cpus to import it into monotone, yielding a db of 3.0 GB:

mtn -d netbsd.mtn -b org.NetBSD.src cvs_import /cvsroot/src
{chug, chug ... 19 hours later}

domU22:riz  ~> du -sh netbsd.mtn
3.0G    netbsd.mtn

This is great!  Much better than a couple years ago.  It's actually 
doable, for one.  :-}

Now, as far as actually *using* the repository goes...

domU22:riz  /usr/src> time mtn -d ~/netbsd.mtn -b org.NetBSD.src co .
mtn -d ~/netbsd.mtn -b org.NetBSD.src co .  50.41s user 867.94s system 
26% cpu 57:22.98 total

... nearly an hour to check out!  it's 1.2GB of sources, checked out.

Then, *switching branches* took almost 2h, and failed!

domU22:riz  /usr/src> time mtn update -r h:org.NetBSD.src.netbsd-6 
--move-conflicting-paths
mtn: expanding selection 'h:org.NetBSD.src.netbsd-6'
mtn: expanded to '115b26019a7cca13433bf1915c5ee025df11134b'
mtn: selected update target 115b26019a7cca13433bf1915c5ee025df11134b
mtn: target revision is not in current branch
mtn: switching to branch 'org.NetBSD.src.netbsd-6'
mtn: [left]  5347af45af8d2944ac167f5ce5cdfe59cc123e9d
mtn: [right] 115b26019a7cca13433bf1915c5ee025df11134b
mtn: warning: cannot drop non-empty directory ''
mtn: moved conflicting path 'OLD' to '_MTN/resolutions/OLD'
mtn: error: renaming 'crypto/external/bsd/openssl/dist/crypto/sha' to 
'_MTN/detached/314500' failed: Too many links
mtn update -r h:org.NetBSD.src.netbsd-6 --move-conflicting-paths 75.34s 
user 2874.20s system 47% cpu 1:43:52.94 total
domU22:riz  /usr/src>


Clearly, there's still a way to go.  If anyone wants to work with me on 
this (I'm not a C++ guy, though I can do some C), please let me know.

+j



[1] OK, maybe 1.5 years ago.  But it *feels* recent!