[Monotone-devel] Speeding up automate stdio

[Patrick Georgi]

Hi,

I'm still working on an hg2mtn facility and tried to import the files of 
the first revision of that hg repository (34000 files) with automate 
stdio using the put_file command.

On my regular filesystem (ZFS), it took approx. 20 minutes.
For fun, I tried again with the monotone repository on tmpfs (swap based 
ramdisk), and came out with only a bit more than 5 minutes.

ZFS is usually a pretty fast filesystem with aggressive caching before 
it writes to disk, but it wrote small chunks to disk all the time - so I 
blamed transactions, and added a big transaction in automate stdio 
around "everything".
With that change, it took only 4 minutes on tmpfs, and feeling more 
courageous, I removed the transaction guards in put_file (there's a big 
one around it, after all), which gave another 30 seconds.

Then I tried on ZFS again - and it finished in 4:50, so it's 4 times 
faster! (for this absolutely non-scientific and specialized test)


What to do with it result: I'm thinking about adding an option to 
automate stdio that tells it to use one global lock 
("--all-or-nothing"?) and disables the transaction guards for every 
single commands it calls, but I wanted to discuss it first, as I have no 
idea what kind of impact such a change might have.

I'm thinking of implementing it as follows:
- add a transaction_guard around exec_from_automate(..) invocation in 
"automate" (non-stdio)
- add an optional transaction_guard around exec_from_automate(..) in 
"automate stdio"
- add global transaction_guard in "automate stdio" (or not: that should 
be covered by the first change already)
- remove all old transaction_guards in automate commands

I see one problem with this change: Reusing the automate commands in 
some other code path means that you get to do your transactions yourself 
(and might forget them)

Comments highly appreciated :-)


Regards,
Patrick Georgi