[Monotone-devel] Memory allocation during commits

[Eric Anderson]

During a discussion with Nathaniel about a patch for improving the
performance of push/pulls with binary files
(https://savannah.nongnu.org/bugs/?func=detailitem&item_id=13300) that
got a 70x CPU reduction, the discussion changed to another patch in
the mainline to reduce memory usage.

I've now had a chance to look through memory usage committing a single
add of a 100MB file, the results are below.  The summary is that it
ends up bloating up to a ~800MB memory size during the execution.  My
analysis follows (line numbers are against revision 0.19) and then a
question.  I haven't yet done a similar analysis against network
push/pull, I though I would fix the problem here first.
        I 
Most of the memory usage is coming in database.cc:911 which is doing
base64<gzip<data> > dat_packed, which for a x byte buffer will
allocate 4*x bytes of storage for the packing.

Then another big chunk in database.cc:653 where the insertion is
executed and memory is allocated again. -- worse, all the intermediate
values are still allocated even though I think at this point they
could be freed.

Then another big chunk at database.cc:654 where we convert the c* to a
string, and then prune off the back if it is large.  Sadly, we then do
exactly the same vprintf again in the sqlite3_exec_vprintf, and then
sqllite3 does 4 allocations of similar size during the parsing and
execution.

Question 1: in database::put is it save to clear out dat, or is there
some codepath that could use that value after the put?  I would like
to clear it out to re-use that memory, but I was unable to tell for
sure that nothing could use it.

Question 2: what are people's feelings about patching sqlite if that
is necessary to eliminate some of the memory allocation?
        -Eric