Re: [Chicken-users] clueless about memory consumtion

[Felix]

From: Jörg "F. Wittenberger" <address@hidden>
Subject: [Chicken-users] clueless about memory consumtion
Date: Mon, 05 Jul 2010 21:09:44 +0200

> As it appears from more or less regular watching "top" I see the process
> as if was growing once open a time for a while in steps of 2M per second
> an then rest a long time.  There is no apparent correlation with
> external activities.  But this is kind of analysis not only tedious...
> 
> Some chicken debug features might be taboo: anything, which will write
> to (current-output-port) during GC, since this has been cluttering my
> files before (e.g., usage of "print" from chicken's library.scm when
> running finalizers).  (Besides this memory statistics being too complex
> for my brain to digest.)

Thanks for the hint, that line really should go to stderr. I agree
that the output is rather dull to look at. It would be interesting to
see heap-resizing information. Can you correlate the change in
memory consumption with heap-resizings?

> 
> I already ruled out nearby possibilities:
> - no malloc(2)/free(2) (including gc-root handling) mismatch in my code
> - no callbacks to Scheme from C

Ok.

> 
> + There is probably no pure algorithmic error; the "same" program
> (except for the compatibility layer) runs in rscheme with quite a
> different consumption pattern.  (It starts out with about 2 times a much
> memory, grows within the first hour to 4 times the initial amount and
> stays there almost constant - whereby the difference has a change to be
> attributed to some string->symbol usage.)  But since I can't completely
> rule that out...

Any comparisons with rscheme are pointless, since too many variables
influence the memory consumption. Tiny implementation details can
heavily change the allocation and reclamation behaviour of a program.

> 
> So the questions:
> 
> - how could I find out, how much memory chicken thinks it has allocated?

(##sys#memory-info) returns a vector containing the current heap- and
nursery sizes.

> - how could I hook into the memory allocation process to signal back to
> chicken for logging or other actions?

You can't for nursery allocations. For heap-allocations (allocations
that want a larger chunk than is seen fit for the nursery take place
in C_allocate_vector - there is no hook, but we can add one).
You can hook into GC, see C_pre_gc_hook and C_post_gc_hook.

> - how could I plot/walk the allocated memory to find out which objects
> are there and why?

If you find the debug output too complex for your brain, then this will
not help you. Analysis of memory consumption is rather difficult. It should
at least be possible to dump the type of objects on the heap. I'll look
into that.

I suggest you enable "-:dw" (after fixing that "forcing finalizers" output)
and we look at how the heap grows.


cheers,
felix