Re: Memory again

[Dmitry Antipov]

On 12/17/2011 01:55 AM, Stefan Monnier wrote:

> - Many memory problems are misdiagnosed as fragmentation problems, where
>    the real cause is either inefficient memory use, or memory leaks.
> - Real fragmentation problems exist but are fairly rare.

It would be nice to have a built-in (optionally selected at configure time)
method to provide a 'shot' of the heap to see how it's (mis)used. It's also
interesting whether it's possible to write a set of gdb macros for doing
something similar.

> - In some "fragmentation" cases, the Emacs process holds on to memory
>    without any good reason, i.e. it's been "free"d but the malloc library
>    does not want or can't return it to the OS because it did not use mmap
>    to allocate it.  This can be fixed, but this memory would really be
>    unused; it can still appear in RSS but only because the memory
>    pressure hasn't yet bumped it out of RAM.  IOW these cases may show
>    high memory use in terms of VSZ or RSS but fixing them is low priority
>    because their only direct cost is use of of swap space.

IIUC, this is not true, at least for Linux (see how zap_pte_range() updates
MM_ANONPAGES RSS counter; it's done when munmap() happens). Unmapped
(but still resident in RAM) pages aren't accounted as RSS of any process;
they're accounted separately and amount of space occupied by such pages is
'Active(anon)' in /proc/meminfo.

> - Fixing the remaining real fragmentation problems probably requires
>    a different allocator that can move objects to compact the memory
>    space.  Maybe such an allocator can be retrofitted into Emacs
>    (e.g. a mostly-copying allocator), but noone has tried it yet
>    (probably because the outcome is hard to predict, the problem it
>    attempts to fix only occurs rather rarely, and it can be difficult to
>    ensure that it doesn't affect negatively the more common cases).

It's not so hard to predict the memory-saving benefits of copying or
compacting collector - ideally such a collector should free everything
which is on a free lists in current collector, so an output of
garbage-collect may be used as a rough estimate of how much data can
be compacted. In my measurements, the typical amount of space which is
hold of a free lists is ~3-5% of total heap size; it's reasonable to expect
that copying/compacting collector may also decrease an overall heap
fragmentation, which may give a few more percents. Anyway, ~5% isn't
convincing enough to start a hard work on a new GC; instead, I believe
that some minor optimization of current algorithm and Lisp data representation
(vector allocation, compact or 'immediate' strings and something like cdr-coding
or unrolled lists) may give comparable, or even better, effect in the sense
of memory consumption.

Dmitry