bug#43389: 28.0.50; Emacs memory leaks

[Florian Weimer]

* Eli Zaretskii:

> For some time (several months, I think) we have reports from Emacs
> users that the memory footprints of their Emacs sessions sometimes
> start growing very quickly, from several hundreds of MBytes to several
> gigabytes in a day or even just few hours, and in some cases causing
> the OOMK to kick in and kill the Emacs process.

The last time I saw this was a genuine memory leak in the Emacs C code.
Just saying. 8-)

> The questions that I'd like to eventually be able to answer are:
>
>   . is this indeed due to some malloc'ed chunk that is being used for
>     prolonged periods of time, and prevents releasing parts of the
>     heap to the system?  IOW, is this pathological, but correct
>     behavior, or is this some bug?
>
>   . if this is correct behavior, can Emacs do something to avoid
>     triggering it?  For example, should we consider tuning glibc's
>     malloc in some way, by changing the 3 calls to mallopt in
>     init_alloc_once_for_pdumper?
>
> Your thoughts and help in investigating these problems will be highly
> appreciated.  Please feel free to ask any questions you come up with,
> including about the details of Emacs's memory management and anything
> related.

There is an issue with reusing posix_memalign allocations.  On my system
(running Emacs 27.1 as supplied by Fedora 32), I only see such
allocations as the backing storage for the glib (sic) slab allocator.
It gets exercised mostly when creating UI elements, as far as I can
tell.  In theory, these backing allocations should be really long-term
and somewhat limited, so the fragmentation peculiar to aligned
allocations issue should not be a concern.

There is actually a glibc patch floating around that fixes the aligned
allocation problem, at some (hopefully limited) performance cost to
aligned allocations.  We want to get it reviewed and integrated into
upstream glibc.  If there is a working reproducer, we could run it
against a patched glibc.

The other issue we have is that thread counts has exceeded in recent
times more than system memory, and glibc basically scales RSS overhead
with thread count, not memory.  A use of libgomp suggests that many
threads might indeed be spawned.  If their lifetimes overlap, it would
not be unheard of to end up with some RSS overhead in the order of
peak-usage-per-thread times 8 times the number of hardware threads
supported by the system.  Setting MALLOC_ARENA_MAX to a small value
counteracts that, so it's very simple to experiment with it if you have
a working reproducer.

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill