bug#45200: [PATCH] Force Glibc to free the memory freed

[Konstantin Kharlamov]

On Wed, 2021-02-03 at 11:02 -0500, Stefan Monnier wrote:
> > To answer you question in another email about memory benefits given default
> > Emacs settings: well, today I tried creating a testcase that would reproduce
> > problem with default settings, but haven't really succeeded at it.  I have
> > a testcase where Emacs without the patch has ≈20M more memory than the one
> > with the patch, but that's pretty small difference, and offhand I didn't
> > manage to get it increased any further.
>
> Thanks.  At least that seems to indicate that glibc does its job
> properly for the way we normally use it.
>
> > So, I'm thinking of wiring the functional of memory trimming to on-idle
> > hook, without possibility to disable it.
>
> That seems hard to do (luckily), since AFAICT idle hooks only exist via
> `run-with-idle-timer` and those can always be disabled with things like
> `cancel-timer`.

Ah, okay then. I just didn't know how on-idle code should be implemented; if it 
would
be just making a ELisp wrapper and adding it to the hook, which people can then
remove if they want to, that's fine. I just thought there's some additional work
required to make it "disableable", I'm not opposed to the idea of it being so 
per se.

> > Given how small performance impact in this case would be, I see no
> > reason to even implement an option to disable it.
> > Thoughts?
>
> My main thought is that if `malloc_trim` indeed makes a significant
> difference, it's a sign that Emacs's own code did its job (it called
> `free` as it should) and that the problem is in how glibc decided not
> to return the memory to the OS.
>
> That's a behavior that can (and will) change over time outside of
> our control.  So calling `malloc_trim` every time I stop typing for 10s,
> just on account of "maybe glibc didn't reclaim quite as much memory as
> we'd like this time" doesn't sound very appealing to me.
>
> It sounds like an ad-hoc quick hack.  I love being able to use ad-hoc
> quick hacks, but I don't like enabling such things by default when the
> only use-cases where they're known to be useful are fairly specialized
> (and discouraged by Emacs maintainers ;-)

I totally agree with you, and I too consider it a hack.

> So I think we need more info: do the glibc maintainers consider it
> normal for glibc to behave this way?  Why does it behave this way?

Very good question! I hope Glibc mainainers that are on CC list will be able to
answer. Even though I created a report on Glibc just some months ago, the 
problem per
se existed for a long time. And I've seen Carlos O'Donell leaving a small 
comment on a
similar issue with Ruby 2 years ago, which implies they're aware of this 
situation.

> Would the equivalent of `malloc_trim` happen anyway "at some point in
> the future"?  E.g. If you create a test case where you disable GC, let
> the memory use grow to 1GB, then reset the GC vars to their default and
> keep using Emacs modestly, will the memory ever be returned to the OS or
> is an explicit call to `malloc_trim` really indispensable?

No, the memory will never be returned to OS. I can tell that right away, 
because the
only difference would be `free()` getting called more often.

I think it is worth mentioning here that Glibc usually does return memory to 
the OS
without any need in malloc_trim(0). What happens in affected applications (such 
as
here) is that an application stumbles upon a very special allocation pattern, 
which
kinda breaks Glibc algorithms of returning memory.

That's btw to the point of whether this GLibc behavior is a bug: well, 
sometimes it
works, sometimes it doesn't — doesn't look like purposeful behavior to me :)

> But until we get all the answers to these questions, we can already
> install the code that exposes `malloc_trim` to ELisp.
>
>
>         Stefan
>