bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

[Ihor Radchenko]

> The doc string assumes "normal" operation, it doesn't assume some Lisp
> command runs for a long time, thus preventing redisplay.

Sounds counter-intuitive. Elisp commands can run for a long time
(pointing at org-agenda). 

> I think that's a simple result of how clear-image-cache is
> implemented, I suggest to read the code.

I tried to look at the code, but I have to admit that I don't understand
the reason. Probably because of my poor C. 

In any case, I tried different approach trying to reduce memory
fragmentation. Instead of using the default malloc/free, I used jemalloc
through LD_PRELOAD. For periodic cache clearing (every 2,10,50, and 100
opened/killed images), there was no noticeable difference in the memory
consumption. However, when I tried to open a bunch of images all
together, killed them, and cleared the cache (as in my initial bug
report), the jemalloc instance of emacs showed much lower memory
consumption (~240Mb vs 1200Mb). 

The 240Mb is still higher in comparison with the memory consumption at
init, but a lot more manageable. So, I will try running emacs with
jemalloc for my main emacs instance and see if I am still getting the
high memory consumption in my workflow. 
Not sure if it is feasible to consider changing the memory allocator
upstream. Though large heap size might slow down garbage-collect (as
from discussion in
https://lists.gnu.org/archive/html/emacs-devel/2019-11/msg01137.html)
and any means of reducing the heap may also help responsiveness of
emacs. 

The relevant memory graphs are attached (note that I had a lot of noise in
the previous graphs. It was because of two instances of emacs running at
the same time).

Regards,
Ihor

images-new.png
Description: PNG image

images-jemalloc.png
Description: PNG image

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: juri@linkov.net, 38345@debbugs.gnu.org
>> Date: Fri, 29 Nov 2019 01:27:39 +0800
>> 
>> The behaviour you describe does not follow the docstring:
>> > "*Maximum time* after which images are removed from the cache.
>> > ...
>> > If the cache contains
>> > a large number of images, the actual eviction time may be shorter.
>> > ...
>
> The doc string assumes "normal" operation, it doesn't assume some Lisp
> command runs for a long time, thus preventing redisplay.
>
>> Also, I repeated my tests manually calling image-clear-cache every Nth
>> image. Every invocation of image-clear-cache does reduce the memory
>> consumption, but there is always some residual remaining unrealeased
>> (see the attached images). The residual seems to scale with the number
>> of images in the cache before clearing.
>
> I think that's a simple result of how clear-image-cache is
> implemented, I suggest to read the code.