Re: Emacs bzr memory footprint

[Nix]

On 21 Oct 2011, Stefan Monnier uttered the following:

> I've just installed memory-usage.el as a package in the `elpa' branch
> (you'll soon be able to install it using package.el and in the mean time
> you can download it from 
> http://bzr.savannah.gnu.org/lh/emacs/elpa/download/head:/memoryusage.el-20111021130523-o1v7kfuzrat3pcd9-2/memory-usage.el).
>
> It basically looks at the garbage-collect output and the various
> buffer's size to provide you with a human-readable description of the
> memory in use, from Elisp's point of view.  Don't expect too much of it,
> but it can be helpful.
>
> Could you try it out?

Two rounds of stats. Firstly, absolute peak legitimate memory usage,
when reading a truly massive nonexpired newsgroup:

Garbage collection stats:
((1682002 . 1142162) (78246 . 155) (25344 . 17561) 88139150 2009895 (2439 . 
9353) (32882 . 12331) (452963 . 384130))

 =>     26912032+18274592 bytes in cons cells
        3755808+7440 bytes in symbols
        1013760+702440 bytes in markers
        39024+149648 bytes in floats
        1841392+690536 bytes in intervals
        14494816+12292160 bytes in string headers
        88139150 bytes of string chars
        88139150 bytes of vector slots

Total bytes in lisp objects: 170322693 (live 138205877, dead 32116816)

Buffer ralloc memory usage:
108 buffers
7086099 bytes total (60682 in gaps)
[...]


In normal usage, with Huge Newsgroup closed but with my usual pair of
nnml and gnutls-backed nnimap groups open (after another GC to push
memory usage down properly):

Garbage collection stats:
((1678832 . 1131753) (77833 . 463) (24737 . 17683) 88021904 2005723 (2428 . 
9364) (32180 . 14186) (449983 . 387110))

 =>     26861312+18108048 bytes in cons cells
        3735984+22224 bytes in symbols
        989480+707320 bytes in markers
        38848+149824 bytes in floats
        1802080+794416 bytes in intervals
        14399456+12387520 bytes in string headers
        88021904 bytes of string chars
        88021904 bytes of vector slots

Total bytes in lisp objects: 170024139 (live 137854787, dead 32169352)

Buffer ralloc memory usage:
104 buffers
6977389 bytes total (55669 in gaps)
      Size      Gap     Name

STIME     TIME   RSS    VSZ
Oct21 00:55:23 524568 716588

So I have 170Mb in Lisp objects, 7Mb in buffers... and a 716Mb VSZ.
500Mb of that is heap, too:

% pmap 2676 | grep anon
0000000002fc2000 499180K rw---    [ anon ]
00007fcd0017b000      4K rw---    [ anon ]
00007fcd05fbf000      4K rw---    [ anon ]
00007fcd08d72000    304K rw---    [ anon ]
00007fcd08e68000    348K rw---    [ anon ]
00007fcd094d1000   1300K rw---    [ anon ]
00007fcd097c6000      4K rw---    [ anon ]
00007fcd0ab36000     96K rw---    [ anon ]
00007fcd0ab9c000    436K rw---    [ anon ]
00007fcd0ac12000    424K rw---    [ anon ]
00007fcd0ac81000    192K rw---    [ anon ]
00007fcd0adea000    152K rw---    [ anon ]
00007fcd0ae23000    256K rw---    [ anon ]
00007fcd0aef9000     84K rw---    [ anon ]
00007fcd0dd80000      4K rw---    [ anon ]
00007fcd0e975000      4K rw---    [ anon ]
00007fcd0f1d7000      8K rw---    [ anon ]
00007fcd0f8db000     12K rw---    [ anon ]
00007fcd10ef8000     20K rw---    [ anon ]
00007fcd12041000      4K rw---    [ anon ]
00007fcd1225a000     16K rw---    [ anon ]
00007fcd12919000    100K rw---    [ anon ]
00007fcd12f1d000      4K rw---    [ anon ]
00007fcd1315e000      4K rw---    [ anon ]
00007fcd13672000      8K rw---    [ anon ]
00007fcd14df3000     64K rw---    [ anon ]
00007fcd15280000     16K rw---    [ anon ]
00007fcd156f1000      4K rw---    [ anon ]
00007fcd15b68000      8K rw---    [ anon ]
00007fcd15d38000    112K rw---    [ anon ]
00007fcd15d85000      8K rw---    [ anon ]
00007fcd15d89000      4K rw---    [ anon ]
00007fffca9ff000      4K r-x--    [ anon ]
ffffffffff600000      4K r-x--    [ anon ]

Looky looky, leak.

> If this output does not explain the process's size you're seeing, then
> we have a leak in the C code somewhere.

Oh yes.

> I'd hack src/alloc.c to export the needed info to Elisp.  But maybe
> memory-usage will already give us enough info that this won't
> be necessary.

Here's the last summary line from Dmitry Antipov's patch written for
exactly this purpose, which emits a *lot* of other usage info too
(if you want a 5Mb upload I can give you the whole log, lzmaed, but
look out, uncompressed it comes to 264M):

-- GC507 end: lisp data 179658 Kb (148315 live, 31342 free), 152325 mem
   nodes (7140 Kb), heap 418617 Kb used, 117794 Kb free

That comes from this printf():

+  fprintf (stderr, 
+          "-- GC%d end: lisp data %d Kb (%d live, %d free), "
+#if GC_MARK_STACK || defined GC_MALLOC_CHECK
+          "%d mem nodes (%d Kb), "
+#endif
+          "heap %d Kb used, %d Kb free\n",
+          /* GC count.  */
+          gcs_done,
+          /* Total Lisp data.  */
+          ((total_conses + total_free_conses) * sizeof (struct Lisp_Cons) +
+           (total_markers + total_free_markers) * sizeof (union Lisp_Misc) +
+           (total_symbols + total_free_symbols) * sizeof (struct Lisp_Symbol) +
+           total_string_size +
+           total_vector_size * sizeof (Lisp_Object) +
+           (total_floats + total_free_floats) * sizeof (struct Lisp_Float) +
+           (total_intervals + total_free_intervals) * sizeof (struct interval) 
+
+           (total_strings + total_free_strings) * sizeof (struct Lisp_String)) 
/ 1024,
+          /* Live Lisp data.  */
+          (total_conses * sizeof (struct Lisp_Cons) +
+           total_markers * sizeof (union Lisp_Misc) +
+           total_symbols * sizeof (struct Lisp_Symbol) +
+           total_string_size +
+           total_vector_size * sizeof (Lisp_Object) +
+           total_floats * sizeof (struct Lisp_Float) +
+           total_intervals * sizeof (struct interval) +
+           total_strings * sizeof (struct Lisp_String)) / 1024,
+          /* Free Lisp data.  */
+          (total_free_conses * sizeof (struct Lisp_Cons) +
+           total_free_markers * sizeof (union Lisp_Misc) +
+           total_free_symbols * sizeof (struct Lisp_Symbol) +
+           total_free_floats * sizeof (struct Lisp_Float) +
+           total_free_intervals * sizeof (struct interval) +
+           total_free_strings * sizeof (struct Lisp_String)) / 1024,
+#if GC_MARK_STACK || defined GC_MALLOC_CHECK
+          total_mem_nodes, total_mem_nodes * sizeof (struct mem_node) / 1024,
+#endif
+#ifdef DOUG_LEA_MALLOC
+          mallinfo ().uordblks / 1024, mallinfo ().fordblks / 1024);
+#else
+          0, 0);
+#endif