Re: flist -- "Killed" -- oom (*not* 1.8 related)

[Ken Hornstein]

>ah, great! yes, that works. and, yes, to my ignorant eye, it appears
>that the call from `folder_read()` to `mh_xmalloc()` is where we are
>going south.
>[...]
>#2  0x000055555555f898 in mh_xmalloc (size=42269452928) at sbr/utils.c:38
>#3  0x000055555555acf6 in folder_read (name=0x5555555d5400 
>"/home/minshall/Mail/mhe-index", lockflag=0) at sbr/folder_read.c:138

Exactly HOW many messages are in mhe-index?

Ah, I think I see what's happening.  That line is this:

        mp->msgstats = mh_xmalloc (MSGSTATSIZE(mp));

MSGSTATSIZE is defined as:

#define MSGSTATSIZE(mp) ((mp)->num_msgstats * sizeof *(mp)->msgstats)

num_msgstats is set by the previous line:

    mp->num_msgstats = MSGSTATNUM (mp->lowoff, mp->hghoff);

Which is defined as:

#define MSGSTATNUM(lo, hi) ((size_t) ((hi) - (lo) + 1))

So ... the summary here is that nmh (and MH before it) allocates a
"message status" element for every possible message.  The possible
number of messages is the range between the LOWEST message number and
the HIGHEST message number.  So if you just had 1000000 and 10000002 in
a folder, it would allocate 3 elements.  But if you had 1 and 1000000,
it would allocate a million elements.  A msgstat structure is an array
of "struct bvector" which might be ... 8 + 8 + 16 bytes per message on
a 64 bit platform.  That suggests there are either 1320920404 messages
in that folder (1.2 billion) or there's a huge message number gap (that
has come up before when someone had a huge gap; the my memory is the
consensus was you just had to deal).

In general nmh will try to handle messages and folders up to the virtual
memory limit and it seems like you reached it.

--Ken