Re: [Readline] controlling out-of-memory behavior

[Chet Ramey]

On 11/21/22 2:43 AM, John Scott wrote:
> Hi,
>
> Apologies, I couldn't find a more appropriate mailing list for Readline
> help.
>
> I think Readline is an excellent library, but I would like to have
> control over the behavior when it fails to allocate memory for the line.
> If I recall correctly, right now it aborts the process.

Readline, like a lot of GNU software, manages memory using the
xmalloc/xrealloc/xfree set of functions. There are a very few exceptions,
but in the main, readline uses xmalloc/xrealloc and throws a fatal error
when it fails to allocate memory. Those exceptions are places readline
handles a NULL return itself.

If you want, your application can provide its own definitions of these
functions -- bash does -- and implement whatever failure mode behavior you
like. Bash uses it to provide a little bit more information about where the
failed allocation came from, the requested size, and how much memory the
application has allocated so far.

If you do this, make sure never to return a NULL pointer from xmalloc or
xrealloc; your program will crash. You can use a minimal implementation of
xfree, or leave it out altogether; readline implements it in a separate
source file. (You're supposed to be able to use xfree and free
interchangeably anyway, as readline does. If your xfree does anything other
than that, or if your xmalloc returns memory that can't be freed with free,
your program will crash.)

If you want to allocate memory using mmap/mremap/mfree, it's better to
replace malloc. The bash malloc has a hybrid implementation that uses
sbrk for smaller allocations where it populates lists of free blocks and
mmap for larger allocations that it doesn't want to retain after free().

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/