Re: [bug-gawk] Memory leak

[Andrew J. Schorr]

Hi Stéphane,

I'm surprised, since that's the only leak I can see in the valgrind output.
If the debug messages about the NODE and BUCKET counts still show very small
numbers, then I just don't see what else could be leaking. Do you have
any valgrind output that shows this leak?

There are basically 2 possibilities for why the memory footprint would grow:

1. If you're building a huge array in memory, then that consumes lots
of NODEs and BUCKETs, and it takes a lot of space. But you say that you're
not storing a large array, and the debug messages show that very few NODEs
and BUCKETs are being allocated, unless your 60MM record dataset is much 
different than your smaller 1MM and 2MM datasets. Is that one building a huge
in-memory array?

2. If you're not building a large array in memory, then there would have to
be a leak of the type I fixed in that patch. I don't see any other leaks
in the valgrind output you sent.

Regards,
Andy

P.S. I am attaching a more elegant version of the patch, but it should give the
exact same results. Arnold -- what do you think of this issue? It seems like a
memory leak to me.

On Thu, Mar 30, 2017 at 04:26:20PM +0000, Stephane Delsert wrote:
> Thanks Andrew,
> 
> 
> 
> I did the modification but  that seems that doesn't work :  I reach 1gb of 
> memory  for almost 60 MM of records and that still growth
> 
> 
> 
> Best regards,
> 
> 
> 
> Stéphane.
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Andrew J. Schorr [mailto:address@hidden
> Sent: jeudi 30 mars 2017 17:20
> To: Stephane Delsert <address@hidden>
> Cc: address@hidden; address@hidden
> Subject: Re: [bug-gawk] Memory leak
> 
> 
> 
> Hi,
> 
> 
> 
> > ==19925== 11,301,792 bytes in 117,727 blocks are still reachable in loss 
> > record 124 of 124
> 
> > ==19925==    at 0x4A0717A: malloc (vg_replace_malloc.c:298)
> 
> > ==19925==    by 0x480FDE: emalloc_real (awk.h:1820)
> 
> > ==19925==    by 0x482599: bcalloc (symbol.c:722)
> 
> > ==19925==    by 0x40BBDA: assoc_list (array.c:1329)
> 
> > ==19925==    by 0x4477C9: r_interpret (interpret.h:900)
> 
> > ==19925==    by 0x4605F8: main (main.c:518)
> 
> 
> 
> This is what worries me. It does seem to correspond to a leak related to 
> using a user-defined sorting function. Looking at the code, I don't see why 
> it's not getting freed, unless this test is bailing out early before we free 
> the allocated instruction below:
> 
> 
> 
>         if (list == NULL || ! cmp_func || (assoc_kind & (AASC|ADESC)) != 0)
> 
>                                 return list;    /* empty list or unsorted, or 
> list already sorted */
> 
> 
> 
> We know that cmp_func is not NULL, but perhaps list is NULL or the assoc_kind 
> flags were modified by symbol->alist. If this test fails, then we return 
> before we pop the instruction and free it, so it's still on the stack and 
> reachable...
> 
> 
> 
> Does the attached, not well-tested, patch fix your memory leak?
> 
> 
> 
> Regards,
> 
> Andy
> ***************************************************************************
> The information contained in this communication is confidential, is
> intended only for the use of the recipient named above, and may be legally
> privileged.
> 
> If the reader of this message is not the intended recipient, you are
> hereby notified that any dissemination, distribution or copying of this
> communication is strictly prohibited.
> 
> If you have received this communication in error, please resend this
> communication to the sender and delete the original message or any copy
> of it from your computer system.
> 
> Thank You.
> ****************************************************************************

-- 
Andrew Schorr                      e-mail: address@hidden
Telemetry Investments, L.L.C.      phone:  917-305-1748
545 Fifth Ave, Suite 1108          fax:    212-425-5550
New York, NY 10017-3630

assoc_list2.patch
Description: Text document