Re: [bug-gawk] Memory leak

[Andrew J. Schorr]

On Tue, Mar 28, 2017 at 08:00:38AM +0000, Stephane Delsert wrote:
> To see the problem you have to duplicate the sample file to reach at least a 
> couple of MM of records .
...
> The command : 
> zcat sdelse1.pip.gz.5t | tail -n +2 | gawk -F '|' -f test.awk | gzip > 
> out.sdelse1.pip.gz.5t

Unless I missed it, I don't think you mentioned the "tail -n +2" aspect
until now. In other words, one needs to skip the first record of the input
file that you sent. That makes a big difference. One now gets some output.

However, repeating the contents of the sample data file you sent doesn't seem
to help, I guess because the duplicate records are filtered out.

> The progression of the memory leak  seems linear

Linear memory overhead is normal when storing data in a huge array.

> and be linked to an allocation memory during the call of the sort subscript.

How can you be sure that it's related to the sort function? When you look
at the memory usage, how many bytes per record are being used? How does
that compare to the size fo the actual record that you are storing in the
array?

Regards,
Andy