Re: [bug-gawk] [BugReport] GAWK 4.1.1 memory leak

[Anurag Dubey]

hey,

Found the cause of the memory usage. Another script was running this
awk and piping the outputs from around 570 files into this awk program
using xargs. Thus, both arrays ended up storing huge number of keys
before the END section. This resulted in the memory usage exceeding
10Gigs.

Regards
Anurag Dubey

On Mon, Dec 22, 2014 at 4:43 AM, Anurag Dubey <address@hidden> wrote:
> Hey,
>
> While trying to pipe a file into awk and count occurrences of a
> particular pattern, my awk process starts consuming more than 8-10GB
> of memory, even though the input file is less than 50MB text. Here is
> the code:
>
> sudo zcat $fullpath | sudo awk '
> BEGIN { FS="@#@" }
>
>         NF != 24 { err = sprintf("%s:%d: skipped: NF != 24\n",
> "'"$filename"'", FNR);
>         print err > "/var/log/error.log" ; next }
>                     {
>                         split($6,a," ");
>                         date=a[1];
>                         hour=substr(a[2],1,2);
>           
> cnt[date"@#@"hour"@#@"$17"@#@"$16"@#@"$22"@#@"$18"@#@"$5"@#@"$23"@#@"$24"@#@"$19"@#@"$4]++;
>
> sum[date"@#@"hour"@#@"$17"@#@"$16"@#@"$22"@#@"$18"@#@"$5"@#@"$23"@#@"$24"@#@"$19"@#@"$4]
> += $14
>                 }
>
>                 END
>                 {
>                 for ( key in cnt  )
>                 { print key"~`~"cnt[key]"~`~"sum[key];}
>                 }
> '
>                 > $dirname"rts/"$filename.temp
>
> Here, cnt is the counter and sum is for summing particular column
> values corresponding to the pattern. On running the script, the memory
> usage starts increasing initially from a few hundred MBs, later to
> even more than 10GB.
> OS is centos5 and version are GNU Awk 4.1.1, API: 1.1.
> All the awk versions below this also face the same problem.
>
> Regards
> Anurag Dubey