Re: datamash performance question

[Jake VanEck]

I've tried similar commands but doesn't awk need to put the entire dataset into memory for this? If so, the data is far larger than memory can handle but I'll test it out anyway

Thanks for the suggestion

-Jake

On Fri, Jun 25, 2021, 4:38 PM Dima Kogan <dima@secretsauce.net> wrote:

I'd be curious to see ways to make datamash work faster here, but if all
you're doing is computing sums, you can use awk instead of datamash:

  mawk '{s[$2] += $1;} END { for (k in s) {print k, s[k]; } }'

Then you can probably also get rid of your sort and sed calls.