Re: parallel sort at fault? [Re: [PATCH] tests: avoid gross inefficiency...

[Jim Meyering]

Pádraig Brady wrote:
...
> I still get bad performance for the above with SUBTHREAD_LINES_HEURISTIC=128K

Sorry I haven't had time for this today.
I'll investigate tomorrow.

> So as you suggested, the large mem allocation when reading from a pipe
> is a problem,
> and in fact seems to be the main problem.  Now given the memory isn't
> actually used
> it shouldn't be a such an issue, but if one has MALLOC_PERTURB_ set,
> then it is used,
> and it has a huge impact. Compare:
>
> $ for i in $(seq 33); do seq 88| MALLOC_PERTURB_= timeout 2 sort
> --para=1 >/dev/null & done
> $ for i in $(seq 33); do seq 88| MALLOC_PERTURB_=1 timeout 2 sort
> --para=1 >/dev/null & done

Good point!

> So we should be more conservative in memory allocation in sort,
> and be more aligned with CPU cache sizes than RAM sizes I suspect.
> This will be an increasing problem as we tend to run more in ||.
> It would be interesting I think to sort first by L1 cache size,
> then by L2, etc, but as a first pass, a more sensible default
> of 8MB or so seems appropriate.
>
> As a general note, MALLOC_PERTURB_ should be unset when benchmarking
> anything to do with `sort`