Re: sort with a large buffer size (-S)

[Bob Proulx]

Ryan Morin wrote:
> In other words, I would like to be able to set -S to allow sort to
> consume much more memory to save the necessity of writing large temp
> files to disk (unless necessary). >b

Seems reasonable to me.

> I presume there is a way to compile sort to allow it to utilize more
> than 4G of memory on an x86_64 machine but have not been able to
> determine how.

If you are compiling your own code are you compiling a 32-bit version
or a 64-bit version?  A 64-bit version should be able to use all
available memory but of course a 32-bit machine is limited to 2^32 or
4G maximum since the pointer size is limited to that there.

> I work on x86_64 Macs and also x86_64 machines running RHEL.

I am not familiar with Macs at all.  I think I recall that RHEL x86_64
ships with a 32-bit version of sort.  Could you please run the
following and tell us the bit size of the architecture?  (If it is in
/bin/sort instead then please do the right thing.)

  file /usr/bin/sort

I suspect it will say 32-bit and that is why you are limited.  But you
are on a 64-bit machine and so could run a 64-bit version of sort.  If
you had it available or compiled it should be able to use all
available memory.

Also, as long as we are here, please tell us the version of sort too.
(Useful in general but probably doesn't matter at this point.)

  sort --version

Bob