Re: findutils-4.1.20: a comment on xargs.c arg_max

[James Youngman]

On Thu, Dec 09, 2004 at 08:00:44AM -0700, Nelson H. F. Beebe wrote:
> The code for findutils-4.1.20/xargs/xargs.c contains this fragment:
> 
>   /* Sanity check for systems with huge ARG_MAX defines (e.g., Suns which
>      have it at 1 meg).  Things will work fine with a large ARG_MAX but it
>      will probably hurt the system more than it needs to; an array of this
>      size is allocated.  */
>   if (arg_max > 20 * 1024)
>     arg_max = 20 * 1024;
> 
> In earlier releases of GNU findutils, I had made a local modification
> to comment out that statement.

This is also much improves improved in findutils-4.2.x.  IIRC, the
figure is raised to 128Mb.

> We no longer live in a PDP-11 world, and modern systems often have
> many gigabytes of main memory.  It seems utterly draconian to limit
> arg_max to 20KB, and I'm very sceptical that larger values will "hurt
> the system".  

Very large values will (or may) cause swapping.  The value only needs
to be large enough that the extra cost of the fork() and exec(), when
amortised over the number of command-line arguments that have been
used, is "small".  There is not a great deal of point in increasing
arg_max beyond that "sweet spot".

> Computers are supposed to work for people, not the other way around.
> When the environment size is restricted, users suffer from nonsense
> like this (from an SGI IRIX 6.5 system):
[...]
>       # What is the POSIX minimum?
>       % getconf _POSIX_ARG_MAX
>       4096
[...]
>       % find /usr/include -type f | ./xargs cat | wc -l
>       DEBUG: xargs: ARG_MAX           =               5120
>       DEBUG: xargs: LONG_MAX          =         2147483647
>       DEBUG: xargs: orig_arg_max      =               3072
>       DEBUG: xargs: env_size(environ) =               5120
>       DEBUG: xargs: capped arg_max    =               3072
>       DEBUG: xargs: reduced arg_max   =              -2724
>       DEBUG: xargs: expanded arg_max  =            1048576
>       680880
> 
> The origin of the "environment is too large for exec" diagnostic and
> immediate exit is now clear: the reduced arg_max is negative.
[...]
> Guaranteeing a minimum of 1MB worked around the problem, and xargs ran
> correctly, compared to what SGI's version does:

I believe that that is a bug in IRIX's "xargs" or execve() system
call, because if the argument length exceeds ARG_MAX, execve() should
fail with E2BIG:-

|| ERRORS
|| The exec functions shall fail if:
|| 
|| [E2BIG] 
|| The number of bytes used by the new process image's argument list and
|| environment list is greater than the system-imposed limit of {ARG_MAX}
|| bytes.


> I have a large collection of architectures to test code on, including
> all of the major Unix flavors on all of the major CPU types, and will
> be happy to assist in any testing that such changes might entail.

That's a very generous offer.  At the very least, it will be useful to
make sure that findutils compiles on platforms.  The ones I have had
most problems with recently have been Solaris and UNICOS.  It's a
shame that the findutils test suite is not very extensive, or this
offer would be much more useful.


> Thanks to the simh and Hercules simulator projects, I also now have
> several historical Unix releases on simulated historical architectures
> (PDP-11, Interdata-32, VAX, and soon, IBM S/360).  On the VAX at
> least, I have gcc-2.95, so it should be possible to build most modern
> packages on it.

I wish I had a Cray emulator - it would make it much easier to check
things work on UNICOS :)