xargs: exec's with too many arguments

[Jim Meyering]

address@hidden (James Youngman) wrote:
...
> Many systems now include the 'buffer' program (I have it here on an
> Ubuntu box).  If you have it, try piping the data through 'buffer -z
> 4096' (or whatever PIPE_BUF is on your machine if it's not 4096).

Hi James,

Funny seeing a message from you, then minutes later seeing the symptom
of a bug in xargs.  I was running xargs to grep through the files in an
mh-style folder and got an `Argument list too long' error.

Here's a stand-alone example to demonstrate the problem:

  $ seq 100000|/usr/bin/xargs grep foo
  /usr/bin/xargs: grep: Argument list too long
  [Exit 126]
  $ /usr/bin/xargs --version
  GNU xargs version 4.2.25

This is on a 64-bit system running linux-2.4.16.
Here's a tighter example:

  $ yes|head -n 16382|tr '\n' ' ' |xargs echo > /dev/null
  $ yes|head -n 16383|tr '\n' ' ' |xargs echo > /dev/null
  xargs: echo: Argument list too long
  [Exit 126]

With the patch included below, it works:

  $ yes|head -n 16383|tr '\n' ' ' |./xargs echo > /dev/null && echo ok
  ok

You can demonstrate the bug on 32-bit systems too,
but the limits are higher, so for normal usage (with non-trivial
arguments), it's a lot less likely to trigger:

  g$ yes|head -n 32766|tr '\n' ' '|xargs echo > /dev/null && echo ok
  ok
  g$ yes|head -n 32767|tr '\n' ' '|xargs echo
  xargs: echo: Argument list too long
  [Exit 126]

This is because Linux's execve implementation requires that the sum of
the sizes of all argument string pointers not exceed 128K (the actual
limit is ARG_MAX - sizeof (void*)).  The difference between 32- and
64-bit systems is because sizeof (void*) differs (4 vs. 8).

2005-11-18  Jim Meyering  <address@hidden>

        * lib/buildcmd.c (bc_push_arg): When exec'ing, don't exceed
        Linux's limit on the maximum number of command line arguments.

Index: lib/buildcmd.c
===================================================================
RCS file: /cvsroot/findutils/findutils/lib/buildcmd.c,v
retrieving revision 1.9
diff -u -p -r1.9 buildcmd.c
--- lib/buildcmd.c      2 Jul 2005 09:54:34 -0000       1.9
+++ lib/buildcmd.c      18 Nov 2005 13:48:17 -0000
@@ -250,9 +250,10 @@ bc_push_arg (const struct buildcmd_contr
        * conditional on arg!=NULL, since do_exec() 
        * actually calls bc_push_arg(ctl, state, NULL, 0, false).
        */
-      if (!initial_args
-         && ctl->args_per_exec
-         && (state->cmd_argc - ctl->initial_argc) == ctl->args_per_exec)
+      if ((!initial_args
+          && ctl->args_per_exec
+          && (state->cmd_argc - ctl->initial_argc) == ctl->args_per_exec)
+         || state->cmd_argc == ARG_MAX / sizeof (void *) - 1)
        do_exec (ctl, state);
     }