Re: execvp: /bin/sh: Argument list too long

[Pippijn van Steenhoven]

On Mon, Nov 08, 2010 at 08:03:06PM +0100, Ralf Wildenhues wrote:
> * Pippijn van Steenhoven wrote on Mon, Nov 08, 2010 at 04:55:45PM CET:
> > I looked through the mailing list archives and found several questions
> > and proposed solutions related to this problem:
> > 
> >   - Split up file lists so several rules are generated:
> >   http://lists.gnu.org/archive/html/automake/2004-11/msg00106.html
> > 
> >   - "What is your env's size?"
> >   http://lists.gnu.org/archive/html/automake/2009-08/msg00027.html
> > 
> >   - It will fail. In other words "Sorry, can't fix"?
> >   http://lists.gnu.org/archive/html/automake/2007-02/msg00013.html
> 
> Also, we wrote a short section in the manual about this (in recent
> versions):
>   info Automake "Length Limitations"

I hadn't seen that, before. Now I read it, I noticed something:

  Automake itself employs a couple of strategies to avoid long command
  lines. For example, when ‘${srcdir}/’ is prepended to file names, as
  can happen with above $(data_DATA) lists, it limits the amount of
  arguments passed to external commands.

Why doesn't it do that, always? Is the performance hit too large?

> > My problem is that I can not make dist in one project and not make check
> > in another project. Both have a large number of tests. The one where make
> > check fails has a few more tests, but make dist will fail there, as well.
> 
> One workaround is to put tests in more than one directory, with more
> than one Makefile.am.  Yes, it is ugly, I know.  Another is to require
> non-VPATH builds on those systems with the most severe restrictions,
> such as MSYS and IRIX (if you can't convince root to increase the
> limit).

I am root on my (Linux) system and I set the stack size to unlimited. The
libtool macro reported a few billion (or something other really large)
for maximum argument list length, bash also agreed (it easily executed
the "distdir" target when copied into a bash script), but make doesn't.
Both gnu make and pmake abort with the "too long" message.

> > What are the plans on fixing this?
> 
> We'll fix what we can fix.  Without GNU make-specifics, that is hard.
> One thing I'd definitely like to see done is allowing multiple
> parallel-tests test suites in one Makefile.am, so you can still have a
> nonrecursive setup.  That may mean that 'make dist' still fails, but
> my hope is that is not needed to work on every platform.
> 
> Would that be sufficient for your needs?

I'd be happy if make dist worked on my system, but it doesn't and I
didn't find a way to make it work, yet (any suggestions here?).

> > but since by that time, Makefile.in is
> > already generated, it can't help. Maybe storing the list of dist files
> > and check files in a different file and reading this with xargs < $file
> > would help.
> 
> One problem is that with many constructs, automake would like to know
> the exact set of files it deals with.  We can try to deviate from that
> in some places but my guess is that will always look like a hack that
> avoids some but not all issues.

Automake can know the maximum set of files it deals with. Wouldn't that
help, already?

> One other strategy we maybe could implement is alternative GNU
> make-specific rules that avoid limitations by dealing with files with
> make functions.  That is usually doable, but doing this consistently
> will be quite some work and cause quite some duplication in the
> Makefile.in.  I don't have current plans to pursue this;

I would definitely prefer no GNU make stuff in the generated makefiles.

> you can see
> e.g., the write_entries_to_file macro in the gcc/Makefile.in file in the
> GCC tree for how this can be done in principle.

I'll take a look at it.

-- 
Pippijn van Steenhoven

signature.asc
Description: Digital signature