bug#7868: splitting up test suites

[Ralf Wildenhues]

Hi Stefano,

* Stefano Lattarini wrote on Thu, Jan 20, 2011 at 10:19:24PM CET:
> On Thursday 20 January 2011, Ralf Wildenhues wrote:
> > * Stefano Lattarini wrote on Thu, Jan 20, 2011 at 12:38:17PM CET:
> > > Hmmm... while this feature might be worth having even indipendently
> > > from the issue at hand (but see below for small nits), I still think
> > > that in the long run it would be nicer to transparently work around
> > > such command-line length issues in the test driver, if possible.
> > 
> > No, that is not possible with portable make.
> >
> Yes :-(  I've found that out while trying to write a a proper test
> 'parallel-tests-long-cmdline.test' (attached, just for reference).
[...]

> But this makes me think.  If I substitute the `$MAKE check' call
> in my test with a `$MAKE dist' call, I get:
> 
>   + make dist
>   { test ! -d "parallel-tests-long-cmdline-1.0" || \
>     { find "parallel-tests-long-cmdline-1.0" -type d ! -perm -200 -exec chmod 
> u+w {} ';' \
>       && rm -fr "parallel-tests-long-cmdline-1.0"; }; }
>   test -d "parallel-tests-long-cmdline-1.0" || mkdir 
> "parallel-tests-long-cmdline-1.0"
>   make: execvp: /bin/sh: Argument list too long
>   make: *** [distdir] Error 127
> 
> which means that, with the current implementation of the `dist' target,
> even breaking the testsuite in two or more testsuites *in the same
> directory* won't help at "make dist" time.  Or am I missing something?

No, you are correct.  This was similarly reported even for GNU/Linux by
Xan Lopez on the automake list over a year ago (and probably before that
too).

I'm not too worried about dist however, for a couple of reasons:

dist is not usually required for downloading, building, testing, and
installing software.  As such, even if it is GCS-mandated, it is not
quite as big a problem when it doesn't work everywhere; but of course
it needs to work on developer machines.  Furthermore, it is almost
always possible for the developer to restructure the package so that
line lengths are not exceeded, usually by introducing (more) Makefile
recursion.  See the `Length Limitations' node in the manual.


That said, I would like to address distdir in the long run as well.
I see two possible strategies:

As far as possible, automake can try to split the copying part of
distdir into several targets that copy parts of the files.  This isn't
as easy as it sounds, because usually it's one of the variables SOURCES,
EXTRA_DIST, etc. alone that exceeds the limit.  But the splitting can be
done at automake time.  (Note that if @substitutions@ are involved, then
a single expansion can currently not exceed the line length limit by
itself, that would already break at configure or config.status time.)

Second, and much simpler: when we have gnu-make support, we can simply
use GNU make-specific code to avoid running into limits.  This can be
done transparently, and since in practice virtually all package
maintainers at least have access to GNU make, it should be sufficient
in practice.

I have a half-done patch for the second scheme.  I hope to post all the
gnu-make stuff soonish, sorry for the delay.

> But maybe allowing the user to say, e.g.:
> 
>  TESTS = test1 test2 test3
>  EXTRA_TESTS_LIST = @file-with-list-of-test
> 
> might still be worth after all; if he prefers to lose some make semantics
> in order not to have to split the testsuite, then he should be allowed to
> do so.  It remains to see if that's doable (preferably in an easy way).

Yes, it may be worth it.  But if we can avoid it, all the better.
I prefer not breaking our usual semantic scheme.

> > If we can handle the latter via automake's register_language machinery,
> > then putting tests in *_SOURCES variables would be fairly natural.
> > 
> > OTOH, you are right in that elsewhere, _SOURCES are usually meant to be
> > final, nonderived files,
> >
> That's not exactly true, as automke allows (as it IMHO should)
> generated files to be places in *_SOURCES.  But automake also needs
> to know the contents of a foo_SOURCES variable statically [1] at
> Makefile.in's generation time, while it wants to allow the user to
> specify *at make runtime* the list of TESTS to run.

Yes, that is one of the other complications/semantic differences:
users usually cope with not being able to override *_OBJECTS.
We considered declaring TESTS an internal detail too, but I just
find it too helpful to give up this API.

>  [1] OK, it's smart enough to resolve variable indirections and
>      conditionals, but won't allow @var@ substitution to be placed
>      in a *_SOURCES variable (by design).

FWIW, I regard that a necessary limitation, not a design feature.

> > >  # These are all specified by the user:
> > >  TEST_SUITES = suite1 suite2 suite3 ..
[...]
> > >  # These are then produced by automake:
> > >  TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
> > >  TEST_SUITE_HTMLS = $(TEST_SUITES:=.html)
> > >  suite1_LOGS = $(am__helper_var:.test=.log)
> > 
> > Unfortunately, this is error-prone, because some GNU make versions
> > expand
> >   empty =
> >   TEST_SUITES = foo $(empty)
> >   TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
> > 
> > into `foo.log .log' rather than `foo.log '.  This happens in practice
> > if `empty' is set only under some Automake conditional, as in
> >   if COND
> >   TESTS += bar
> >   endif
> >
> And I guess that specifying dummy `.log:' (and maybe `.html:' etc.)
> rules won't work either, right? :-(

Nope.

> > That is the reason the check-TESTS rule is so ugly (and recursive) in
> > the first place.  I really would like to avoid more instances of this
> > wart; so specifying files without extension is Not Good(TM).
> >
> OK, noted (and these considerations could IMHO end up somewhere in the
> manual).

Good suggestion.  They should be mentioned in the portability section of
autoconf.texi if they aren't already, and as ## comments in check.am.
Let me see about a quick patch.

> > Separate check/recheck targets are of course nice, but IMVHO optional
> > for the first iteration.
> >
> I agree; the only important thing is to devise a design that will allow
> them to be easily added in later refinings.

Don't worry.

Cheers,
Ralf