Re: parallel testsuite execution

[Ralf Wildenhues]

Let's get back to this issue.  I seem to constantly be about one year
behind everything.

* Akim Demaille wrote on Fri, Oct 26, 2007 at 01:53:20PM CEST:
> >>> "RW" == Ralf Wildenhues <address@hidden> writes:
> > * Akim wrote: 
>  >> So I would like to be able to write something like
>  >> 
>  >> TEST_SUITES = foo bar
>  >> foo_TESTS = foo1.chk foo2.test foo3
>  >> bar_TESTS = bar1.test bar2.sh bar3.c
>  >> 
>  >> and be able to write on the side rules explaining how to .chk -> .log,
>  >> .test -> .log etc.

Do you want to write those rules yourself?  If yes, why?
Wouldn't it suffice if you specified the tests, then they
would be run with 'make check'?  And regarding suffixes,
would it be sufficient to just specify a number of literal ones?
(see below)

Also, do you need to be able to specify the *sources* of
compiled tests?  Is it not sufficient to use check_PROGRAMS there?

>  >> Maybe instead of _TESTS we could reuse _SOURCES?  That's not
>  >> inconsistent: they are really sources which must be compiled into
>  >> *.log files which are then linked together to produce foo.log and
>  >> bar.log.

The semantics are at least a bit different.  With _SOURCES, you have
several pre-defined suffixes.  Here, it seems you want to be able to
specify freely.

With programs, the set of inputs doesn't change at 'make' time.
With tests, I feel running just a few:
  make check TESTS='just a couple of them'

is quite useful to have, even if you need to write it like this:
  make foo.log foo_TESTS='foo2.test'

Lazy completion, hard errors, are good, $(EXEEXT) should be taken care
of, parallel is good, should all be in portable make.

We should allow for interdependent tests, that would be quite helpful.

What I am currently thinking is something like this:

  TEST_SUITES = foo.log bar.log
  foo_log_TESTS = foo1.chk foo2.test foo3
  foo_log_SUFFIXES = $(EXEEXT) .chk .test
  bar_log_TESTS = bar1.test bar2.sh bar3
  bar_log_SUFFIXES = .test .sh
  check_PROGRAMS = foo3 bar3
  bar3_SOURCES = bar3.c`
  bar3.log: bar2.log

To be able to override TESTS (resp. foo_log_TESTS), it is necessary to
have the individual logs depend dynamically on $(TESTS).  IOW, something
like this would be generated by automake:

  am__foo_log_LOGS1 = $(TESTS:=.log)
  am__foo_log_LOGS2 = $(am__foo_log_LOGS1:@address@hidden)
  am__foo_log_LOGS3 = $(am__foo_log_LOGS2:.chk.log=.log)
  foo_log_LOGS = $(am__foo_log_LOGS3:.test.log=.log)

(@EXEEXT@ instead of $(EXEEXT) because putting macros there is
unspecified by POSIX; it doesn't work with IRIX make).

This however requires that test logs end up in subdirs (think
TESTS=sub/foo), thus depend on suitable dirstamps.  As if
subdir-objects were the default for TESTS.

Are we on the same page here?

Thanks,
Ralf