Re: Documentation for the parallel-tests driver. [4/4]

[Akim Demaille]

>>> "RW" == Ralf Wildenhues <address@hidden> writes:

 > address@hidden Simple tests using @samp{parallel-tests}
 > address@hidden @option{parallel-tests}, Using
 > +The option @option{parallel-tests} (@pxref{Options}) enables a test
 > +suite driver that is mostly compatible to the simple test driver
 > +described above, but provides a few more features and slightly different
 > +semantics.  It features concurrent execution of tests with @code{make -j},
 > +allows to specify inter-test dependencies, lazy reruns of tests that
 > +have not completed in a prior run,

That's one use, but more importantly, this is perfect for unit-tests.
Unit-tests are often self-contained, in which case if none of their
dependencies changed, there is no point in running the test again, the
results will be the same.

 > +summary and verbose output in @samp{RST} (reStructuredText) and
 > address@hidden format, and hard errors for early abort of test runs.
 > +Similar to the simple test driver, @code{TESTS_ENVIRONMENT},
 > address@hidden, @code{XFAIL_TESTS}, and the @code{check_*}
 > +variables are honored, and the environment variable @env{srcdir}
 > +is set during test execution.

 > address@hidden VERBOSE
 > +As with the simple driver above, by default one status line is printed
 > +per completed test, and a short summary after the suite has completed.
 > +If the variable @samp{VERBOSE} is set, the @file{test-suite.log} file is
 > +appended after the summary.

IMHO, you should emphasize that tests should be verbose by default.
VERBOSE is no longer an envvar that test should read to decide whether
displaying something or not: they should be verbose by default, it
goes into the log file.  Now VERBOSE is meant for "make check" itself.




I think that you should detail how to write a .test->.log rule,
showing the default value for instance (I use GNU Make syntax).

  # From a test file to a log file.
  # Do not use a regular `.test.log:' rule here, since in that case the
  # following rule (without incoming extension) will mask this one.
  %.log: %.test $(check_programs)
        @$(am__check_pre) $${dir}$< $(am__check_post)

because that's where the user can explain how the test is run.
Something I dislike very much in the current framework for tests in
Automake is that each test must be an executable, so it is very
inconvenient to test a given program foo with many different inputs.
In that case I use

   TESTS = 1.foo 2.foo 3.foo ...

and

  %.log: %.test $(check_programs)
        @$(am__check_pre) $(top_builddir)/src/foo $${dir}$< $(am__check_post)

i.e., the testee is foo, and TESTS are plain files, not a bazillion of
1.sh that simple "exec $(top_builddir)/src/foo 1.foo".