Re: [FYI] New public temporary branch, about documentation for custom test drivers

[Stefano Lattarini]

On Monday 27 June 2011, Stefano Lattarini wrote:
> 
> Attached is the squash-in with the fixes you suggested.  I hope to
> be soon able to finish up this part of the documentation; I'll post
> and additional squash-in when I'm done.
> 
OK the documentation is basically complete now (it still lacks a
working example I'd like to add in the `doc/' subdir, similar to
what is done with the "amhello" package).  I've pushed it to the
temp branch; attached is the diff w.r.t. the previous status, for
reference.

Regards,
  Stefano

diff --git a/ChangeLog b/ChangeLog
index 6acd090..628d879 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,4 +1,4 @@
-2011-06-25  Stefano Lattarini  <address@hidden>
+2011-06-27  Stefano Lattarini  <address@hidden>
 
        docs: document custom test drivers and protocols
        * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
@@ -10,10 +10,12 @@
        (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
        (APIs for Custom Test Drivers): Likewise.
        (Options): Update description of color-tests.
-       * lib/am/check (recheck, recheck-html): Minor adjustments to better
-       conform to the documentation (this should cause no semantic changes
-       w.r.t. the former behaviour); minor improvements and extensions to
-       existing comments.
+       * lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
+       moved into the manual.
+       (recheck, recheck-html): Minor adjustments to better conform to the
+       documentation (this should cause no semantic changes w.r.t. the
+       former behaviour); minor improvements and extensions to existing
+       comments.
        * tests/test-driver-create-log-dir.test: New test.
        * tests/test-driver-strip-vpath.test: Likewise.
        * tests/test-driver-global-log.test: Likewise.
diff --git a/doc/automake.texi b/doc/automake.texi
index 8a6871c..39192ba 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -9119,9 +9119,9 @@ for new features or to satisfy additional portability 
requirements.
 
 The main characteristic of these APIs is that they are designed to share
 as much infrastructure, semantics, and implementation details as possible
-with the plain @option{parallel-tests} harness.  So everything said in
-the previous section should apply here too, unless explicitly stated
-otherwise.
+with the @option{parallel-tests} harness and its default driver.  So
+everything said in the previous section should apply here too, unless
+explicitly stated otherwise.
 
 @menu
 * Command-line arguments for test drivers::
@@ -9135,9 +9135,10 @@ otherwise.
 
 A custom driver can rely on various command-line options and arguments
 being passed to it automatically by the Automake's @option{parallel-tests}
-harness.  It is @emph{mandatory} that it understands all of them (even if
-the exact interpretation of the associated semantics can change between a
-test driver and another, and even be a no-op in some drivers).
+harness.  It is @emph{mandatory} that it understands all of them (even
+if the exact interpretation of the associated semantics can legitimately
+change between a test driver and another, and even be a no-op in some
+drivers).
 
 @noindent
 Here is the list of options:
@@ -9151,10 +9152,10 @@ results (@pxref{Console output during testsuite runs}).
 @item address@hidden
 The log file the test driver must create.  If it has a directory component
 (as in e.g., @file{sub/foo.log}), the Automake harness will ensure that
-such directory exist before the test driver is called.
+such directory exist @emph{before} the test driver is called.
 @item address@hidden|address@hidden
-Whether the console output should be colored or not (@pxref{Simple
-tests and color-tests} to learn when this option gets activated and
+Whether the console output should be colorized or not (@pxref{Simple
+tests and color-tests}, to learn when this option gets activated and
 when it doesn't).
 @item address@hidden|address@hidden
 Whether the tested program is expected to fail.
@@ -9181,23 +9182,14 @@ consistency and a more pleasant user experience.
 @node Log files generation and test results recording
 @subsubsection Log files generation and test results recording
 
-One of the main features of the new testsuite harness is the ability to
-support test protocols that allow a single test script to run more
-test cases, @emph{each with its distinct result}.  In order for the
-testsuite summary report to be correct in this case, the test driver
-implementing support for the protocol must follow the following
-convention.
-
-TODO: In order to have a correct count of test cases in the final
-testsuite summary, the driver must use special syntax in the created
-.log file (see below, `:test-result:').
-
 The test driver must correctly generate the file specified by the
address@hidden option (even when the tested program fails, of
-course!).  This file is quite free-form, but still it has to conform
-to the following conventions:
address@hidden option (of course even when the tested program
+fails or crashes).  This file is quite free-form, but still it has
+to conform to the following conventions, in order to work with the
address@hidden harness and take advantage of its features.
 
 @itemize @bullet
+
 @item
 @cindex Global test script result
 @cindex Test result, global
@@ -9209,9 +9201,10 @@ in the first line of the @file{.log} file.
 If this line does not begin with either @code{PASS:}, @code{XFAIL:} or
 @code{SKIP:}, the test script will be considered failed, and it will be
 re-run by @code{make recheck}.  Also, if this line does not begin with
-either @code{PASS:} or @code{XFAIL:}, the content of and its @file{.log}
-file will be copied into the global @file{test-suite.log} (in order to
-help in debugging and bug-report analysis).
+either @code{PASS:} or @code{XFAIL:}, the content of the @file{.log} file
+will be copied into the global @file{test-suite.log} (in order to help in
+debugging and bug-report analysis).
+
 @item
 @cindex Single test case result
 @cindex Test result, single
@@ -9220,67 +9213,90 @@ help in debugging and bug-report analysis).
 One of the main features of the new testsuite harness is the ability to
 support test protocols that allow a single test script to run more
 test cases, @emph{each with its distinct result}.  In order for the
-testsuite report to be correct in this case, the test driver must
+testsuite summary to be correct in this case, the test driver must
 register @emph{each} such result in the generated @file{.log} file, using
-the @code{:test-result:} reStructuredText field.
-
-FIXME: currently, that field is recognized anywhere in the @file{.log}
-file; this is possibly prone to generating spurious results, in case
-of verbose tests.  We should allow a special @code{:test-result:} that
-stops any further parsing (maybe a line @code{:test-result:END} will
-do?).
-
-FIXME: we are probably going to change the current semantics of
-testsuite summary generation, in order to allow test protocols with
-only one result per test case not to be forced to also declare that
-result with @code{:test-results:}; that would be a repetition, since
-the result is already declared in the @file{.log} first line.
+the @code{:test-result:} reStructuredText field.  Otherwise, only the
+global test result will be considered (as it is the case of the default
address@hidden driver).
+
address@hidden Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
+The only recognized test results are currently @code{PASS}, @code{XFAIL},
address@hidden, @code{FAIL} and @code{XPASS}.  These results, when declared
+with @code{:test-result:}, can be optionally followed by text holding
+the name and/or a brief description of the corresponding test; the
address@hidden harness will ignore such extra text when
+generating @file{test-suite.log} and preparing the testsuite summary.
+
+For example, if a @file{.log} file contains the following lines:
address@hidden
+:test-result: PASS server starts
+:test-result: PASS HTTP/1.1 request
+:test-result: FAIL HTTP/1.0 request
+:test-result: SKIP HTTPS request (TLS library wasn't available)
+:test-result: PASS server stops
address@hidden example
+it means that the corresponding test script contributes with five test
+results to the testsuite summary (three of these tests being successful,
+one failed, and one skipped).
+
address@hidden FIXME: currently, the @code{:test-result:} field is recognized 
anywhere
address@hidden in the @file{.log} file; this is possibly prone to generating 
spurious
address@hidden results, in case of verbose tests.  We should allow a special
address@hidden @code{:test-result:} that stops any further parsing (maybe a line
address@hidden @code{:test-result:END} will do?).
+
 @end itemize
 
address@hidden: Explain why supporting the Global test result declaration
-in the first line is useful also for test protocols that allow more
-test cases per test script.
-
address@hidden
address@hidden  * A test driver is free (but not forced) to register multiple 
test
address@hidden    results for a single test script, through the use of a custom
address@hidden    reStructuredText field ``test-result''.  Each of this results 
will
address@hidden    be accounted for separately in the final testsuite summary;
address@hidden    TODO: give an example
address@hidden    TODO: a special field to terminate parsing of .log?
address@hidden    But note that the task of reporting this results on the 
screen is
address@hidden    left to the individual test driver though; more about this 
below.
address@hidden    For example, if the test script executes
address@hidden    two test cases, one successful and one failing, and skip 
another
address@hidden    test case, the driver should end up writing the following in 
the
address@hidden    test log:
address@hidden      :test-result: PASS [name of passed test, and/or details 
about it]
address@hidden      :test-result: FAIL [name of failed test, and/or details 
about it]
address@hidden      :test-result: SKIP [name of skipped test, and/or details 
about it]
address@hidden    The above lines (each of which *must* be followed by a blank 
line
address@hidden    in order for the HTML output generation to work) are used only
address@hidden    when generating the `test-suite.log' from the individual test
address@hidden    logs, and can be placed in any order and position within the 
logs
address@hidden    itself.
address@hidden
address@hidden
+Finally, note that the declaration of the global test result in the first
+line, apart from being needed for backward-compatibility with the default
address@hidden driver, can be useful also for test protocols
+that allow more test results per test script: using it, a custom test
+driver is allowed to decide what the global outcome of the test script
+is in case of conflicting test results within the script.  For example,
+if a test script reports 8 successful test cases and 2 skipped test
+cases, some drivers might report that globally as a SKIP, while others
+as a PASS.
 
 @node Console output during testsuite runs
 @subsubsection Console output during testsuite runs
 
-TODO.
address@hidden  * explain what kind of console output the driver is expected to 
produce;
address@hidden    it can have some leeway here, but IMHO should provide a "look 
& feel"
address@hidden    similar to the existing "parallel-tests" output.
address@hidden  * tell that this is where the --color option is to be 
influential.
+A custom test driver has also the task of displaying, on the standard
+output, the test results as soon as they become available.  Depending on
+the protocol in use, it can also display the reasons for failures and
+skips, and, more generally, any useful diagnostic output (but remember
+that each line on the screen is precious, so that cluttering the screen
+with overly verbose information is bad idea).  The exact format of this
+progress output is left up completely to the test drive; in fact, a custom
+test driver might @emph{theoretically} even decide not to do any such
+report, leaving it all to the testsuite summary (that would be a very
+lousy idea, of course, and serves only to illustrate the flexibility that
+is granted here).
+
+Remember that consistency is good; so, if possible, try to be consistent
+with the output of the built-in Automake test drivers, providing a similar
+``look & feel''.  In particular, the testsuite progress output should be
+colorized when the @option{--color-tests} is passed to the driver.  On the
+other end, if you are using a known and widespread test protocol with
+well-established implementations, being consistent with those
+implementations' output might be a good idea too.
+
address@hidden TODO: Give an example, maybe inspired to py.test-style output.
address@hidden TODO: That is a good idea because it shows a test driver that 
allows
address@hidden TODO: for different levels of verbosity in the progress output 
(could
address@hidden TODO: be implemented either using a driver cmdline flag, or an
address@hidden TODO: environment variable, or both).
 
 @node HTML generation from testsuite logs
 @subsubsection HTML generation from testsuite logs
 
 If HTML testsuite output (with @code{check-html}) is to be supported,
 the generated @file{.log} files must contain syntactically valid
-reStructuredText.  If this is not the case, the HTML generation will not
-work, but all the other functionalities of the Automake testsuite harness
-should remain untouched, and continue to work correctly.
+reStructuredText (among the other things, this means that every
address@hidden:test-result:} line must be followed by a blank line).  If this
+is not the case, the HTML generation will not work, although all the
+other functionalities of the Automake testsuite harness should remain
+untouched, and continue to work correctly.
 
 @node DejaGnu Tests
 @section DejaGnu Tests
diff --git a/lib/am/check.am b/lib/am/check.am
index cbf0acb..b02d6c5 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -205,16 +205,8 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
          echo;                                                         \
          echo ".. contents:: :depth: 2";                               \
          echo;                                                         \
-## Here we assume that the test driver writes a proper summary for the
-## test script on the first line.  Requiring this is not a limitation,
-## but a feature, since this way a custom test driver is allowed to decide
-## what the outcome is in case of conflicting testcase results in a test
-## script.  For example, if a test script reports 8 successful testcases
-## and 2 skipped testcases, some drivers might report that globally as a
-## SKIP, while others as a PASS.
-## FIXME: This should be documented in the automake manual.  The above
-## FIXME: explanation is indeed more appropriate for the manual than for
-## FIXME: comments in code.
+## Here we assume that the test driver writes a proper "summarizing
+## result" for the test script on the first line.
          for f in $$list; do                                           \
            test -r $$f && read line < $$f || line=;                    \
            case $$line in                                              \