[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH 2/2] test harness: use new `.trs' files to hold test metadata (wa
From: |
Stefano Lattarini |
Subject: |
[PATCH 2/2] test harness: use new `.trs' files to hold test metadata (was: Re: [PATCH 1/2] test harness: allow more metadata in log files) |
Date: |
Tue, 2 Aug 2011 00:09:28 +0200 |
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
On Sunday 31 July 2011, Stefano Lattarini wrote:
> On Friday 29 July 2011, Stefano Lattarini wrote:
> > Following up on this:
> > <https://lists.gnu.org/archive/html/automake-patches/2011-07/msg00070.html>
> >
> > On Tuesday 19 July 2011, Stefano Lattarini wrote:
> > > On Monday 18 July 2011, Ralf Wildenhues wrote:
> > > > * Stefano Lattarini wrote on Fri, Jul 15, 2011 at 12:36:17AM CEST:
> > > > > The new code for parsing the testsuite-generated `.log' files,
> > > > > as introduced in commit `v1.11-872-gc96b881', considers each
> > > > > `:test-result:' field anywhere in a `.log' file as a declaration
> > > > > of a test result, and accounts for it as such in the testsuite
> > > > > summary. Unfortunately this could easily cause spurious test
> > > > > failures being reported in the testsuite summary. This happened
> > > > > in practice with the Automake's own testsuite; for example:
> > > > >
> > > > > $ make check TESTS='check12-p.test'; echo exit: $?
> > > > > ...
> > > > > PASS: check12-p.test
> > > > > =====================================
> > > > > 4 of 5 tests failed
> > > > > See tests/test-suite.log
> > > > > Please report to address@hidden
> > > > > =====================================
> > > > > make[2]: *** [test-suite.log] Error 1
> > > > > make: *** [check-am] Error 2
> > > > > exit: 2
> > > > >
> > > > > This change introduces a new special `:test-result:' "END", that,
> > > > > when seen, prevents the rest of the log file from being parsed.
> > > > >
> > > > > For more information, refer to the thread:
> > > > > <http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html>
> > > > >
> > > > > * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log
> > > > > file as soon as the special ":test-result:END" directive is seen.
> > > > > Related changes and enhancements.
> > > > > * lib/test-driver: Protect the rest of the log after the result
> > > > > lined with a ":test-result:END" directive.
> > > > > * tests/parallel-tests-no-spurious-summary.test: New test.
> > > > > * tests/test-driver-end-test-results.test: Likewise.
> > > > > * tests/Makefile.am (TESTS): Update.
> > > >
> > > > I'm still not sold on this. It is not as robust as a test protocol
> > > > could be; also I haven't seen this approach being used in any other test
> > > > suite environments. Whether some line is considered having a result or
> > > > not depends on (possibly far-away) context, on whether aggregation with
> > > > other results has happened. When passed through email, misquoting can
> > > > change not only the interpretation of the misquoted text (which could be
> > > > expected) but possibly also of later, correctly quoted (or not quoted)
> > > > text.
> > > >
> > > Good points. Hmm... the more I think about this issue, the more I'm
> > > convinced that we are (ok, I am ;-) starting to unduly use the `.log'
> > > files for two ortoghonal purposes:
> > > 1. keeping a verbose and detailed log of a test script run; and
> > > 2. keeping a sort of "dumb database" of test results, which is used
> > > to determine the final testsuite summary (and also by "make recheck"
> > > to decide which tests to re-run).
> > > Maybe it's time we begin to use two different set of files for this two
> > > different roles? E.g., for each `foo.test', we generate a `foo.log' file
> > > and a, say, `foo.tdat' (Test DATa file) containing the test results and
> > > possibly other metadata used by our harness? This might also allow for
> > > easier extensibility of such metadata in the future.
> > >
> > I'm pursuing this strategy now; it requires some trickery, but it seems
> > to work quite well, and not to cause any codebase bloating. Also, the
> > documentation of the new behaviour will be clearer and smaller, and the
> > addition of new complexity in the testsuite harness will be paid off with
> > a decrease in complexity of the test drivers (a win in the long term, as
> > we plan to add more test drivers in the future).
> >
> > Here is a refactoring/preparatory patch, which makes the metadata in the
> > log files more flexible and more sophisticated; while doing so, it IMHO
> > also shows that a new set of generated files holding only test metadata
> > would be more natural and flexible.
> >
> > Given what I've said, please do not judge this patch in isolation;
> > consider it only a "transitional step" towards the new APIs.
> >
> And here is the patch implementing this new APIs. It still lacks
> documentation in `doc/automake.texi' (I hope I'll be able to post that
> tomorrow),
>
Attached is the squash-in with the documentation addition, plus some
tweakings "sparked" by it. The amended patch is also attached, for
reference.
Regards,
Stefano
diff --git a/ChangeLog b/ChangeLog
index 82a8981..c1f13b6 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -7,6 +7,9 @@
extension of test metadata, and for more flexibility in the
format of the `.log' files. Note that this change makes the
`:end-metadata:' field obsolete.
+ * doc/automake.texi (Parallel Test Harness, Log files generation
+ and test results recording): Document the new APIs and behaviour;
+ some related minor rewordings and fixlets.
* automake.in (handle_tests): When bringing in the content of
`check2.am', substitute %BASE% with the basename of the `.log'
file being created by a rule. Add the generated `.trs' files
@@ -34,21 +37,20 @@
`.trs' files, not in the `.log' files.
* lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust
the call to the test driver, in particularly passing the new
- option `--result-file', which specifies the path for the `.trs'
- file where the metadata for the test run will be saved.
+ option `--trs-file'.
[%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise.
* lib/tap-driver ($USAGE): Adjust the help screen.
- (Getopt::Long::GetOptions): Handle the `--result-file' option,
+ (Getopt::Long::GetOptions): Handle the `--trs-file' option,
through the use of ...
- ($result_file): ... this new global variable.
- (finish): Write metadata for the test run to `$result_file' rather
+ ($trs_file): ... this new global variable.
+ (finish): Write metadata for the test run to `$trs_file' rather
then to `$log_file', through the use of ...
(write_test_results): ... this new function.
* lib/test-driver (print_usage): Update the help screen.
- (Option parsing): Handle the `--result-file' option, through the
+ (Option parsing): Handle the `--trs-file' option, through the
use of ...
($resfile): ... this new global variable.
- (Main code): Write metadata for the test run to `$resfile' rather
+ (Main code): Write metadata for the test run to `$trsfile' rather
than to `$logfile'.
Minor related adjustments to comments.
* tests/.gitignore: Ignore `*.trs' files.
@@ -73,14 +75,6 @@
* test-trs-recover2.test: Likewise.
* tests/Makefile.am (TESTS): Update.
-## We choose the `.trs' extension (as in Test ReSults) for the files holding
-## our test results. These seems a good choice because such an extension is
-## not already used for other significant purposes; see:
-## - http://filext.com/file-extension/trs
-## - http://www.file-extensions.org/search/?searchstring=trs
-am__TEST_RESULTS = $(am__TEST_BASES:=.trs)
-
-
2011-07-27 Stefano Lattarini <address@hidden>
test harness: allow more metadata in log files
diff --git a/doc/automake.texi b/doc/automake.texi
index 2e5e2ba..d02e6ca 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8975,12 +8975,27 @@ order to satisfy additional portability requirements.
@vindex TEST_SUITE_LOG
@vindex TESTS
-The parallel harness operates by defining a set of @command{make} rules
-to create a summary log file, @code{TEST_SUITE_LOG}, which defaults to
address@hidden and requires a @file{.log} suffix. This file
-depends upon log files created for each single test program listed in
address@hidden, which in turn contain all output produced by the
-corresponding tests.
address@hidden @file{.log} files
address@hidden @file{.trs} files
address@hidden test metadata
+The parallel test harness operates by defining a set of @command{make}
+rules that run the test scripts listed in @code{TESTS}, and, for each
+such script, save its output in a corresponding @file{.log} file and
+its results (and other ``metadata'', @pxref{API for Custom Test Drivers})
+in a corresponding @file{.trs} (as in @b{T}est @address@hidden) file.
address@hidden We choose the `.trs' extension also because, at the time of
writing,
address@hidden it isn't already used for other significant purposes; see e.g.:
address@hidden - http://filext.com/file-extension/trs
address@hidden - http://www.file-extensions.org/search/?searchstring=trs
+The @file{.log} file will contain all the output emitted by the test on
+its standard output and its standard error. The @file{.trs} file will
+contain, among the other things, the results of the test cases run by
+the script.
+
+The parallel test harness will also create a summary log file,
address@hidden, which defaults to @file{test-suite.log} and requires
+a @file{.log} suffix. This file depends upon all the @file{.log} and
address@hidden files created for the test scripts listed in @code{TESTS}.
@vindex VERBOSE
As with the serial harness above, by default one status line is printed
@@ -8989,16 +9004,17 @@ However, standard output and standard error of the test
are redirected
to a per-test log file, so that parallel execution does not produce
intermingled output. The output from failed tests is collected in the
@file{test-suite.log} file. If the variable @samp{VERBOSE} is set, this
-file is output after the summary. For best results, the tests should be
-verbose by default now.
+file is output after the summary.
address@hidden FIXME: we should be clearer about what we mean exactly here ...
+For best results, the tests should be verbose by default now.
@vindex TEST_EXTENSIONS
@vindex TEST_LOGS
-Each log file is created when the corresponding test has completed.
-The set of log files is listed in the read-only variable
address@hidden, and defaults to @code{TESTS}, with the executable
-extension if any (@pxref{EXEEXT}), as well as any suffix listed in
address@hidden removed, and @file{.log} appended.
+Each couple of @file{.log} and @file{.trs} files is created when the
+corresponding test has completed. The set of log files is listed in
+the read-only variable @code{TEST_LOGS}, and defaults to @code{TESTS},
+with the executable extension if any (@pxref{EXEEXT}), as well as any
+suffix listed in @code{TEST_EXTENSIONS} removed, and @file{.log} appended.
@code{TEST_EXTENSIONS} defaults to @file{.test}. Results are undefined
if a test file name ends in several concatenated suffixes.
@@ -9036,7 +9052,9 @@ AM_LOG_FLAGS = -d
@noindent
will invoke @samp{$(PERL) -w foo.pl}, @samp{$(PYTHON) -v bar.py},
and @samp{./wrapper-script -d baz} to produce @file{foo.log},
address@hidden, and @file{baz.log}, respectively.
address@hidden, and @file{baz.log}, respectively. The @file{foo.trs},
address@hidden and @file{baz.trs} files will be automatically produced
+as a side-effect.
It's important to note that, differently from what we've seen for the
serial test harness (@pxref{Parallel Test Harness}), the
@@ -9110,14 +9128,15 @@ here too.
@item
@vindex RECHECK_LOGS
@cindex lazy test execution
-By default, the test harness removes all old per-test log files before
-it starts running tests to regenerate them. The variable
address@hidden contains the set of log files which are removed.
address@hidden defaults to @code{TEST_LOGS}, which means all tests
-need to be rechecked. By overriding this variable, you can choose which
-tests need to be reconsidered. For example, you can lazily rerun only
-those tests which are outdated, i.e., older than their prerequisite test
-files, by setting this variable to the empty value:
+By default, the test harness removes all old per-test @file{.log} and
address@hidden files before it starts running tests to regenerate them. The
+variable @code{RECHECK_LOGS} contains the set of @file{.log} (and, by
+implication, @file{.trs}) files which are removed. @code{RECHECK_LOGS}
+defaults to @code{TEST_LOGS}, which means all tests need to be rechecked.
+By overriding this variable, you can choose which tests need to be
+reconsidered. For example, you can lazily rerun only those tests which
+are outdated, i.e., older than their prerequisite test files, by setting
+this variable to the empty value:
@example
env RECHECK_LOGS= make -e check
@@ -9130,14 +9149,14 @@ You can ensure that all tests are rerun which have
failed or passed
unexpectedly, by running @code{make recheck} in the test directory.
This convenience target will set @code{RECHECK_LOGS} appropriately
before invoking the main test harness. The @code{recheck-html} target
-does the same as @code{recheck} but again converts the resulting log
-file in HTML format, like the @code{check-html} target.
+does the same as @code{recheck} but again converts the resulting
address@hidden file in HTML format, like the @code{check-html} target.
@end itemize
In order to guarantee an ordering between tests even with @code{make
address@hidden, dependencies between the corresponding log files may be
-specified through usual @command{make} dependencies. For example, the
-following snippet lets the test named @file{foo-execute.test} depend
address@hidden, dependencies between the corresponding @file{.log} files
+may be specified through usual @command{make} dependencies. For example,
+the following snippet lets the test named @file{foo-execute.test} depend
upon completion of the test @file{foo-compile.test}:
@example
@@ -9201,12 +9220,12 @@ the package authors to use third-party custom test
drivers, in case the
default ones are inadequate for their purposes, or do not support their
testing protocol of choice.
-A custom test driver is expected to properly run the test programs
-passed to it, including the command-line arguments passed to it,
-to analyze their execution and outcome, to create the @file{.log}
-files associated to these test runs, and to display the test results
-on the console. It is responsibility of the author of the test driver
-to ensure that it implements all the above steps meaningfully and
+A custom test driver is expected to properly run the test programs passed
+to it (including the command-line arguments passed to those programs, if
+any), to analyze their execution and outcome, to create the @file{.log}
+and @file{.trs} files associated to these test runs, and to display the test
+results on the console. It is responsibility of the author of the test
+driver to ensure that it implements all the above steps meaningfully and
correctly; Automake isn't and can't be of any help here. On the other
hand, the Automake-provided code for testsuite summary generation offers
support for test drivers allowing several test results per test script,
@@ -9233,8 +9252,8 @@ runtime through the redefinition of @code{TESTS} or
@code{TEST_LOGS};
@item
concurrency through the use of @command{make}'s option @option{-j};
@item
-per-test @file{.log} files, and generation of a summary @file{.log}
-file from them;
+per-test @file{.log} and @file{.trs} files, and generation of a summary
address@hidden file from them;
@item
@code{recheck} target, @code{RECHECK_LOGS} variable, and lazy reruns
of tests;
@@ -9337,10 +9356,14 @@ The name of the test, with VPATH prefix (if any)
removed. This can have a
suffix and a directory component (as in e.g., @file{sub/foo.test}), and is
mostly meant to be used in console reports about testsuite advancements and
results (@pxref{Testsuite progress output}).
address@hidden address@hidden
-The log file the test driver must create. If it has a directory component
-(as in e.g., @file{sub/foo.log}), the test harness will ensure that such
-directory exist @emph{before} the test driver is called.
address@hidden address@hidden@var{PATH}.log}
+The @file{.log} file the test driver must create. If it has a directory
+component (as in e.g., @file{sub/foo.log}), the test harness will ensure
+that such directory exist @emph{before} the test driver is called.
address@hidden address@hidden@var{PATH}.trs}
+The @file{.trs} file the test driver must create. If it has a directory
+component (as in e.g., @file{sub/foo.trs}), the test harness will ensure
+that such directory exist @emph{before} the test driver is called.
@item address@hidden|address@hidden
Whether the console output should be colorized or not (@pxref{Simple
tests and color-tests}, to learn when this option gets activated and
@@ -9370,14 +9393,24 @@ consistency and a more pleasant user experience.
@node Log files generation and test results recording
@subsubsection Log files generation and test results recording
-The test driver must correctly generate the file specified by the
address@hidden option (of course even when the tested program
-fails or crashes). This file, while being quite free-form, @emph{must}
-register some metadata through the use of custom reStructuredText fields,
-in order to work with the @option{parallel-tests} harness and take
-advantage of its features. Note that such fields will be recognized only
-when used @emph{exactly} at the beginning of a line -- not even leading
-whitespace is permitted.
+The test driver must correctly generate the files specified by the
address@hidden and @option{--trs-file} option (even when the tested
+program fails or crashes).
+
+The @file{.log} file should ideally contain all the output produced by the
+tested program, plus optionally other information that might facilitate
+debugging or analysis of bug reports. Apart from that, its format is
+basically free; the only limitation being that it must parse validly as
+reStructuredText if the @file{.log} -> @file{.html} conversion is to be
+supported.
+
+The @file{.trs} file is used to register some metadata through the use
+of custom reStructuredText fields. This metadata is expected to be
+employed in various ways by the parallel test harness; for example, to
+count the test results when printing the testsuite summary, or to decide
+which tests to re-run upon @command{make reheck}. Unrecognized metadata
+in a @file{.trs} file is currently ignored by the harness, but this might
+change in the future. The list of currently recognized metadata follows.
@table @code
@@ -9388,10 +9421,11 @@ whitespace is permitted.
@cindex Test case result, registering
@cindex @code{:test-result:}
@cindex reStructuredText field, @code{:test-result:}
-The test driver must use this filed to register the results of @emph{each}
+The test driver must use this field to register the results of @emph{each}
test case run by a test script file. Several @code{:test-result:} fields
-can be present in the same log file; this is done in order to support test
-protocols that allow a single test script to run more test cases.
+can be present in the same @file{.trs} file; this is done in order to
+support test protocols that allow a single test script to run more test
+cases.
@c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
The only recognized test results are currently @code{PASS}, @code{XFAIL},
@@ -9401,16 +9435,16 @@ text holding the name and/or a brief description of the
corresponding
test; the @option{parallel-tests} harness will ignore such extra text when
generating @file{test-suite.log} and preparing the testsuite summary.
address@hidden Keep in sync with 'test-driver-recheck.test'.
address@hidden Keep in sync with 'test-metadata-recheck.test'.
@item @code{:recheck:}
@cindex :recheck:
@cindex reStructuredText field, @code{:recheck:}
If this field is present and defined to @code{no}, then the corresponding
test script will @emph{not} be run upon a @command{make recheck}. What
-happens when two or more @code{:recheck:} fields are present in a
address@hidden file is undefined behaviour.
+happens when two or more @code{:recheck:} fields are present in the same
address@hidden file is undefined behaviour.
address@hidden Keep in sync with 'test-driver-global-log.test'.
address@hidden Keep in sync with 'test-metadata-global-log.test'.
@item @code{:copy-in-global-log:}
@cindex :copy-in-global-log:
@cindex reStructuredText field, @code{:copy-in-global-log:}
@@ -9420,43 +9454,30 @@ of the @file{.log} file will @emph{not} be copied into
the global
it can be useful in debugging and analysis of bug report, it can also be
just a waste of space in normal situations, e.g., when a test script is
successful. What happens when two or more @code{:copy-in-global-log:}
-fields are present in a @file{.log} file is undefined behaviour.
-
address@hidden Keep in sync with 'test-driver-end-metadata.test'.
address@hidden @code{:end-metadata:}
address@hidden :end-metadata:
address@hidden reStructuredText field, @code{:end-metadata:}
-This special field causes all the following text in the @file{.log}
-file not to be scanned by the testsuite harness in search for other
-test metadata. This feature is used internally by the default
address@hidden driver, and can be useful to e.g., avoid
-spurious testsuite results.
+fields are present in the same @file{.trs} file is undefined behaviour.
@end table
@noindent
-Let's see a small example. Assume a @file{.log} file contains the
-following lines (in this order, but possibly interleaved with other
-``non-metadata'' lines):
+Let's see a small example. Assume a @file{.trs} file contains the
+following lines:
@example
:test-result: PASS server starts
+:global-log-copy: no
:test-result: PASS HTTP/1.1 request
:test-result: FAIL HTTP/1.0 request
:recheck: yes
:test-result: SKIP HTTPS request (TLS library wasn't available)
:test-result: PASS server stops
-:global-log-copy: no
-:metadata-end:
-:test-result: XPASS (this shouldn't be parsed as a test result)
@end example
@noindent
Then the corresponding test script will be re-run by @command{make check},
will contribute with @emph{five} test results to the testsuite summary
(three of these tests being successful, one failed, and one skipped), and
-the @file{.log} file content will @emph{not} be copied in the global log
-file @file{test-suite.log}.
+the content of the corresponding @file{.log} file will @emph{not} be
+copied in the global log file @file{test-suite.log}.
@node Testsuite progress output
@subsubsection Testsuite progress output
@@ -9492,11 +9513,9 @@ implementations' output might be a good idea too.
If HTML testsuite output (with @code{check-html}) is to be supported,
the generated @file{.log} files must contain syntactically valid
-reStructuredText (among the other things, this means that every
address@hidden:test-result:} line must be followed by a blank line). If this
-is not the case, the HTML generation will not work, although all the
-other functionalities of the Automake testsuite harness should remain
-untouched, and continue to work correctly.
+reStructuredText. If this is not the case, the HTML generation will not
+work, although all the other functionalities of the Automake testsuite
+harness should remain untouched, and continue to work correctly.
@node Using the TAP test protocol
@section Using the TAP test protocol
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index d6a6152..78a7e4a 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -496,7 +496,7 @@ recheck recheck-html:
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS)
TEST_LOGS="'"$$list"'"'
.pl.log:
@p='$<'; $(am__check_pre) $(PL_LOG_DRIVER) --test-name "$$f" \
- --log-file '$*.log' --result-file '$*.trs' \
+ --log-file '$*.log' --trs-file '$*.trs' \
$(am__common_driver_flags) $(AM_PL_LOG_DRIVER_FLAGS)
$(PL_LOG_DRIVER_FLAGS) -- $(PL_LOG_COMPILE) "$$tst" \
$(AM_TESTS_FD_REDIRECT)
diff --git a/lib/am/check2.am b/lib/am/check2.am
index 8b9a703..ad0a4aa 100644
--- a/lib/am/check2.am
+++ b/lib/am/check2.am
@@ -18,7 +18,7 @@
?GENERIC?%EXT%.log:
?!GENERIC?%OBJ%: %SOURCE%
@p='%SOURCE%'; $(am__check_pre) %DRIVER% --test-name "$$f" \
- --log-file '%BASE%.log' --result-file '%BASE%.trs' \
+ --log-file '%BASE%.log' --trs-file '%BASE%.trs' \
$(am__common_driver_flags) %DRIVER_FLAGS% -- %COMPILE% "$$tst" \
$(AM_TESTS_FD_REDIRECT)
@@ -29,7 +29,7 @@
if %am__EXEEXT%
?GENERIC?%EXT%$(EXEEXT).log:
@p='%SOURCE%'; $(am__check_pre) %DRIVER% --test-name "$$f" \
- --log-file '%BASE%.log' --result-file '%BASE%.trs' \
+ --log-file '%BASE%.log' --trs-file '%BASE%.trs' \
$(am__common_driver_flags) %DRIVER_FLAGS% -- %COMPILE% "$$tst" \
$(AM_TESTS_FD_REDIRECT)
endif %am__EXEEXT%
diff --git a/lib/tap-driver b/lib/tap-driver
index 7681fb4..8237fec 100755
--- a/lib/tap-driver
+++ b/lib/tap-driver
@@ -15,7 +15,7 @@ my $ME = "tap-driver";
my $USAGE = <<'END';
Usage:
- tap-driver --test-name=NAME --log-file=PATH --result-file=PATH
+ tap-driver --test-name=NAME --log-file=PATH --trs-file=PATH
[--expect-failure={yes|no}] [--color-tests={yes|no}]
[--enable-hard-errors={yes|no}] [--merge|--no-merge]
[--ignore-exit] [--comments|--no-comments] [--] TEST-COMMAND
@@ -62,14 +62,14 @@ my %cfg = (
my $test_script_name = undef;
my $log_file = undef;
-my $result_file = undef;
+my $trs_file = undef;
Getopt::Long::GetOptions (
'help' => sub { print $HELP; exit 0; },
'version' => sub { print "$ME $VERSION"; exit 0; },
'test-name=s' => \$test_script_name,
'log-file=s' => \$log_file,
- 'result-file=s' => \$result_file,
+ 'res-file=s' => \$trs_file,
'color-tests=s' => \&bool_opt,
'expect-failure=s' => \&bool_opt,
'enable-hard-errors=s' => \&bool_opt,
@@ -168,7 +168,7 @@ TEST_RESULTS :
sub write_test_results ()
{
- open RES, ">", $result_file or die "opening $result_file: $!\n";
+ open RES, ">", $trs_file or die "opening $trs_file: $!\n";
print RES ":recheck: " .
(must_recheck ? "yes" : "no") . "\n";
print RES ":copy-in-global-log: " .
@@ -177,7 +177,7 @@ sub write_test_results ()
{
print RES ":test-result: $result\n";
}
- close RES or die "closing $result_file: $!\n";
+ close RES or die "closing $trs_file: $!\n";
}
sub start (@)
diff --git a/lib/test-driver b/lib/test-driver
index ccdf873..472ba49 100755
--- a/lib/test-driver
+++ b/lib/test-driver
@@ -1,7 +1,7 @@
#! /bin/sh
# test-driver - basic driver script for the `parallel-tests' mode.
-scriptversion=2011-07-27.12; # UTC
+scriptversion=2011-08-01.21; # UTC
# Copyright (C) 2011 Free Software Foundation, Inc.
#
@@ -48,7 +48,7 @@ print_usage ()
{
cat <<END
Usage:
- test-driver --test-name=NAME --log-file=PATH --result-file=PATH
+ test-driver --test-name=NAME --log-file=PATH --trs-file=PATH
[--expect-failure={yes|no}] [--color-tests={yes|no}]
[--enable-hard-errors={yes|no}] [--] TEST-SCRIPT
The \`--test-name' and \`--log-file' options are mandatory.
@@ -59,10 +59,10 @@ END
rst_section () { sed 'p;s/./=/g;p;g'; }
# TODO: better error handling in option parsing (in particular, ensure
-# TODO: $logfile, $resfile and $test_name are defined).
+# TODO: $logfile, $trsfile and $test_name are defined).
test_name= # Used for reporting.
logfile= # Where to save the output of the test script.
-resfile= # Where to save the result(s) the test script.
+trsfile= # Where to save the result(s) the test script.
expect_failure=no
color_tests=no
enable_hard_errors=yes
@@ -72,7 +72,7 @@ while test $# -gt 0; do
--version) echo "test-driver $scriptversion"; exit $?;;
--test-name) test_name=$2; shift;;
--log-file) logfile=$2; shift;;
- --result-file) resfile=$2; shift;;
+ --trs-file) trsfile=$2; shift;;
--color-tests) color_tests=$2; shift;;
--expect-failure) expect_failure=$2; shift;;
--enable-hard-errors) enable_hard_errors=$2; shift;;
@@ -121,9 +121,9 @@ esac
echo "${col}${res}${std}: $test_name"
# Register the test result, and other relevant metadata.
-echo ":test-result: $res (exit status: $estatus)" > $resfile
-echo ":recheck: $recheck" >> $resfile
-echo ":copy-in-global-log: $gcopy" >> $resfile
+echo ":test-result: $res (exit status: $estatus)" > $trsfile
+echo ":recheck: $recheck" >> $trsfile
+echo ":copy-in-global-log: $gcopy" >> $trsfile
# Now write log file.
{
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 43a57d5..1732555 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -1615,7 +1615,7 @@ recheck recheck-html:
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS)
TEST_LOGS="'"$$list"'"'
.test.log:
@p='$<'; $(am__check_pre) $(TEST_LOG_DRIVER) --test-name "$$f" \
- --log-file '$*.log' --result-file '$*.trs' \
+ --log-file '$*.log' --trs-file '$*.trs' \
$(am__common_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS)
$(TEST_LOG_DRIVER_FLAGS) -- $(TEST_LOG_COMPILE) "$$tst" \
$(AM_TESTS_FD_REDIRECT)
diff --git a/tests/test-driver-create-log-dir.test
b/tests/test-driver-create-log-dir.test
index a7f83ba..17f56c4 100755
--- a/tests/test-driver-create-log-dir.test
+++ b/tests/test-driver-create-log-dir.test
@@ -52,7 +52,7 @@ set -u
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
- --result-file) trs_file=$2; shift;;
+ --rts-file) trs_file=$2; shift;;
--test-name|--expect-failure|--color-tests|--enable-hard-errors) shift;;
--) shift; break;;
*) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
diff --git a/tests/test-driver-custom-xfail-tests.test
b/tests/test-driver-custom-xfail-tests.test
index aabf2f1..e68e04b 100755
--- a/tests/test-driver-custom-xfail-tests.test
+++ b/tests/test-driver-custom-xfail-tests.test
@@ -98,7 +98,7 @@ while test $# -gt 0; do
--test-name) test_name=$2; shift;;
--expect-failure) expect_failure=$2; shift;;
--log-file) log_file=$2; shift;;
- --result-file) trs_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
# Ignored.
--color-tests) shift;;
--enable-hard-errors) shift;;
diff --git a/tests/test-driver-custom.test b/tests/test-driver-custom.test
index dec6aee..8305047 100755
--- a/tests/test-driver-custom.test
+++ b/tests/test-driver-custom.test
@@ -65,7 +65,7 @@ while test $# -gt 0; do
case $1 in
--test-name) test_name=$2; shift;;
--log-file) log_file=$2; shift;;
- --result-file) trs_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
# Ignored.
--expect-failure) shift;;
--color-tests) shift;;
diff --git a/tests/test-driver-strip-vpath.test
b/tests/test-driver-strip-vpath.test
index c2b3113..5b21cc0 100755
--- a/tests/test-driver-strip-vpath.test
+++ b/tests/test-driver-strip-vpath.test
@@ -51,7 +51,7 @@ set -e
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
- --result-file) trs_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
--test-name) test_name=$2; shift;;
--expect-failure|--color-tests|--enable-hard-errors) shift;;
--) shift; break;;
diff --git a/tests/test-metadata-global-log.test
b/tests/test-metadata-global-log.test
index c949bce..a6f0999 100755
--- a/tests/test-metadata-global-log.test
+++ b/tests/test-metadata-global-log.test
@@ -38,7 +38,7 @@ set -e; set -u;
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
- --result-file) trs_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
--test-name) test_name=$2; shift;;
--expect-failure|--color-tests|--enable-hard-errors) shift;;
--) shift; break;;
diff --git a/tests/test-metadata-recheck.test b/tests/test-metadata-recheck.test
index 462a32e..6a372f3 100755
--- a/tests/test-metadata-recheck.test
+++ b/tests/test-metadata-recheck.test
@@ -123,7 +123,7 @@ set -e; set -u
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
- --result-file) trs_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
--test-name) test_name=$2; shift;;
--expect-failure|--color-tests|--enable-hard-errors) shift;;
--) shift; break;;
diff --git a/tests/test-metadata-results.test b/tests/test-metadata-results.test
index 0cd97b7..effa488 100755
--- a/tests/test-metadata-results.test
+++ b/tests/test-metadata-results.test
@@ -35,7 +35,7 @@ set -e; set -u
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
- --result-file) trs_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
--test-name) test_name=$2; shift;;
--expect-failure|--color-tests|--enable-hard-errors) shift;;
--) shift; break;;
diff --git a/tests/trivial-test-driver b/tests/trivial-test-driver
index ab73d8d..7b3ef10 100644
--- a/tests/trivial-test-driver
+++ b/tests/trivial-test-driver
@@ -41,7 +41,7 @@ while test $# -gt 0; do
case $1 in
--test-name) test_name=$2; shift;;
--log-file) log_file=$2; shift;;
- --result-file) trs_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
# Ignored.
--expect-failure) shift;;
--color-tests) shift;;
From bd16d593264f77ab6b048589a557e5eadbcc8bb4 Mon Sep 17 00:00:00 2001
Message-Id: <address@hidden>
From: Stefano Lattarini <address@hidden>
Date: Thu, 28 Jul 2011 12:25:22 +0200
Subject: [PATCH] test harness: use new `.trs' files to hold test metadata
With this change, the test harness will keep test metadata in
dedicated `.trs' files, instead of having them embedded into the
`.log' files. This allows for easier forward-compatibility and
extension of test metadata, and for more flexibility in the
format of the `.log' files. Note that this change makes the
`:end-metadata:' field obsolete.
* doc/automake.texi (Parallel Test Harness, Log files generation
and test results recording): Document the new APIs and behaviour;
some related minor rewordings and fixlets.
* automake.in (handle_tests): When bringing in the content of
`check2.am', substitute %BASE% with the basename of the `.log'
file being created by a rule. Add the generated `.trs' files
to the list of files to be cleaned by "make mostlyclean".
* lib/am/check.am (am__test_driver_flags): Rename ...
(am__common_driver_flags): ... to this, and remove the flags
`--test-name' and `--log-file' from it: they are now define in
the proper rules in `check2.am'.
(am__TEST_BASES): New internal variable, holding the names of
the tests, with any registered extension removed.
(am__stealth_MAKE): New internal variable, can be used instead of
$(MAKE) in recipes requiring a recursive call to make, but which
are not intended to be executed by "make -n".
(.log.trs): New suffix rule, to recover from deletion of `.trs'
files.
($(TEST_SUITE_LOG)): Almost completely rewritten to follow the
new API of "test logs in `.log' files, test metadata in `.trs'
files". It goes to some length to work correctly in face of
unreadable or missing `.log' and `.trs' files, and to error out
with proper error messages when this is not possible.
[%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale"
`.trs' files (in addition to `.log files) before remaking the
$(TEST_SUITE_LOG).
(recheck, recheck-html): Look for the `:recheck:' field in the
`.trs' files, not in the `.log' files.
* lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust
the call to the test driver, in particularly passing the new
option `--trs-file'.
[%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise.
* lib/tap-driver ($USAGE): Adjust the help screen.
(Getopt::Long::GetOptions): Handle the `--trs-file' option,
through the use of ...
($trs_file): ... this new global variable.
(finish): Write metadata for the test run to `$trs_file' rather
then to `$log_file', through the use of ...
(write_test_results): ... this new function.
* lib/test-driver (print_usage): Update the help screen.
(Option parsing): Handle the `--trs-file' option, through the
use of ...
($resfile): ... this new global variable.
(Main code): Write metadata for the test run to `$trsfile' rather
than to `$logfile'.
Minor related adjustments to comments.
* tests/.gitignore: Ignore `*.trs' files.
* tests/parallel-tests-unreadable-log.test: Moved ...
* tests/parallel-tests-unreadable.test: ... to this, and extended
to also check the semantics for unreadable `.trs' files.
* tests/test-driver-end-metadata.test: Deleted as obsolete.
* tests/test-driver-metadata-no-leading-space.test: Likewise.
* tests/test-driver-global-log.test: Renamed ...
* tests/test-metadata-global-log.test: ... to this, and modified
as to verify the new APIs and semantics.
* tests/test-driver-recheck: Renamed ...
* tests/test-metadata-recheck.test: ... to this, and modified
likewise.
* tests/parallel-tests-once.test: New test.
* tests/parallel-tests-make-n.test: Likewise.
* test-metadata-results.test: Likewise.
* test-missing.test: Likewise.
* test-missing2.test: Likewise.
* test-trs-basic.test: Likewise.
* test-trs-recover.test: Likewise.
* test-trs-recover2.test: Likewise.
* tests/Makefile.am (TESTS): Update.
---
ChangeLog | 77 +++++++++
automake.in | 4 +
doc/automake.texi | 175 +++++++++++---------
lib/Automake/tests/Makefile.in | 118 ++++++++------
lib/am/check.am | 155 ++++++++++--------
lib/am/check2.am | 16 +-
lib/tap-driver | 42 ++---
lib/test-driver | 30 ++--
tests/.gitignore | 1 +
tests/Makefile.am | 16 ++-
tests/Makefile.in | 134 +++++++++-------
tests/parallel-tests-make-n.test | 107 ++++++++++++
...nreadable-log.test => parallel-tests-once.test} | 32 +---
...ble-log.test => parallel-tests-unreadable.test} | 45 ++++--
tests/parallel-tests9.test | 9 -
tests/test-driver-create-log-dir.test | 18 ++-
tests/test-driver-custom-multitest.test | 1 +
tests/test-driver-custom-no-html.test | 4 +-
tests/test-driver-custom-xfail-tests.test | 36 +++--
tests/test-driver-custom.test | 4 +
tests/test-driver-end-metadata.test | 120 -------------
tests/test-driver-metadata-no-leading-space.test | 94 -----------
tests/test-driver-strip-vpath.test | 2 +
...test => test-driver-trs-suffix-registered.test} | 62 ++++----
...obal-log.test => test-metadata-global-log.test} | 132 ++++++++-------
...ver-recheck.test => test-metadata-recheck.test} | 22 ++-
tests/test-metadata-results.test | 169 +++++++++++++++++++
tests/test-missing.test | 72 ++++++++
...ests-unreadable-log.test => test-missing2.test} | 44 ++---
tests/test-trs-basic.test | 138 +++++++++++++++
tests/test-trs-recover.test | 166 +++++++++++++++++++
tests/test-trs-recover2.test | 133 +++++++++++++++
tests/trivial-test-driver | 58 +++----
33 files changed, 1484 insertions(+), 752 deletions(-)
create mode 100755 tests/parallel-tests-make-n.test
copy tests/{parallel-tests-unreadable-log.test => parallel-tests-once.test}
(59%)
copy tests/{parallel-tests-unreadable-log.test =>
parallel-tests-unreadable.test} (58%)
delete mode 100755 tests/test-driver-end-metadata.test
delete mode 100755 tests/test-driver-metadata-no-leading-space.test
copy tests/{parallel-tests-unreadable-log.test =>
test-driver-trs-suffix-registered.test} (51%)
rename tests/{test-driver-global-log.test => test-metadata-global-log.test}
(62%)
rename tests/{test-driver-recheck.test => test-metadata-recheck.test} (93%)
create mode 100755 tests/test-metadata-results.test
create mode 100755 tests/test-missing.test
rename tests/{parallel-tests-unreadable-log.test => test-missing2.test} (56%)
create mode 100755 tests/test-trs-basic.test
create mode 100755 tests/test-trs-recover.test
create mode 100755 tests/test-trs-recover2.test
diff --git a/ChangeLog b/ChangeLog
index f40b307..039f043 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,80 @@
+2011-08-01 Stefano Lattarini <address@hidden>
+
+ test harness: use new `.trs' files to hold test metadata
+ With this change, the test harness will keep test metadata in
+ dedicated `.trs' files, instead of having them embedded into the
+ `.log' files. This allows for easier forward-compatibility and
+ extension of test metadata, and for more flexibility in the
+ format of the `.log' files. Note that this change makes the
+ `:end-metadata:' field obsolete.
+ * doc/automake.texi (Parallel Test Harness, Log files generation
+ and test results recording): Document the new APIs and behaviour;
+ some related minor rewordings and fixlets.
+ * automake.in (handle_tests): When bringing in the content of
+ `check2.am', substitute %BASE% with the basename of the `.log'
+ file being created by a rule. Add the generated `.trs' files
+ to the list of files to be cleaned by "make mostlyclean".
+ * lib/am/check.am (am__test_driver_flags): Rename ...
+ (am__common_driver_flags): ... to this, and remove the flags
+ `--test-name' and `--log-file' from it: they are now define in
+ the proper rules in `check2.am'.
+ (am__TEST_BASES): New internal variable, holding the names of
+ the tests, with any registered extension removed.
+ (am__stealth_MAKE): New internal variable, can be used instead of
+ $(MAKE) in recipes requiring a recursive call to make, but which
+ are not intended to be executed by "make -n".
+ (.log.trs): New suffix rule, to recover from deletion of `.trs'
+ files.
+ ($(TEST_SUITE_LOG)): Almost completely rewritten to follow the
+ new API of "test logs in `.log' files, test metadata in `.trs'
+ files". It goes to some length to work correctly in face of
+ unreadable or missing `.log' and `.trs' files, and to error out
+ with proper error messages when this is not possible.
+ [%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale"
+ `.trs' files (in addition to `.log files) before remaking the
+ $(TEST_SUITE_LOG).
+ (recheck, recheck-html): Look for the `:recheck:' field in the
+ `.trs' files, not in the `.log' files.
+ * lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust
+ the call to the test driver, in particularly passing the new
+ option `--trs-file'.
+ [%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise.
+ * lib/tap-driver ($USAGE): Adjust the help screen.
+ (Getopt::Long::GetOptions): Handle the `--trs-file' option,
+ through the use of ...
+ ($trs_file): ... this new global variable.
+ (finish): Write metadata for the test run to `$trs_file' rather
+ then to `$log_file', through the use of ...
+ (write_test_results): ... this new function.
+ * lib/test-driver (print_usage): Update the help screen.
+ (Option parsing): Handle the `--trs-file' option, through the
+ use of ...
+ ($resfile): ... this new global variable.
+ (Main code): Write metadata for the test run to `$trsfile' rather
+ than to `$logfile'.
+ Minor related adjustments to comments.
+ * tests/.gitignore: Ignore `*.trs' files.
+ * tests/parallel-tests-unreadable-log.test: Moved ...
+ * tests/parallel-tests-unreadable.test: ... to this, and extended
+ to also check the semantics for unreadable `.trs' files.
+ * tests/test-driver-end-metadata.test: Deleted as obsolete.
+ * tests/test-driver-metadata-no-leading-space.test: Likewise.
+ * tests/test-driver-global-log.test: Renamed ...
+ * tests/test-metadata-global-log.test: ... to this, and modified
+ as to verify the new APIs and semantics.
+ * tests/test-driver-recheck: Renamed ...
+ * tests/test-metadata-recheck.test: ... to this, and modified
+ likewise.
+ * tests/parallel-tests-once.test: New test.
+ * tests/parallel-tests-make-n.test: Likewise.
+ * test-metadata-results.test: Likewise.
+ * test-missing.test: Likewise.
+ * test-missing2.test: Likewise.
+ * test-trs-basic.test: Likewise.
+ * test-trs-recover.test: Likewise.
+ * test-trs-recover2.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
2011-07-27 Stefano Lattarini <address@hidden>
test harness: allow more metadata in log files
diff --git a/automake.in b/automake.in
index 2ad8dc2..ad58f06 100644
--- a/automake.in
+++ b/automake.in
@@ -5033,6 +5033,7 @@ sub handle_tests
return substr ($obj, 0, length ($obj) - length
($test_suffix)) . '.log'
if substr ($obj, - length ($test_suffix)) eq $test_suffix;
}
+ my $base = $obj;
$obj .= '.log';
# The "test driver" program, deputed to handle tests protocol
used by
# test scripts. By default, it's assumed that no protocol is
used,
@@ -5054,6 +5055,7 @@ sub handle_tests
$output_rules .= file_contents ('check2', new
Automake::Location,
GENERIC => 0,
OBJ => $obj,
+ BASE => $base,
SOURCE => $val,
DRIVER => $driver,
DRIVER_FLAGS => $driver_flags,
@@ -5107,6 +5109,7 @@ sub handle_tests
$output_rules .= file_contents ('check2', new
Automake::Location,
GENERIC => 1,
OBJ => '',
+ BASE => '$*',
SOURCE => '$<',
DRIVER => $driver,
DRIVER_FLAGS => $driver_flags,
@@ -5122,6 +5125,7 @@ sub handle_tests
$clean_files{'$(TEST_LOGS_TMP)'} = MOSTLY_CLEAN;
$clean_files{'$(TEST_LOGS)'} = MOSTLY_CLEAN;
+ $clean_files{'$(am__TEST_BASES:=.trs)'} = MOSTLY_CLEAN;
$clean_files{'$(TEST_SUITE_LOG)'} = MOSTLY_CLEAN;
$clean_files{'$(TEST_SUITE_HTML)'} = MOSTLY_CLEAN;
}
diff --git a/doc/automake.texi b/doc/automake.texi
index 2e5e2ba..d02e6ca 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8975,12 +8975,27 @@ order to satisfy additional portability requirements.
@vindex TEST_SUITE_LOG
@vindex TESTS
-The parallel harness operates by defining a set of @command{make} rules
-to create a summary log file, @code{TEST_SUITE_LOG}, which defaults to
address@hidden and requires a @file{.log} suffix. This file
-depends upon log files created for each single test program listed in
address@hidden, which in turn contain all output produced by the
-corresponding tests.
address@hidden @file{.log} files
address@hidden @file{.trs} files
address@hidden test metadata
+The parallel test harness operates by defining a set of @command{make}
+rules that run the test scripts listed in @code{TESTS}, and, for each
+such script, save its output in a corresponding @file{.log} file and
+its results (and other ``metadata'', @pxref{API for Custom Test Drivers})
+in a corresponding @file{.trs} (as in @b{T}est @address@hidden) file.
address@hidden We choose the `.trs' extension also because, at the time of
writing,
address@hidden it isn't already used for other significant purposes; see e.g.:
address@hidden - http://filext.com/file-extension/trs
address@hidden - http://www.file-extensions.org/search/?searchstring=trs
+The @file{.log} file will contain all the output emitted by the test on
+its standard output and its standard error. The @file{.trs} file will
+contain, among the other things, the results of the test cases run by
+the script.
+
+The parallel test harness will also create a summary log file,
address@hidden, which defaults to @file{test-suite.log} and requires
+a @file{.log} suffix. This file depends upon all the @file{.log} and
address@hidden files created for the test scripts listed in @code{TESTS}.
@vindex VERBOSE
As with the serial harness above, by default one status line is printed
@@ -8989,16 +9004,17 @@ However, standard output and standard error of the test
are redirected
to a per-test log file, so that parallel execution does not produce
intermingled output. The output from failed tests is collected in the
@file{test-suite.log} file. If the variable @samp{VERBOSE} is set, this
-file is output after the summary. For best results, the tests should be
-verbose by default now.
+file is output after the summary.
address@hidden FIXME: we should be clearer about what we mean exactly here ...
+For best results, the tests should be verbose by default now.
@vindex TEST_EXTENSIONS
@vindex TEST_LOGS
-Each log file is created when the corresponding test has completed.
-The set of log files is listed in the read-only variable
address@hidden, and defaults to @code{TESTS}, with the executable
-extension if any (@pxref{EXEEXT}), as well as any suffix listed in
address@hidden removed, and @file{.log} appended.
+Each couple of @file{.log} and @file{.trs} files is created when the
+corresponding test has completed. The set of log files is listed in
+the read-only variable @code{TEST_LOGS}, and defaults to @code{TESTS},
+with the executable extension if any (@pxref{EXEEXT}), as well as any
+suffix listed in @code{TEST_EXTENSIONS} removed, and @file{.log} appended.
@code{TEST_EXTENSIONS} defaults to @file{.test}. Results are undefined
if a test file name ends in several concatenated suffixes.
@@ -9036,7 +9052,9 @@ AM_LOG_FLAGS = -d
@noindent
will invoke @samp{$(PERL) -w foo.pl}, @samp{$(PYTHON) -v bar.py},
and @samp{./wrapper-script -d baz} to produce @file{foo.log},
address@hidden, and @file{baz.log}, respectively.
address@hidden, and @file{baz.log}, respectively. The @file{foo.trs},
address@hidden and @file{baz.trs} files will be automatically produced
+as a side-effect.
It's important to note that, differently from what we've seen for the
serial test harness (@pxref{Parallel Test Harness}), the
@@ -9110,14 +9128,15 @@ here too.
@item
@vindex RECHECK_LOGS
@cindex lazy test execution
-By default, the test harness removes all old per-test log files before
-it starts running tests to regenerate them. The variable
address@hidden contains the set of log files which are removed.
address@hidden defaults to @code{TEST_LOGS}, which means all tests
-need to be rechecked. By overriding this variable, you can choose which
-tests need to be reconsidered. For example, you can lazily rerun only
-those tests which are outdated, i.e., older than their prerequisite test
-files, by setting this variable to the empty value:
+By default, the test harness removes all old per-test @file{.log} and
address@hidden files before it starts running tests to regenerate them. The
+variable @code{RECHECK_LOGS} contains the set of @file{.log} (and, by
+implication, @file{.trs}) files which are removed. @code{RECHECK_LOGS}
+defaults to @code{TEST_LOGS}, which means all tests need to be rechecked.
+By overriding this variable, you can choose which tests need to be
+reconsidered. For example, you can lazily rerun only those tests which
+are outdated, i.e., older than their prerequisite test files, by setting
+this variable to the empty value:
@example
env RECHECK_LOGS= make -e check
@@ -9130,14 +9149,14 @@ You can ensure that all tests are rerun which have
failed or passed
unexpectedly, by running @code{make recheck} in the test directory.
This convenience target will set @code{RECHECK_LOGS} appropriately
before invoking the main test harness. The @code{recheck-html} target
-does the same as @code{recheck} but again converts the resulting log
-file in HTML format, like the @code{check-html} target.
+does the same as @code{recheck} but again converts the resulting
address@hidden file in HTML format, like the @code{check-html} target.
@end itemize
In order to guarantee an ordering between tests even with @code{make
address@hidden, dependencies between the corresponding log files may be
-specified through usual @command{make} dependencies. For example, the
-following snippet lets the test named @file{foo-execute.test} depend
address@hidden, dependencies between the corresponding @file{.log} files
+may be specified through usual @command{make} dependencies. For example,
+the following snippet lets the test named @file{foo-execute.test} depend
upon completion of the test @file{foo-compile.test}:
@example
@@ -9201,12 +9220,12 @@ the package authors to use third-party custom test
drivers, in case the
default ones are inadequate for their purposes, or do not support their
testing protocol of choice.
-A custom test driver is expected to properly run the test programs
-passed to it, including the command-line arguments passed to it,
-to analyze their execution and outcome, to create the @file{.log}
-files associated to these test runs, and to display the test results
-on the console. It is responsibility of the author of the test driver
-to ensure that it implements all the above steps meaningfully and
+A custom test driver is expected to properly run the test programs passed
+to it (including the command-line arguments passed to those programs, if
+any), to analyze their execution and outcome, to create the @file{.log}
+and @file{.trs} files associated to these test runs, and to display the test
+results on the console. It is responsibility of the author of the test
+driver to ensure that it implements all the above steps meaningfully and
correctly; Automake isn't and can't be of any help here. On the other
hand, the Automake-provided code for testsuite summary generation offers
support for test drivers allowing several test results per test script,
@@ -9233,8 +9252,8 @@ runtime through the redefinition of @code{TESTS} or
@code{TEST_LOGS};
@item
concurrency through the use of @command{make}'s option @option{-j};
@item
-per-test @file{.log} files, and generation of a summary @file{.log}
-file from them;
+per-test @file{.log} and @file{.trs} files, and generation of a summary
address@hidden file from them;
@item
@code{recheck} target, @code{RECHECK_LOGS} variable, and lazy reruns
of tests;
@@ -9337,10 +9356,14 @@ The name of the test, with VPATH prefix (if any)
removed. This can have a
suffix and a directory component (as in e.g., @file{sub/foo.test}), and is
mostly meant to be used in console reports about testsuite advancements and
results (@pxref{Testsuite progress output}).
address@hidden address@hidden
-The log file the test driver must create. If it has a directory component
-(as in e.g., @file{sub/foo.log}), the test harness will ensure that such
-directory exist @emph{before} the test driver is called.
address@hidden address@hidden@var{PATH}.log}
+The @file{.log} file the test driver must create. If it has a directory
+component (as in e.g., @file{sub/foo.log}), the test harness will ensure
+that such directory exist @emph{before} the test driver is called.
address@hidden address@hidden@var{PATH}.trs}
+The @file{.trs} file the test driver must create. If it has a directory
+component (as in e.g., @file{sub/foo.trs}), the test harness will ensure
+that such directory exist @emph{before} the test driver is called.
@item address@hidden|address@hidden
Whether the console output should be colorized or not (@pxref{Simple
tests and color-tests}, to learn when this option gets activated and
@@ -9370,14 +9393,24 @@ consistency and a more pleasant user experience.
@node Log files generation and test results recording
@subsubsection Log files generation and test results recording
-The test driver must correctly generate the file specified by the
address@hidden option (of course even when the tested program
-fails or crashes). This file, while being quite free-form, @emph{must}
-register some metadata through the use of custom reStructuredText fields,
-in order to work with the @option{parallel-tests} harness and take
-advantage of its features. Note that such fields will be recognized only
-when used @emph{exactly} at the beginning of a line -- not even leading
-whitespace is permitted.
+The test driver must correctly generate the files specified by the
address@hidden and @option{--trs-file} option (even when the tested
+program fails or crashes).
+
+The @file{.log} file should ideally contain all the output produced by the
+tested program, plus optionally other information that might facilitate
+debugging or analysis of bug reports. Apart from that, its format is
+basically free; the only limitation being that it must parse validly as
+reStructuredText if the @file{.log} -> @file{.html} conversion is to be
+supported.
+
+The @file{.trs} file is used to register some metadata through the use
+of custom reStructuredText fields. This metadata is expected to be
+employed in various ways by the parallel test harness; for example, to
+count the test results when printing the testsuite summary, or to decide
+which tests to re-run upon @command{make reheck}. Unrecognized metadata
+in a @file{.trs} file is currently ignored by the harness, but this might
+change in the future. The list of currently recognized metadata follows.
@table @code
@@ -9388,10 +9421,11 @@ whitespace is permitted.
@cindex Test case result, registering
@cindex @code{:test-result:}
@cindex reStructuredText field, @code{:test-result:}
-The test driver must use this filed to register the results of @emph{each}
+The test driver must use this field to register the results of @emph{each}
test case run by a test script file. Several @code{:test-result:} fields
-can be present in the same log file; this is done in order to support test
-protocols that allow a single test script to run more test cases.
+can be present in the same @file{.trs} file; this is done in order to
+support test protocols that allow a single test script to run more test
+cases.
@c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
The only recognized test results are currently @code{PASS}, @code{XFAIL},
@@ -9401,16 +9435,16 @@ text holding the name and/or a brief description of the
corresponding
test; the @option{parallel-tests} harness will ignore such extra text when
generating @file{test-suite.log} and preparing the testsuite summary.
address@hidden Keep in sync with 'test-driver-recheck.test'.
address@hidden Keep in sync with 'test-metadata-recheck.test'.
@item @code{:recheck:}
@cindex :recheck:
@cindex reStructuredText field, @code{:recheck:}
If this field is present and defined to @code{no}, then the corresponding
test script will @emph{not} be run upon a @command{make recheck}. What
-happens when two or more @code{:recheck:} fields are present in a
address@hidden file is undefined behaviour.
+happens when two or more @code{:recheck:} fields are present in the same
address@hidden file is undefined behaviour.
address@hidden Keep in sync with 'test-driver-global-log.test'.
address@hidden Keep in sync with 'test-metadata-global-log.test'.
@item @code{:copy-in-global-log:}
@cindex :copy-in-global-log:
@cindex reStructuredText field, @code{:copy-in-global-log:}
@@ -9420,43 +9454,30 @@ of the @file{.log} file will @emph{not} be copied into
the global
it can be useful in debugging and analysis of bug report, it can also be
just a waste of space in normal situations, e.g., when a test script is
successful. What happens when two or more @code{:copy-in-global-log:}
-fields are present in a @file{.log} file is undefined behaviour.
-
address@hidden Keep in sync with 'test-driver-end-metadata.test'.
address@hidden @code{:end-metadata:}
address@hidden :end-metadata:
address@hidden reStructuredText field, @code{:end-metadata:}
-This special field causes all the following text in the @file{.log}
-file not to be scanned by the testsuite harness in search for other
-test metadata. This feature is used internally by the default
address@hidden driver, and can be useful to e.g., avoid
-spurious testsuite results.
+fields are present in the same @file{.trs} file is undefined behaviour.
@end table
@noindent
-Let's see a small example. Assume a @file{.log} file contains the
-following lines (in this order, but possibly interleaved with other
-``non-metadata'' lines):
+Let's see a small example. Assume a @file{.trs} file contains the
+following lines:
@example
:test-result: PASS server starts
+:global-log-copy: no
:test-result: PASS HTTP/1.1 request
:test-result: FAIL HTTP/1.0 request
:recheck: yes
:test-result: SKIP HTTPS request (TLS library wasn't available)
:test-result: PASS server stops
-:global-log-copy: no
-:metadata-end:
-:test-result: XPASS (this shouldn't be parsed as a test result)
@end example
@noindent
Then the corresponding test script will be re-run by @command{make check},
will contribute with @emph{five} test results to the testsuite summary
(three of these tests being successful, one failed, and one skipped), and
-the @file{.log} file content will @emph{not} be copied in the global log
-file @file{test-suite.log}.
+the content of the corresponding @file{.log} file will @emph{not} be
+copied in the global log file @file{test-suite.log}.
@node Testsuite progress output
@subsubsection Testsuite progress output
@@ -9492,11 +9513,9 @@ implementations' output might be a good idea too.
If HTML testsuite output (with @code{check-html}) is to be supported,
the generated @file{.log} files must contain syntactically valid
-reStructuredText (among the other things, this means that every
address@hidden:test-result:} line must be followed by a blank line). If this
-is not the case, the HTML generation will not work, although all the
-other functionalities of the Automake testsuite harness should remain
-untouched, and continue to work correctly.
+reStructuredText. If this is not the case, the HTML generation will not
+work, although all the other functionalities of the Automake testsuite
+harness should remain untouched, and continue to work correctly.
@node Using the TAP test protocol
@section Using the TAP test protocol
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 9b45aa2..78a7e4a 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -119,10 +119,8 @@ am__rst_title = sed 's/.*/ &
/;h;s/./=/g;p;x;p;g;p;s/.*//'
# pass "-e" to $(SHELL), and POSIX 2008 even requires this. Work around it
# by disabling -e (using the XSI extension "set +e") if it's set.
am__sh_e_setup = case $$- in *e*) set +e;; esac
-# Default flags passed to all log compiler wrappers.
-am__test_driver_flags = \
- --test-name "$$f" \
- --log-file '$@' \
+# Default flags passed to test drivers.
+am__common_driver_flags = \
--color-tests "$$am__color_tests" \
--enable-hard-errors "$$am__enable_hard_errors" \
--expect-failure "$$am__expect_failure"
@@ -156,6 +154,14 @@ case " $(XFAIL_TESTS) " in \
am__expect_failure=no;; \
esac; \
$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
+# The names of the tests, with any registered extension removed. Or
+# equivalently, the names of the test logs, with the `.log' extension
+# renoved. This honours runtime overriding of TESTS and TEST_LOGS.
+am__TEST_BASES = $(TEST_LOGS:.log=)
+# This can be used instead of $(MAKE) in recipes requiring a recursive call
+# to make, but which are not intended to be executed by "make -n". See the
+# GNU make manual for more details.
+am__stealth_MAKE = $(MAKE)
RECHECK_LOGS = $(TEST_LOGS)
AM_RECURSIVE_TARGETS = check check-html recheck recheck-html
TEST_SUITE_LOG = test-suite.log
@@ -289,7 +295,7 @@ EXTRA_DIST = $(TESTS)
all: all-am
.SUFFIXES:
-.SUFFIXES: .html .log .pl
+.SUFFIXES: .html .log .pl .trs
$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(am__configure_deps)
@for dep in $?; do \
case '$(am__configure_deps)' in \
@@ -329,43 +335,46 @@ CTAGS:
cscope cscopelist:
+# Recover from deleted `.trs' file; this should ensure that
+# "rm -f foo.log; make foo.trs" re-run `foo.test', and re-create
+# both `foo.log' and `foo.trs'.
+.log.trs:
+ rm -f $< $@ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $<
+
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
- tab=' '; \
- nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
- IFS=" $$tab$$nl"; oIFS=$$IFS; \
- ws="[ $$tab]"; \
- list='$(TEST_LOGS)'; \
- list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
- results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
- results2=''; \
- exec 5<&0; \
- for f in $$list2; do \
- exec 0<$$f; \
- IFS=''; \
- while read line; do \
- IFS=$$oIFS; \
- case $$line in \
- :end-metadata:*) \
- break;; \
- :test-result:*) \
- res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
- results2="$$results2$$nl$$res";; \
- esac; \
- IFS=''; \
+ bases='$(am__TEST_BASES)'; \
+ ws='[ ]'; \
+ redo_bases=`for b in $$bases; do \
+ test -f $$b.trs && test -r $$b.trs \
+ && test -f $$b.log && test -r $$b.log \
+ || echo $$b; \
+ done`; \
+ if test -n "$$redo_bases"; then \
+ redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \
+ redo_results=`for i in $$redo_bases; do echo $$i.trs; done`; \
+ rm -f $$redo_logs && rm -f $$redo_results \
+ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $$redo_logs; \
+ fi; \
+ st=0; \
+ for b in $$redo_bases; do \
+ for e in trs log; do \
+ if test ! -f $$b.$$e || test ! -r $$b.$$e; then \
+ echo "fatal: making $@: failed to create $$b.$$e" >&2; \
+ st=1; \
+ fi; \
done; \
- IFS=$$oIFS; \
- exec 0<&5; \
done; \
- exec 5<&-; \
- results=`echo "$$results1" && echo "$$results2"`; \
- all=`echo "$$results" | sed '/^$$/d' | wc -l | sed -e 's/^[
]*//'`; \
- fail=`echo "$$results" | grep -c '^FAIL'`; \
- pass=`echo "$$results" | grep -c '^PASS'`; \
- skip=`echo "$$results" | grep -c '^SKIP'`; \
- xfail=`echo "$$results" | grep -c '^XFAIL'`; \
- xpass=`echo "$$results" | grep -c '^XPASS'`; \
- error=`echo "$$results" | grep -c '^ERROR'`; \
+ test $$st -eq 0 || exit 1; \
+ results=`for b in $$bases; do echo $$b.trs; done`; \
+ test -n "$$results" || results=/dev/null; \
+ all=` grep "^$$ws*:test-result:" $$results | wc -l`; \
+ pass=` grep "^$$ws*:test-result:$$ws*PASS" $$results | wc -l`; \
+ fail=` grep "^$$ws*:test-result:$$ws*FAIL" $$results | wc -l`; \
+ skip=` grep "^$$ws*:test-result:$$ws*SKIP" $$results | wc -l`; \
+ xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \
+ xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \
+ error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \
if test `expr $$fail + $$xpass + $$error` -eq 0; then \
success=true; \
else \
@@ -407,10 +416,15 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo; \
echo ".. contents:: :depth: 2"; \
echo; \
- for f in $$list; do \
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
- || { echo && cat $$f; } \
+ for b in $$bases; do \
+ if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null; then :; \
+ elif test ! -r $$b.log; then \
+ echo "ERROR: cannot read $$b.log" >&2; \
+ echo && echo "WARNING: could not read $$b.log!"; \
+ else \
+ echo && cat $$b.log; \
+ fi; \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
@@ -437,6 +451,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
# Run all the tests.
check-TESTS:
@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
+ @list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
@list='$(TEST_LOGS)'; \
list=`for f in $$list; do \
@@ -471,19 +486,19 @@ check-html:
recheck recheck-html:
@ws='[ ]'; \
target=`echo $@ | sed 's,^re,,'`; \
- list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
- || echo $$f; \
+ bases='$(am__TEST_BASES)'; \
+ list=`for b in $$bases; do \
+ test ! -f $$b.trs && test ! -f $$b.log && continue; \
+ grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null 2>&1 || echo $$b.log; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS)
TEST_LOGS="'"$$list"'"'
.pl.log:
- @p='$<'; $(am__check_pre) \
- $(PL_LOG_DRIVER) $(am__test_driver_flags) $(AM_PL_LOG_DRIVER_FLAGS)
$(PL_LOG_DRIVER_FLAGS) -- \
- $(PL_LOG_COMPILE) "$$tst" $(AM_TESTS_FD_REDIRECT)
+ @p='$<'; $(am__check_pre) $(PL_LOG_DRIVER) --test-name "$$f" \
+ --log-file '$*.log' --trs-file '$*.trs' \
+ $(am__common_driver_flags) $(AM_PL_LOG_DRIVER_FLAGS)
$(PL_LOG_DRIVER_FLAGS) -- $(PL_LOG_COMPILE) "$$tst" \
+ $(AM_TESTS_FD_REDIRECT)
distdir: $(DISTFILES)
@srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
@@ -544,6 +559,7 @@ mostlyclean-generic:
-test -z "$(TEST_LOGS_TMP)" || rm -f $(TEST_LOGS_TMP)
-test -z "$(TEST_SUITE_HTML)" || rm -f $(TEST_SUITE_HTML)
-test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
+ -test -z "$(am__TEST_BASES:=.trs)" || rm -f $(am__TEST_BASES:=.trs)
clean-generic:
diff --git a/lib/am/check.am b/lib/am/check.am
index 272fc0a..c406e9f 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -83,10 +83,8 @@ am__rst_title = sed 's/.*/ &
/;h;s/./=/g;p;x;p;g;p;s/.*//'
# by disabling -e (using the XSI extension "set +e") if it's set.
am__sh_e_setup = case $$- in *e*) set +e;; esac
-# Default flags passed to all log compiler wrappers.
-am__test_driver_flags = \
- --test-name "$$f" \
- --log-file '$@' \
+# Default flags passed to test drivers.
+am__common_driver_flags = \
--color-tests "$$am__color_tests" \
--enable-hard-errors "$$am__enable_hard_errors" \
--expect-failure "$$am__expect_failure"
@@ -124,64 +122,71 @@ case " $(XFAIL_TESTS) " in
\
esac; \
$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
+# The names of the tests, with any registered extension removed. Or
+# equivalently, the names of the test logs, with the `.log' extension
+# renoved. This honours runtime overriding of TESTS and TEST_LOGS.
+am__TEST_BASES = $(TEST_LOGS:.log=)
+
+# This can be used instead of $(MAKE) in recipes requiring a recursive call
+# to make, but which are not intended to be executed by "make -n". See the
+# GNU make manual for more details.
+am__stealth_MAKE = $(MAKE)
+
+# Recover from deleted `.trs' file; this should ensure that
+# "rm -f foo.log; make foo.trs" re-run `foo.test', and re-create
+# both `foo.log' and `foo.trs'.
+.log.trs:
+ rm -f $< $@ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $<
+
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
- tab=' '; \
-## The following line will define the `$nl' shell variable to a literal
-## newline. Idiom taken from the Autoconf manual.
- nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
-## Sanitize internal field separator for the shell: "space tab newline".
- IFS=" $$tab$$nl"; oIFS=$$IFS; \
-## Regular expression to match a whitespace.
- ws="[ $$tab]"; \
-## All test logs.
- list='$(TEST_LOGS)'; \
-## Readable test logs.
- list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
-## Each unreadable test log counts as a failed test.
- results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
-## Now we're going to extract the outcome of all the testcases from the
-## test logs.
- results2=''; \
-## Can't do the redirections directly in the "for" loop below, otherwise
-## some shells would run the loop body in a subshell.
- exec 5<&0; \
- for f in $$list2; do \
- exec 0<$$f; \
-## We want to support special reStructuredText fields only when they're
-## placed at the beginning of a line (as documented in the manual), so
-## we have to temporarily clear the $IFS variable to prevent automatic
-## whitespace stripping by the shell.
- IFS=''; \
- while read line; do \
- IFS=$$oIFS; \
- case $$line in \
-## Special field that tells not to scan the rest of the log file.
- :end-metadata:*) \
- break;; \
-## A test test result -- register it.
- :test-result:*) \
- res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
- results2="$$results2$$nl$$res";; \
- esac; \
-## For the next loop.
- IFS=''; \
+ bases='$(am__TEST_BASES)'; \
+ ws='[ ]'; \
+## We need to ensures that all the required `.trs' and `.log' files will
+## be present and readable. The direct dependencies of $(TEST_SUITE_LOG)
+## only ensure that all the `.log' files exists; they don't ensure that
+## the `.log' files are readable, and worse, they don't ensure that the
+## `.trs' files even exist.
+ redo_bases=`for b in $$bases; do \
+ test -f $$b.trs && test -r $$b.trs \
+ && test -f $$b.log && test -r $$b.log \
+ || echo $$b; \
+ done`; \
+ if test -n "$$redo_bases"; then \
+## Uh-oh, either some `.log' files were unreadable, or some `.trs' files
+## were missing (or unreadable). We need to re-run the corresponding
+## tests in order to re-create them.
+ redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \
+ redo_results=`for i in $$redo_bases; do echo $$i.trs; done`; \
+ rm -f $$redo_logs && rm -f $$redo_results \
+ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $$redo_logs; \
+ fi; \
+## Sanity check: each unreadable or non-existent test result file should
+## has been properly remade at this point, as should the corresponding log
+## file.
+ st=0; \
+ for b in $$redo_bases; do \
+ for e in trs log; do \
+ if test ! -f $$b.$$e || test ! -r $$b.$$e; then \
+ echo "fatal: making $@: failed to create $$b.$$e" >&2; \
+ st=1; \
+ fi; \
done; \
-## Restore original standard input and internal field separator.
- IFS=$$oIFS; \
- exec 0<&5; \
done; \
-## We don't need this anymore.
- exec 5<&-; \
-## Prepare data for the test suite summary.
- results=`echo "$$results1" && echo "$$results2"`; \
- all=`echo "$$results" | sed '/^$$/d' | wc -l | sed -e 's/^[
]*//'`; \
- fail=`echo "$$results" | grep -c '^FAIL'`; \
- pass=`echo "$$results" | grep -c '^PASS'`; \
- skip=`echo "$$results" | grep -c '^SKIP'`; \
- xfail=`echo "$$results" | grep -c '^XFAIL'`; \
- xpass=`echo "$$results" | grep -c '^XPASS'`; \
- error=`echo "$$results" | grep -c '^ERROR'`; \
+ test $$st -eq 0 || exit 1; \
+## List of test result files.
+ results=`for b in $$bases; do echo $$b.trs; done`; \
+ test -n "$$results" || results=/dev/null; \
+## Prepare data for the test suite summary. These do not take into account
+## unreadable test results, but they'll be appropriately updated later if
+## needed.
+ all=` grep "^$$ws*:test-result:" $$results | wc -l`; \
+ pass=` grep "^$$ws*:test-result:$$ws*PASS" $$results | wc -l`; \
+ fail=` grep "^$$ws*:test-result:$$ws*FAIL" $$results | wc -l`; \
+ skip=` grep "^$$ws*:test-result:$$ws*SKIP" $$results | wc -l`; \
+ xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \
+ xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \
+ error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \
## Whether the testsuite was successful or not.
if test `expr $$fail + $$xpass + $$error` -eq 0; then \
success=true; \
@@ -233,11 +238,19 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo; \
echo ".. contents:: :depth: 2"; \
echo; \
- for f in $$list; do \
-## FIXME: two forks per test -- this is horrendously inefficient!
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
- || { echo && cat $$f; } \
+ for b in $$bases; do \
+## FIXME: one fork per test -- this is horrendously inefficient!
+ if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null; then :; \
+## If we cannot read the .log file of a test ...
+ elif test ! -r $$b.log; then \
+## ... print an error message to the console ...
+ echo "ERROR: cannot read $$b.log" >&2; \
+## ... and a warning in the `test-suite.log' file.
+ echo && echo "WARNING: could not read $$b.log!"; \
+ else \
+ echo && cat $$b.log; \
+ fi; \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
@@ -272,6 +285,7 @@ RECHECK_LOGS = $(TEST_LOGS)
check-TESTS:
## Expand $(RECHECK_LOGS) only once, to avoid exceeding line length limits.
@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
+ @list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list
## We always have to remove TEST_SUITE_LOG, to ensure its rule is run
## in any case even in lazy mode: otherwise, if no test needs rerunning,
## or a prior run plus reruns all happen within the same timestamp
@@ -335,13 +349,14 @@ AM_RECURSIVE_TARGETS += check-html
recheck recheck-html:
@ws='[ ]'; \
target=`echo $@ | sed 's,^re,,'`; \
- list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
-## FIXME: two forks per test -- this is horrendously inefficient!
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
- || echo $$f; \
+ bases='$(am__TEST_BASES)'; \
+ list=`for b in $$bases; do \
+## Skip tests that haven't been run, but recover gracefully from deleted
+## `.trs' files.
+ test ! -f $$b.trs && test ! -f $$b.log && continue; \
+## FIXME: one fork per test -- this is horrendously inefficient!
+ grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null 2>&1 || echo $$b.log; \
done | tr '\012\015' ' '`; \
## This apparently useless munging helps to avoid a nasty bug (a
## segmentation fault!) on Solaris XPG4 make.
diff --git a/lib/am/check2.am b/lib/am/check2.am
index f4fb3c9..ad0a4aa 100644
--- a/lib/am/check2.am
+++ b/lib/am/check2.am
@@ -14,12 +14,13 @@
## You should have received a copy of the GNU General Public License
## along with this program. If not, see <http://www.gnu.org/licenses/>.
-## From a test file to a log file.
+## From a test file to a .log and .trs file.
?GENERIC?%EXT%.log:
?!GENERIC?%OBJ%: %SOURCE%
- @p='%SOURCE%'; $(am__check_pre) \
- %DRIVER% $(am__test_driver_flags) %DRIVER_FLAGS% -- \
- %COMPILE% "$$tst" $(AM_TESTS_FD_REDIRECT)
+ @p='%SOURCE%'; $(am__check_pre) %DRIVER% --test-name "$$f" \
+ --log-file '%BASE%.log' --trs-file '%BASE%.trs' \
+ $(am__common_driver_flags) %DRIVER_FLAGS% -- %COMPILE% "$$tst" \
+ $(AM_TESTS_FD_REDIRECT)
## If no programs are built in this package, then this rule is removed
## at automake time. Otherwise, %am__EXEEXT% expands to a configure time
@@ -27,7 +28,8 @@
## conflict with the previous one.
if %am__EXEEXT%
?GENERIC?%EXT%$(EXEEXT).log:
- @p='%SOURCE%'; $(am__check_pre) \
- %DRIVER% $(am__test_driver_flags) %DRIVER_FLAGS% -- \
- %COMPILE% "$$tst" $(AM_TESTS_FD_REDIRECT)
+ @p='%SOURCE%'; $(am__check_pre) %DRIVER% --test-name "$$f" \
+ --log-file '%BASE%.log' --trs-file '%BASE%.trs' \
+ $(am__common_driver_flags) %DRIVER_FLAGS% -- %COMPILE% "$$tst" \
+ $(AM_TESTS_FD_REDIRECT)
endif %am__EXEEXT%
diff --git a/lib/tap-driver b/lib/tap-driver
index 9075d35..8237fec 100755
--- a/lib/tap-driver
+++ b/lib/tap-driver
@@ -15,7 +15,7 @@ my $ME = "tap-driver";
my $USAGE = <<'END';
Usage:
- tap-driver [--help|--version] --test-name=NAME --log-file=PATH
+ tap-driver --test-name=NAME --log-file=PATH --trs-file=PATH
[--expect-failure={yes|no}] [--color-tests={yes|no}]
[--enable-hard-errors={yes|no}] [--merge|--no-merge]
[--ignore-exit] [--comments|--no-comments] [--] TEST-COMMAND
@@ -62,12 +62,14 @@ my %cfg = (
my $test_script_name = undef;
my $log_file = undef;
+my $trs_file = undef;
Getopt::Long::GetOptions (
'help' => sub { print $HELP; exit 0; },
'version' => sub { print "$ME $VERSION"; exit 0; },
'test-name=s' => \$test_script_name,
'log-file=s' => \$log_file,
+ 'res-file=s' => \$trs_file,
'color-tests=s' => \&bool_opt,
'expect-failure=s' => \&bool_opt,
'enable-hard-errors=s' => \&bool_opt,
@@ -164,6 +166,20 @@ TEST_RESULTS :
}
+sub write_test_results ()
+{
+ open RES, ">", $trs_file or die "opening $trs_file: $!\n";
+ print RES ":recheck: " .
+ (must_recheck ? "yes" : "no") . "\n";
+ print RES ":copy-in-global-log: " .
+ (copy_in_global_log ? "yes" : "no") . "\n";
+ foreach my $result (get_test_results)
+ {
+ print RES ":test-result: $result\n";
+ }
+ close RES or die "closing $trs_file: $!\n";
+}
+
sub start (@)
{
# Redirect stderr and stdout to a temporary log file. Save the
@@ -179,35 +195,15 @@ sub start (@)
sub finish ()
{
- open LOG, ">", $log_file or die "opening $log_file: $!\n";
-
- # Some extra trailing empty lines outputted below are required
- # to support reStructuredText -> HTML conversion.
+ write_test_results;
+ open LOG, ">", $log_file or die "opening $log_file: $!\n";
my $global_result = get_global_test_result;
my $global_result_line = "$global_result: $test_script_name";
print LOG "$global_result_line\n";
print LOG "=" x length ($global_result_line) . "\n";
print LOG "\n";
- print LOG ":recheck: " . (must_recheck ? "yes" : "no") . "\n";
- print LOG "\n";
-
- print LOG ":copy-in-global-log: " .
- (copy_in_global_log ? "yes" : "no") . "\n";
- print LOG "\n";
-
- foreach (get_test_results)
- {
- print LOG ":test-result: $_\n";
- print LOG "\n";
- }
-
- # So that the output from the test script won't be parsed for
- # testsuite metadata.
- print LOG ":end-metadata:\n";
- print LOG "\n";
-
close TMP or die "closing $log_file-t: $!\n";
# FIXME: remove this hack!
my $test_output = `cat $log_file-t && rm -f $log_file-t`;
diff --git a/lib/test-driver b/lib/test-driver
index 45b3afe..472ba49 100755
--- a/lib/test-driver
+++ b/lib/test-driver
@@ -1,7 +1,7 @@
#! /bin/sh
# test-driver - basic driver script for the `parallel-tests' mode.
-scriptversion=2011-07-27.12; # UTC
+scriptversion=2011-08-01.21; # UTC
# Copyright (C) 2011 Free Software Foundation, Inc.
#
@@ -48,7 +48,7 @@ print_usage ()
{
cat <<END
Usage:
- test-driver [--help|--version] --test-name=NAME --log-file=PATH
+ test-driver --test-name=NAME --log-file=PATH --trs-file=PATH
[--expect-failure={yes|no}] [--color-tests={yes|no}]
[--enable-hard-errors={yes|no}] [--] TEST-SCRIPT
The \`--test-name' and \`--log-file' options are mandatory.
@@ -59,9 +59,10 @@ END
rst_section () { sed 'p;s/./=/g;p;g'; }
# TODO: better error handling in option parsing (in particular, ensure
-# TODO: $logfile and $test_name are defined).
+# TODO: $logfile, $trsfile and $test_name are defined).
test_name= # Used for reporting.
-logfile= # Where to save the result and output of the test script.
+logfile= # Where to save the output of the test script.
+trsfile= # Where to save the result(s) the test script.
expect_failure=no
color_tests=no
enable_hard_errors=yes
@@ -71,6 +72,7 @@ while test $# -gt 0; do
--version) echo "test-driver $scriptversion"; exit $?;;
--test-name) test_name=$2; shift;;
--log-file) logfile=$2; shift;;
+ --trs-file) trsfile=$2; shift;;
--color-tests) color_tests=$2; shift;;
--expect-failure) expect_failure=$2; shift;;
--enable-hard-errors) enable_hard_errors=$2; shift;;
@@ -118,24 +120,14 @@ esac
# Report outcome to console.
echo "${col}${res}${std}: $test_name"
+# Register the test result, and other relevant metadata.
+echo ":test-result: $res (exit status: $estatus)" > $trsfile
+echo ":recheck: $recheck" >> $trsfile
+echo ":copy-in-global-log: $gcopy" >> $trsfile
+
# Now write log file.
{
echo "$res: $test_name (exit: $estatus)" | rst_section
- # Blank lines required for having the output be valid RST.
- echo
- echo ":test-result: $res (exit status: $estatus)"
- echo
- echo ":recheck: $recheck"
- echo
- echo ":copy-in-global-log: $gcopy"
- echo
- # So that the output from the test script won't be parsed for
- # testsuite metadata.
- echo ":end-metadata:"
- echo
- # Use a reStructuredText transition to better separate the test
- # metadata the test registered output.
- echo ------------
echo
cat $tmpfile
} > $logfile
diff --git a/tests/.gitignore b/tests/.gitignore
index 026e261..0d5a032 100644
--- a/tests/.gitignore
+++ b/tests/.gitignore
@@ -5,6 +5,7 @@ instspc-tests.am
parallel-tests.am
*.dir
*.log
+*.trs
*.log-t
*-p.test
instspc-*-build.test
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 9a0c222..104f70b 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -724,9 +724,10 @@ parallel-tests7.test \
parallel-tests8.test \
parallel-tests9.test \
parallel-tests10.test \
+parallel-tests-make-n.test \
parallel-tests-fd-redirect.test \
parallel-tests-am_tests_environment.test \
-parallel-tests-unreadable-log.test \
+parallel-tests-unreadable.test \
parallel-tests-subdir.test \
parallel-tests-interrupt.test \
parallel-tests-reset-term.test \
@@ -739,12 +740,11 @@ parallel-test-driver-install.test \
parallel-tests-no-spurious-summary.test \
parallel-tests-exit-statuses.test \
parallel-tests-console-output.test \
+parallel-tests-once.test \
testsuite-summary-color.test \
testsuite-summary-count.test \
testsuite-summary-count-many.test \
testsuite-summary-reference-log.test \
-test-driver-end-metadata.test \
-test-driver-metadata-no-leading-space.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
test-driver-custom-xfail-tests.test \
@@ -754,11 +754,17 @@ test-driver-custom-multitest-recheck2.test \
test-driver-custom-html.test \
test-driver-custom-no-html.test \
test-driver-create-log-dir.test \
-test-driver-global-log.test \
-test-driver-recheck.test \
test-driver-strip-vpath.test \
test-driver-fail.test \
test-log.test \
+test-metadata-global-log.test \
+test-metadata-recheck.test \
+test-metadata-results.test \
+test-missing.test \
+test-missing2.test \
+test-trs-basic.test \
+test-trs-recover.test \
+test-trs-recover2.test \
parse.test \
percent.test \
percent2.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 13e850c..1732555 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -124,10 +124,8 @@ am__rst_title = sed 's/.*/ &
/;h;s/./=/g;p;x;p;g;p;s/.*//'
# pass "-e" to $(SHELL), and POSIX 2008 even requires this. Work around it
# by disabling -e (using the XSI extension "set +e") if it's set.
am__sh_e_setup = case $$- in *e*) set +e;; esac
-# Default flags passed to all log compiler wrappers.
-am__test_driver_flags = \
- --test-name "$$f" \
- --log-file '$@' \
+# Default flags passed to test drivers.
+am__common_driver_flags = \
--color-tests "$$am__color_tests" \
--enable-hard-errors "$$am__enable_hard_errors" \
--expect-failure "$$am__expect_failure"
@@ -161,6 +159,14 @@ case " $(XFAIL_TESTS) " in \
am__expect_failure=no;; \
esac; \
$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
+# The names of the tests, with any registered extension removed. Or
+# equivalently, the names of the test logs, with the `.log' extension
+# renoved. This honours runtime overriding of TESTS and TEST_LOGS.
+am__TEST_BASES = $(TEST_LOGS:.log=)
+# This can be used instead of $(MAKE) in recipes requiring a recursive call
+# to make, but which are not intended to be executed by "make -n". See the
+# GNU make manual for more details.
+am__stealth_MAKE = $(MAKE)
RECHECK_LOGS = $(TEST_LOGS)
AM_RECURSIVE_TARGETS = check check-html recheck recheck-html
TEST_SUITE_LOG = test-suite.log
@@ -969,9 +975,10 @@ parallel-tests7.test \
parallel-tests8.test \
parallel-tests9.test \
parallel-tests10.test \
+parallel-tests-make-n.test \
parallel-tests-fd-redirect.test \
parallel-tests-am_tests_environment.test \
-parallel-tests-unreadable-log.test \
+parallel-tests-unreadable.test \
parallel-tests-subdir.test \
parallel-tests-interrupt.test \
parallel-tests-reset-term.test \
@@ -984,12 +991,11 @@ parallel-test-driver-install.test \
parallel-tests-no-spurious-summary.test \
parallel-tests-exit-statuses.test \
parallel-tests-console-output.test \
+parallel-tests-once.test \
testsuite-summary-color.test \
testsuite-summary-count.test \
testsuite-summary-count-many.test \
testsuite-summary-reference-log.test \
-test-driver-end-metadata.test \
-test-driver-metadata-no-leading-space.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
test-driver-custom-xfail-tests.test \
@@ -999,11 +1005,17 @@ test-driver-custom-multitest-recheck2.test \
test-driver-custom-html.test \
test-driver-custom-no-html.test \
test-driver-create-log-dir.test \
-test-driver-global-log.test \
-test-driver-recheck.test \
test-driver-strip-vpath.test \
test-driver-fail.test \
test-log.test \
+test-metadata-global-log.test \
+test-metadata-recheck.test \
+test-metadata-results.test \
+test-missing.test \
+test-missing2.test \
+test-trs-basic.test \
+test-trs-recover.test \
+test-trs-recover2.test \
parse.test \
percent.test \
percent2.test \
@@ -1396,7 +1408,7 @@ $(parallel_tests)
all: all-am
.SUFFIXES:
-.SUFFIXES: .html .log .test
+.SUFFIXES: .html .log .test .trs
$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(srcdir)/parallel-tests.am
$(srcdir)/instspc-tests.am $(am__configure_deps)
@for dep in $?; do \
case '$(am__configure_deps)' in \
@@ -1442,43 +1454,46 @@ CTAGS:
cscope cscopelist:
+# Recover from deleted `.trs' file; this should ensure that
+# "rm -f foo.log; make foo.trs" re-run `foo.test', and re-create
+# both `foo.log' and `foo.trs'.
+.log.trs:
+ rm -f $< $@ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $<
+
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
- tab=' '; \
- nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
- IFS=" $$tab$$nl"; oIFS=$$IFS; \
- ws="[ $$tab]"; \
- list='$(TEST_LOGS)'; \
- list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
- results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
- results2=''; \
- exec 5<&0; \
- for f in $$list2; do \
- exec 0<$$f; \
- IFS=''; \
- while read line; do \
- IFS=$$oIFS; \
- case $$line in \
- :end-metadata:*) \
- break;; \
- :test-result:*) \
- res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
- results2="$$results2$$nl$$res";; \
- esac; \
- IFS=''; \
+ bases='$(am__TEST_BASES)'; \
+ ws='[ ]'; \
+ redo_bases=`for b in $$bases; do \
+ test -f $$b.trs && test -r $$b.trs \
+ && test -f $$b.log && test -r $$b.log \
+ || echo $$b; \
+ done`; \
+ if test -n "$$redo_bases"; then \
+ redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \
+ redo_results=`for i in $$redo_bases; do echo $$i.trs; done`; \
+ rm -f $$redo_logs && rm -f $$redo_results \
+ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $$redo_logs; \
+ fi; \
+ st=0; \
+ for b in $$redo_bases; do \
+ for e in trs log; do \
+ if test ! -f $$b.$$e || test ! -r $$b.$$e; then \
+ echo "fatal: making $@: failed to create $$b.$$e" >&2; \
+ st=1; \
+ fi; \
done; \
- IFS=$$oIFS; \
- exec 0<&5; \
done; \
- exec 5<&-; \
- results=`echo "$$results1" && echo "$$results2"`; \
- all=`echo "$$results" | sed '/^$$/d' | wc -l | sed -e 's/^[
]*//'`; \
- fail=`echo "$$results" | grep -c '^FAIL'`; \
- pass=`echo "$$results" | grep -c '^PASS'`; \
- skip=`echo "$$results" | grep -c '^SKIP'`; \
- xfail=`echo "$$results" | grep -c '^XFAIL'`; \
- xpass=`echo "$$results" | grep -c '^XPASS'`; \
- error=`echo "$$results" | grep -c '^ERROR'`; \
+ test $$st -eq 0 || exit 1; \
+ results=`for b in $$bases; do echo $$b.trs; done`; \
+ test -n "$$results" || results=/dev/null; \
+ all=` grep "^$$ws*:test-result:" $$results | wc -l`; \
+ pass=` grep "^$$ws*:test-result:$$ws*PASS" $$results | wc -l`; \
+ fail=` grep "^$$ws*:test-result:$$ws*FAIL" $$results | wc -l`; \
+ skip=` grep "^$$ws*:test-result:$$ws*SKIP" $$results | wc -l`; \
+ xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \
+ xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \
+ error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \
if test `expr $$fail + $$xpass + $$error` -eq 0; then \
success=true; \
else \
@@ -1520,10 +1535,15 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo; \
echo ".. contents:: :depth: 2"; \
echo; \
- for f in $$list; do \
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
- || { echo && cat $$f; } \
+ for b in $$bases; do \
+ if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null; then :; \
+ elif test ! -r $$b.log; then \
+ echo "ERROR: cannot read $$b.log" >&2; \
+ echo && echo "WARNING: could not read $$b.log!"; \
+ else \
+ echo && cat $$b.log; \
+ fi; \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
@@ -1550,6 +1570,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
# Run all the tests.
check-TESTS:
@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
+ @list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
@list='$(TEST_LOGS)'; \
list=`for f in $$list; do \
@@ -1584,19 +1605,19 @@ check-html:
recheck recheck-html:
@ws='[ ]'; \
target=`echo $@ | sed 's,^re,,'`; \
- list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
- sed -e '/^:end-metadata:/,$$d' $$f | \
- grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
- || echo $$f; \
+ bases='$(am__TEST_BASES)'; \
+ list=`for b in $$bases; do \
+ test ! -f $$b.trs && test ! -f $$b.log && continue; \
+ grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null 2>&1 || echo $$b.log; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS)
TEST_LOGS="'"$$list"'"'
.test.log:
- @p='$<'; $(am__check_pre) \
- $(TEST_LOG_DRIVER) $(am__test_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS)
$(TEST_LOG_DRIVER_FLAGS) -- \
- $(TEST_LOG_COMPILE) "$$tst" $(AM_TESTS_FD_REDIRECT)
+ @p='$<'; $(am__check_pre) $(TEST_LOG_DRIVER) --test-name "$$f" \
+ --log-file '$*.log' --trs-file '$*.trs' \
+ $(am__common_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS)
$(TEST_LOG_DRIVER_FLAGS) -- $(TEST_LOG_COMPILE) "$$tst" \
+ $(AM_TESTS_FD_REDIRECT)
distdir: $(DISTFILES)
@srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
@@ -1657,6 +1678,7 @@ mostlyclean-generic:
-test -z "$(TEST_LOGS_TMP)" || rm -f $(TEST_LOGS_TMP)
-test -z "$(TEST_SUITE_HTML)" || rm -f $(TEST_SUITE_HTML)
-test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
+ -test -z "$(am__TEST_BASES:=.trs)" || rm -f $(am__TEST_BASES:=.trs)
clean-generic:
diff --git a/tests/parallel-tests-make-n.test b/tests/parallel-tests-make-n.test
new file mode 100755
index 0000000..451f782
--- /dev/null
+++ b/tests/parallel-tests-make-n.test
@@ -0,0 +1,107 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check `make -n' for testsuite-related targets, when `parallel-tests'
+# is in use.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am <<'END'
+TESTS = foo.test bar.test
+TEST_LOG_COMPILER = $(SHELL)
+END
+
+$ACLOCAL
+$AUTOMAKE -a
+$AUTOCONF
+
+./configure
+
+make_n_ ()
+{
+ st=0
+ $MAKE -n "$@" >output 2>&1 || { cat output; ls -l; Exit 1; }
+ cat output
+ # Look out for possible errors from common tools used by recipes.
+ $EGREP -i ' (exist|permission|denied|no .*(such|file))' output && Exit 1
+ $EGREP '(mv|cp|rm|cat|grep|sed|awk): ' output && Exit 1
+ :
+}
+
+: > output
+
+files='foo.log bar.log foo.trs bar.trs'
+
+for target in check recheck test-suite.log; do
+ test ! -f foo.log
+ test ! -f foo.trs
+ test ! -f bar.log
+ test ! -f bar.trs
+ test ! -f test-suite.log
+done
+
+echo 'exit 0' > foo.test
+echo 'exit 1' > bar.test
+
+$MAKE check && Exit 1
+
+chmod a-w .
+
+make_n_ clean
+test -f foo.log
+test -f foo.trs
+test -f foo.log
+test -f bar.trs
+
+echo 'echo this is bad; exit 1' > foo.test
+echo 'exit 0' > bar.test
+
+for target in check recheck test-suite.log; do
+ make_n_ $target
+ grep '^:test-result: *FAIL' bar.trs
+ grep 'this is bad' foo.log test-suite.log && Exit 1
+done
+
+chmod a-rw $files
+if test -r foo.log; then
+ : You can still read unreadable files! Skip these checks.
+else
+ for target in check recheck test-suite.log; do
+ make_n_ $target
+ for f in $files; do
+ test -f $f && test ! -r $f || Exit 1
+ done
+ done
+fi
+chmod u+r $files
+
+chmod u+w .
+rm -f foo.log bar.trs
+chmod a-w .
+for target in check recheck test-suite.log $files; do
+ make_n_ $target
+ test ! -f foo.log
+ test -f foo.trs
+ test ! -f bar.trs
+ test -f bar.log
+done
+
+:
diff --git a/tests/parallel-tests-unreadable-log.test
b/tests/parallel-tests-once.test
similarity index 59%
copy from tests/parallel-tests-unreadable-log.test
copy to tests/parallel-tests-once.test
index e8e101b..d676a12 100755
--- a/tests/parallel-tests-unreadable-log.test
+++ b/tests/parallel-tests-once.test
@@ -14,7 +14,9 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check that the testsuite driver copes well with unreadable test logs.
+# Tests shouldn't be run multiple times by a simple "make check" in a
+# clean directory. An early implementation of the `.trs' intermediate
+# files incurred a similar problem.
parallel_tests=yes
. ./defs || Exit 1
@@ -24,37 +26,23 @@ AC_OUTPUT
END
cat > Makefile.am << 'END'
-TESTS = foo.test bar.test
-XFAIL_TESTS = bar.test
+TESTS = foo.test
END
-cat > foo.test << 'END'
+cat > foo.test <<'END'
#! /bin/sh
-exit 0
+test -f foo.run && Exit 1
+: > foo.run
END
-cat > bar.test << 'END'
-#! /bin/sh
-exit 1
-END
-chmod a+x foo.test bar.test
+chmod a+x foo.test
$ACLOCAL
$AUTOCONF
$AUTOMAKE -a
-# The testsuite driver will use this variable, so ensure it sanitizes
-# it and do not allow in spurious values from the environment.
-line=PASS; export line
-
./configure
-$MAKE foo.log
-$MAKE bar.log
-chmod a-r foo.log bar.log
-test ! -r foo.log || Exit 77
-$MAKE test-suite.log >stdout && { cat stdout; Exit 1; }
-cat stdout
-grep '^# ERROR: *2$' stdout
-grep '^# ERROR: *2$' test-suite.log
+$MAKE check
+test -f foo.run # Sanity check.
:
diff --git a/tests/parallel-tests-unreadable-log.test
b/tests/parallel-tests-unreadable.test
similarity index 58%
copy from tests/parallel-tests-unreadable-log.test
copy to tests/parallel-tests-unreadable.test
index e8e101b..47edc9a 100755
--- a/tests/parallel-tests-unreadable-log.test
+++ b/tests/parallel-tests-unreadable.test
@@ -14,47 +14,62 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check that the testsuite driver copes well with unreadable test logs.
+# Check that the testsuite driver copes well with unreadable `.log'
+# and `.trs' files.
parallel_tests=yes
. ./defs || Exit 1
+: > t
+chmod a-r t && test ! -r t || skip_ "you can still read unreadable files"
+rm -f t
+
cat >> configure.in << 'END'
AC_OUTPUT
END
cat > Makefile.am << 'END'
TESTS = foo.test bar.test
-XFAIL_TESTS = bar.test
END
cat > foo.test << 'END'
#! /bin/sh
+echo foofoofoo
exit 0
END
+
cat > bar.test << 'END'
#! /bin/sh
-exit 1
+echo barbarbar
+exit 77
END
+
chmod a+x foo.test bar.test
$ACLOCAL
$AUTOCONF
$AUTOMAKE -a
-# The testsuite driver will use this variable, so ensure it sanitizes
-# it and do not allow in spurious values from the environment.
-line=PASS; export line
-
./configure
-$MAKE foo.log
-$MAKE bar.log
-chmod a-r foo.log bar.log
-test ! -r foo.log || Exit 77
-$MAKE test-suite.log >stdout && { cat stdout; Exit 1; }
-cat stdout
-grep '^# ERROR: *2$' stdout
-grep '^# ERROR: *2$' test-suite.log
+for files in \
+ 'foo.log bar.log' \
+ 'foo.trs bar.trs' \
+ 'foo.trs bar.log' \
+ 'foo.log bar.trs' \
+; do
+ $MAKE check
+ rm -f test-suite.log
+ chmod a-r $files
+ $MAKE test-suite.log || { ls -l; Exit 1; }
+ ls -l
+ grep '^foofoofoo$' foo.log
+ grep '^:test-result: PASS' foo.trs
+ grep '^barbarbar$' bar.log
+ grep '^:test-result: SKIP' bar.trs
+ grep '^SKIP: bar.test' test-suite.log
+ grep '^barbarbar$' test-suite.log
+ $EGREP 'foo\.test|foofoofoo' test-suite.log && Exit 1
+done
:
diff --git a/tests/parallel-tests9.test b/tests/parallel-tests9.test
index 68fd1d4..f86562a 100755
--- a/tests/parallel-tests9.test
+++ b/tests/parallel-tests9.test
@@ -67,15 +67,6 @@ grep 'foo\.test' stdout && Exit 1
grep '^ERROR: bar\.test$' stdout
grep '^FAIL: baz\.test$' stdout
-# If we cannot read the log file, then redo it as well.
-chmod a-r foo.log
-if test ! -r foo.log; then
- $MAKE recheck >stdout && { cat stdout; Exit 1; }
- cat stdout
- count_test_results total=3 pass=1 fail=1 skip=0 xfail=0 xpass=0 error=1
- grep '^PASS: foo\.test$' stdout || Exit 1
-fi
-
# Ensure that recheck builds check_SCRIPTS, and that
# recheck reruns nothing if check has not been run.
$MAKE clean
diff --git a/tests/test-driver-create-log-dir.test
b/tests/test-driver-create-log-dir.test
index 5713b92..17f56c4 100755
--- a/tests/test-driver-create-log-dir.test
+++ b/tests/test-driver-create-log-dir.test
@@ -40,14 +40,19 @@ check-local: $(TEST_SUITE_LOG)
test -f sub/foo.log
test -f sub/subsub/bar.log
test -f sub1/baz.log
+ test -f sub/foo.trs
+ test -f sub/subsub/bar.trs
+ test -f sub1/baz.trs
END
cat > checkdir-driver <<'END'
#! /bin/sh
set -e
+set -u
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
+ --rts-file) trs_file=$2; shift;;
--test-name|--expect-failure|--color-tests|--enable-hard-errors) shift;;
--) shift; break;;
*) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
@@ -55,11 +60,16 @@ while test $# -gt 0; do
shift
done
echo "log: $log_file" # For debugging.
+echo "trs: $trs_file" # Ditto.
case $log_file in */*);; *) exit 1;; esac
-dir=`expr "$log_file" : '\(.*\)/[^/]*'`
-echo "dir: $dir" # For debugging.
-test -d "$dir" || exit 1
-echo dummy > "$log_file"
+dir_log=`expr "$log_file" : '\(.*\)/[^/]*'`
+dir_trs=`expr "$trs_file" : '\(.*\)/[^/]*'`
+echo "dir_log: $dir_log" # For debugging.
+echo "dir_trs: $dir_trs" # Likewise.
+test x"$dir_trs" = x"$dir_log" || exit 1
+test -d "$dir_log" || exit 1
+echo dummy1 > "$log_file"
+echo dummy2 > "$trs_file"
END
chmod a+x checkdir-driver
diff --git a/tests/test-driver-custom-multitest.test
b/tests/test-driver-custom-multitest.test
index df0568c..4fb023c 100755
--- a/tests/test-driver-custom-multitest.test
+++ b/tests/test-driver-custom-multitest.test
@@ -127,6 +127,7 @@ for vpath in : false; do
# `trivial-test-driver' script is changed.
$FGREP INVALID.NAME stdout test-suite.log && Exit 1
test -f BAD.LOG && Exit 1
+ test -f BAD.TRS && Exit 1
# These log files must all have been created by the testsuite.
cat pass.log
cat fail.log
diff --git a/tests/test-driver-custom-no-html.test
b/tests/test-driver-custom-no-html.test
index 683659b..1a2b726 100755
--- a/tests/test-driver-custom-no-html.test
+++ b/tests/test-driver-custom-no-html.test
@@ -35,14 +35,14 @@ END
cat > no-rst <<'END'
#! /bin/sh
+echo ':test-result: SKIP' > foo.trs
+echo ':copy-in-global-log: yes' >> foo.trs
# The genereted log file is deliberately syntactically invalid
# reStructuredText.
cat > foo.log <<'EoL'
SKIP: FooBar
=============
-:test-result: SKIP
-
--------------
dummy title
EoL
diff --git a/tests/test-driver-custom-xfail-tests.test
b/tests/test-driver-custom-xfail-tests.test
index 4f8bb83..e68e04b 100755
--- a/tests/test-driver-custom-xfail-tests.test
+++ b/tests/test-driver-custom-xfail-tests.test
@@ -88,15 +88,17 @@ done
cat > td <<'END'
#! /bin/sh
-set -e
+set -e; set -u
test_name=INVALID
log_file=/dev/null
+trs_file=/dev/null
expect_failure=no
while test $# -gt 0; do
case $1 in
--test-name) test_name=$2; shift;;
--expect-failure) expect_failure=$2; shift;;
--log-file) log_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
# Ignored.
--color-tests) shift;;
--enable-hard-errors) shift;;
@@ -109,23 +111,22 @@ while test $# -gt 0; do
done
st=0
"$@" || st=$?
-rm -f "$log_file"
case $st,$expect_failure in
0,no)
echo "PASS: $test_name" | tee "$log_file"
- echo ":test-result: PASS" >> "$log_file"
+ echo ":test-result: PASS" > "$trs_file"
;;
1,no)
echo "FAIL: $test_name" | tee "$log_file"
- echo ":test-result: FAIL" >> "$log_file"
+ echo ":test-result: FAIL" > "$trs_file"
;;
0,yes)
echo "XPASS: $test_name" | tee "$log_file"
- echo ":test-result: XPASS" >> "$log_file"
+ echo ":test-result: XPASS" > "$trs_file"
;;
1,yes)
echo "XFAIL: $test_name" | tee "$log_file"
- echo ":test-result: XFAIL" >> "$log_file"
+ echo ":test-result: XFAIL" > "$trs_file"
;;
*)
echo "INTERNAL ERROR" >&2
@@ -141,21 +142,24 @@ $AUTOMAKE
./configure
-VERBOSE=yes $MAKE check
-test -f test-suite.log
-test -f sub1/test-suite.log
-test -f sub2/test-suite.log
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+test `grep -c '^PASS:' stdout` -eq 3
+test `grep -c '^XFAIL:' stdout` -eq 13
for dir in sub1 sub2; do
cd $dir
cp pass.test x1.test
cp x2.test pass.test
- $MAKE check && { cat test-suite.log; Exit 1; }
- cat test-suite.log
- grep '^FAIL: pass\.test$' test-suite.log
- grep '^XPASS: x1\.test$' test-suite.log
- test `grep -c '^FAIL' test-suite.log` -eq 1
- test `grep -c '^XPASS' test-suite.log` -eq 1
+ $MAKE check >stdout && { cat stdout; Exit 1; }
+ cat stdout
+ test "`cat pass.trs`" = ":test-result: FAIL"
+ test "`cat x1.trs`" = ":test-result: XPASS"
+ test "`cat x2.trs`" = ":test-result: XFAIL"
+ grep '^FAIL: pass\.test$' stdout
+ grep '^XPASS: x1\.test$' stdout
+ grep '^XFAIL: x2\.test$' stdout
+ count_test_results total=7 pass=0 xpass=1 fail=1 xfail=5 skip=0 error=0
cd ..
done
diff --git a/tests/test-driver-custom.test b/tests/test-driver-custom.test
index b6a33d2..8305047 100755
--- a/tests/test-driver-custom.test
+++ b/tests/test-driver-custom.test
@@ -59,11 +59,13 @@ log_wflags='@log_wflags@'
test_name=INVALID
log_file=BAD.log
+trs_file=BAD.trs
extra_opts=
while test $# -gt 0; do
case $1 in
--test-name) test_name=$2; shift;;
--log-file) log_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
# Ignored.
--expect-failure) shift;;
--color-tests) shift;;
@@ -79,6 +81,7 @@ while test $# -gt 0; do
done
echo "$me" "$test_name" $extra_opts > "$log_file"
+: > "$trs_file"
exec "$@"
exit 127
@@ -123,6 +126,7 @@ VERBOSE=yes $MAKE check
ls -l . sub
test ! -r BAD.log
+test ! -r BAD.trs
echo 'chk-wrapper 1.chk --am-chk --chk' > 1.exp
echo 'test-wrapper 2.test -am-test -test' > 2.exp
diff --git a/tests/test-driver-end-metadata.test
b/tests/test-driver-end-metadata.test
deleted file mode 100755
index 42e7e57..0000000
--- a/tests/test-driver-end-metadata.test
+++ /dev/null
@@ -1,120 +0,0 @@
-#! /bin/sh
-# Copyright (C) 2011 Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# Custom test drivers and parallel-tests harness: check the documented
-# semantics for inhibiting the scanning of the remaining portion of a
-# `.log' file (for what concerns metadata). Currently, this is done
-# with the use of the reStructuredText field `:end-metadata:'.
-
-parallel_tests=yes
-. ./defs || Exit 1
-
-cat >> configure.in << 'END'
-AC_OUTPUT
-END
-
-cat > Makefile.am << 'END'
-TEST_LOG_DRIVER = ./passthrough-driver
-TESTS = foo.test bar.test baz.test
-END
-
-cat > passthrough-driver <<'END'
-#!/bin/sh
-set -e; set -u;
-while test $# -gt 0; do
- case $1 in
- --log-file) log_file=$2; shift;;
- --test-name) test_name=$2; shift;;
- --expect-failure|--color-tests|--enable-hard-errors) shift;;
- --) shift; break;;
- *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
- esac
- shift
-done
-echo "$test_name: RUN"
-cp $1 $log_file
-END
-chmod a+x passthrough-driver
-
-cat > foo.test <<END
-:end-metadata:
-:test-result: FAIL
-:copy-in-global-log: no
-:recheck: no
-foo foo foo
-END
-
-cat > bar.test <<END
-:test-result: PASS
-:recheck: no
-bar bar bar
-:copy-in-global-log: no
-:end-metadata: $tab
-:test-result: ERROR
-:end-metadata:
-END
-
-# `:end-metadata:' directives in other files shouldn't influence this one.
-cat > baz.test <<END
-:test-result: PASS
-:recheck: no
-:copy-in-global-log: no
-baz baz baz
-END
-
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE
-
-./configure
-
-st=0
-$MAKE check >stdout || st=1
-# For debugging.
-cat stdout
-cat foo.log
-cat bar.log
-cat test-suite.log
-
-# Check that only the `:test-result:' in baz.test and the first one in
-# foo.test have been considered.
-test $st -eq 0
-grep '^# TOTAL: *2$' stdout
-grep '^# PASS: *2$' stdout
-for result in SKIP XPASS FAIL XFAIL ERROR; do
- grep "^# $result: *0$" stdout
-done
-
-# Check that a `:end-metadata:' directive does not prevent following text
-# to be copied into the log file.
-$FGREP 'foo foo foo' foo.log
-$FGREP 'bar bar bar' bar.log
-$FGREP 'baz baz baz' baz.log # More of a sanity check really.
-
-# Check that only the `:copy-in-global-log:' in bar.test has been
-# considered.
-$FGREP 'foo foo foo' test-suite.log
-$FGREP 'bar bar bar' test-suite.log && Exit 1
-$FGREP 'baz baz baz' test-suite.log && Exit 1
-
-# Check that only the `:recheck:' in bar.test has been
-# considered.
-$MAKE recheck >stdout || { cat stdout; Exit 1; }
-cat stdout
-grep '^foo\.test: RUN$' stdout
-grep 'ba[rz]\.test.*RUN' stdout && Exit 1
-
-:
diff --git a/tests/test-driver-metadata-no-leading-space.test
b/tests/test-driver-metadata-no-leading-space.test
deleted file mode 100755
index 4c3f026..0000000
--- a/tests/test-driver-metadata-no-leading-space.test
+++ /dev/null
@@ -1,94 +0,0 @@
-#! /bin/sh
-# Copyright (C) 2011 Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# Custom test drivers and parallel-tests harness: check that, in the
-# log files, the reStructuredText fields for metadata declaration are
-# recognized only when placed at the beginning of a line.
-
-parallel_tests=yes
-. ./defs || Exit 1
-
-cat >> configure.in << 'END'
-AC_OUTPUT
-END
-
-cat > Makefile.am << 'END'
-TEST_LOG_DRIVER = ./passthrough-driver
-TESTS = foo.test
-END
-
-cat > passthrough-driver <<'END'
-#!/bin/sh
-set -e; set -u;
-while test $# -gt 0; do
- case $1 in
- --log-file) log_file=$2; shift;;
- --test-name) test_name=$2; shift;;
- --expect-failure|--color-tests|--enable-hard-errors) shift;;
- --) shift; break;;
- *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
- esac
- shift
-done
-echo "$test_name: RUN"
-cp $1 $log_file
-END
-chmod a+x passthrough-driver
-
-cat > foo.test <<END
- $tab :test-result: FAIL
- $tab :end-metadata:
-:test-result: PASS
- $tab :copy-in-global-log: no
- $tab :recheck: no
-foo foo foo
-END
-
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE
-
-./configure
-
-st=0
-$MAKE check >stdout || st=1
-# For debugging.
-cat stdout
-cat foo.log
-cat test-suite.log
-
-# Check that only the second `:test-result:' has been processed.
-test $st -eq 0
-grep '^# TOTAL: *1$' stdout
-grep '^# PASS: *1$' stdout
-for result in SKIP XPASS FAIL XFAIL ERROR; do
- grep "^# $result: *0$" stdout
-done
-
-# Check that `:copy-in-global-log:' hasn't been processed.
-grep 'foo foo foo' test-suite.log
-
-# Check that `:recheck:' hasn't been processed.
-: > older
-$sleep
-$MAKE recheck >stdout || { cat stdout; Exit 1; }
-cat stdout
-grep '^foo\.test: RUN$' stdout
-grep 'foo foo foo' test-suite.log
-is_newest test-suite.log older
-is_newest foo.log older
-
-:
diff --git a/tests/test-driver-strip-vpath.test
b/tests/test-driver-strip-vpath.test
index f1b316c..5b21cc0 100755
--- a/tests/test-driver-strip-vpath.test
+++ b/tests/test-driver-strip-vpath.test
@@ -51,6 +51,7 @@ set -e
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
--test-name) test_name=$2; shift;;
--expect-failure|--color-tests|--enable-hard-errors) shift;;
--) shift; break;;
@@ -64,6 +65,7 @@ case $test_name in
*) exit 1;;
esac
echo dummy > "$log_file"
+echo dummy > "$trs_file"
END
chmod a+x checkstrip-driver
diff --git a/tests/parallel-tests-unreadable-log.test
b/tests/test-driver-trs-suffix-registered.test
similarity index 51%
copy from tests/parallel-tests-unreadable-log.test
copy to tests/test-driver-trs-suffix-registered.test
index e8e101b..7f6522a 100755
--- a/tests/parallel-tests-unreadable-log.test
+++ b/tests/test-driver-trs-suffix-registered.test
@@ -14,47 +14,45 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check that the testsuite driver copes well with unreadable test logs.
+# parallel-tests support: the following are registered with `.SUFFIXES':
+# - .log
+# - .trs (used by files that store test results and metadata)
+# - .test if $(TEST_EXTENSIONS) is not defined
+# - stuff in $(TEST_EXTENSIONS) otherwise
parallel_tests=yes
. ./defs || Exit 1
-cat >> configure.in << 'END'
-AC_OUTPUT
-END
+: > Makefile.am
-cat > Makefile.am << 'END'
-TESTS = foo.test bar.test
-XFAIL_TESTS = bar.test
+cat > 1.am << 'END'
+TESTS =
END
-cat > foo.test << 'END'
-#! /bin/sh
-exit 0
+cat > 2.am << 'END'
+TEST_EXTENSIONS = .SH .abcdef
+TESTS =
END
-cat > bar.test << 'END'
-#! /bin/sh
-exit 1
-END
-chmod a+x foo.test bar.test
+
+: > test-driver
$ACLOCAL
-$AUTOCONF
-$AUTOMAKE -a
-
-# The testsuite driver will use this variable, so ensure it sanitizes
-# it and do not allow in spurious values from the environment.
-line=PASS; export line
-
-./configure
-
-$MAKE foo.log
-$MAKE bar.log
-chmod a-r foo.log bar.log
-test ! -r foo.log || Exit 77
-$MAKE test-suite.log >stdout && { cat stdout; Exit 1; }
-cat stdout
-grep '^# ERROR: *2$' stdout
-grep '^# ERROR: *2$' test-suite.log
+
+$AUTOMAKE 1
+$AUTOMAKE 2
+
+sed -e 's/$/ /' 1.in > mk.1
+sed -e 's/$/ /' 2.in > mk.2
+
+grep '^\.SUFFIXES:' mk.1
+grep '^\.SUFFIXES:' mk.2
+
+for suf in test log trs; do
+ grep "^\\.SUFFIXES:.* \\.$suf " mk.1
+done
+
+for suf in SH abcdef log trs; do
+ grep "^\\.SUFFIXES:.* \\.$suf " mk.2
+done
:
diff --git a/tests/test-driver-global-log.test
b/tests/test-metadata-global-log.test
similarity index 62%
rename from tests/test-driver-global-log.test
rename to tests/test-metadata-global-log.test
index 7596e94..a6f0999 100755
--- a/tests/test-driver-global-log.test
+++ b/tests/test-metadata-global-log.test
@@ -17,7 +17,8 @@
# Custom test drivers and parallel-tests harness: check the documented
# semantics for deciding when the content of a test log file should be
# copied in the global test-suite.log file. Currently, this is done
-# with the use of the reStructuredText field `:copy-in-global-log:'.
+# with the use of the reStructuredText field `:copy-in-global-log:' in
+# the associated `.trs' files.
parallel_tests=yes
. ./defs || Exit 1
@@ -26,86 +27,104 @@ cat >> configure.in << 'END'
AC_OUTPUT
END
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = ./passthrough-driver
+TEST_LOG_COMPILER = $(SHELL) -e
+END
+
+cat > passthrough-driver <<'END'
+#!/bin/sh
+set -e; set -u;
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+echo "$test_name: RUN"
+"$@" >$log_file 2>&1 5>$trs_file
+END
+chmod a+x passthrough-driver
+
# The `:test-result:' and `:recheck:' fields and the first line of the
# log file should be be irrelevant for the decision of whether a test
# output is to be copied in the `test-suite.log'.
cat > no-1.test <<END
-FAIL: no-1.test
-:test-result: SKIP
-:copy-in-global-log: no
-:test-result: FAIL
-:test-result: XPASS
-not seen 1
+echo :test-result: SKIP >&5
+echo :copy-in-global-log: no >&5
+echo :test-result: FAIL >&5
+echo :test-result: XPASS >&5
+echo not seen 1
+END
+
+# In the last line, with leading and trailing whitespace in the value.
+cat > no-2.test <<END
+echo ":test-result: FAIL" >&5
+echo "not seen 2"
+echo ":recheck: yes" >&5
+echo ":copy-in-global-log:$tab $tab no $tab" >&5
END
for RES in XPASS FAIL XFAIL SKIP ERROR UNKNOWN; do
unindent > $RES.test <<END
- $RES: $RES.test
- :test-result: $RES
- :copy-in-global-log: no
- not seen $RES
+ echo :test-result: $RES >&5
+ echo :copy-in-global-log: no >&5
+ echo not seen $RES
END
done
-# In the last line, with leading and trailing whitespace.
-cat > no-2.test <<END
-:test-result: FAIL
-not seen 2
-:recheck: yes
-:copy-in-global-log:$tab $tab no $tab
-END
-
# In the first line, with no whitespace.
cat > no-3.test <<END
-:copy-in-global-log:no
-not seen 3
-:test-result: FAIL
+echo :copy-in-global-log:no >&5
+echo ":test-result: FAIL" >&5
+echo "not seen 3"
+END
+
+# Leading whitespace before the field.
+cat > no-4.test <<END
+echo ":test-result: FAIL" >&5
+echo " $tab $tab$tab :copy-in-global-log: no" >&5
+echo "not seen 4"
END
cat > yes-1.test <<END
-PASS: yes-1.test
-:test-result: PASS
-:copy-in-global-log: yes
-seen yes 1
+echo :test-result: PASS >&5
+echo :copy-in-global-log: yes >&5
+echo seen yes 1
END
# A lacking `:copy-in-global-log:' implies that the content of
# the log file should be copied.
cat > yes-2.test <<END
-PASS: y.test
-:test-result: PASS
-seen yes 2
+echo :test-result: PASS >&5
+echo seen yes 2
END
-# Corner cases.
-echo ':copy-in-global-log:' > corn-1.test
-echo " $tab $tab$tab" > corn-2.test
-cat > corn-3.test <<'END'
-seen corn 31
-:copy-in-global-log:address@hidden
-seen corn 32
+# Three corner cases.
+
+cat > corn-1.test <<END
+echo seen corn 1
+echo ':copy-in-global-log:' >&5
END
-echo TEST_LOG_DRIVER = ./dummy-driver > Makefile.am
-echo TESTS = *.test >> Makefile.am
+cat > corn-2.test <<END
+echo seen corn 2
+echo '$tab $tab$tab' >&5
+END
-cat > dummy-driver <<'END'
-#!/bin/sh
-set -e
-while test $# -gt 0; do
- case $1 in
- --log-file) log_file=$2; shift;;
- --test-name) test_name=$2; shift;;
- --expect-failure|--color-tests|--enable-hard-errors) shift;;
- --) shift; break;;
- *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
- esac
- shift
-done
-cp $1 $log_file
+cat > corn-3.test <<'END'
+echo seen corn 31
+echo ':copy-in-global-log:address@hidden' >&5
+echo seen corn 32
END
-chmod a+x dummy-driver
+
+echo TESTS = *.test >> Makefile.am
$ACLOCAL
$AUTOCONF
@@ -117,15 +136,12 @@ $AUTOMAKE
# should be checked in other tests.
$MAKE check || :
cat test-suite.log
-
grep '^seen yes 1$' test-suite.log
grep '^seen yes 2$' test-suite.log
+grep '^seen corn 1$' test-suite.log
+grep '^seen corn 2$' test-suite.log
grep '^seen corn 31$' test-suite.log
grep '^seen corn 32$' test-suite.log
-grep '^:copy-in-global-log:$' test-suite.log
-grep "^ $tab $tab$tab$" test-suite.log
$FGREP 'not seen' test-suite.log && Exit 1
-$EGREP '^(XPASS|FAIL|SKIP|ERROR|UNKNOWN)' test-suite.log && Exit 1
-test `grep -c ':test-result:' test-suite.log` -eq 2
:
diff --git a/tests/test-driver-recheck.test b/tests/test-metadata-recheck.test
similarity index 93%
rename from tests/test-driver-recheck.test
rename to tests/test-metadata-recheck.test
index a8654c1..6a372f3 100755
--- a/tests/test-driver-recheck.test
+++ b/tests/test-metadata-recheck.test
@@ -15,9 +15,7 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Test the "make recheck" semantics for custom test drivers, as documented
-# in the Automake manual. Currently, that is based on the reStructuredText
-# field `:recheck:'.
-
+# in the Automake manual.
parallel_tests=yes
. ./defs || Exit 1
@@ -44,12 +42,12 @@ echo ":recheck: who cares" > y-5
echo ":recheck: $tab y" > y-6
echo ":recheck: yeah!$tab$tab " > y-7
cat > y-10 <<END
-foo
+:foo:
:recheck: ???
END
cat > y-11 <<END
:recheck: YES
-foo
+:foo:
END
cat > y-12 <<END
foo
@@ -59,6 +57,8 @@ bar
zardoz
END
+echo " $tab $tab$tab :recheck: yes" > y-8
+
# The :test-result: fields and the fist line of the log should be
# irrelevant for the decision of whether "make recheck" should or
# should not re-run a test.
@@ -83,12 +83,12 @@ echo ":recheck: no " > n-2
echo ":recheck: $tab no" > n-3
echo ":recheck: no $tab$tab " > n-4
cat > n-5 <<END
-foo
+:foo:
:recheck:no
END
cat > n-6 <<END
:recheck: no
-foo
+:foo:
END
cat > n-7 <<END
foo
@@ -98,6 +98,8 @@ bar
zardoz
END
+echo " $tab $tab$tab :recheck: no" > n-8
+
# The :test-result: fields should be irrelevant for the decision of
# whether "make recheck" should or should not re-run a test.
cat > n-100 <<END
@@ -117,10 +119,11 @@ cat Makefile.am # For debugging.
cat > dummy-driver <<'END'
#!/bin/sh
-set -e
+set -e; set -u
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
--test-name) test_name=$2; shift;;
--expect-failure|--color-tests|--enable-hard-errors) shift;;
--) shift; break;;
@@ -129,7 +132,8 @@ while test $# -gt 0; do
shift
done
: > $test_name.run
-cp $1 $log_file
+: > $log_file
+cp $1 $trs_file
END
chmod a+x dummy-driver
diff --git a/tests/test-metadata-results.test b/tests/test-metadata-results.test
new file mode 100755
index 0000000..effa488
--- /dev/null
+++ b/tests/test-metadata-results.test
@@ -0,0 +1,169 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Parallel testsuite harness: check APIs for the registering of test
+# results in `*.trs' files, as documented in the automake manual.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = ./dummy-driver
+TESTS = foo.test bar.test
+END
+
+cat > dummy-driver <<'END'
+#! /bin/sh
+set -e; set -u
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+: > $log_file
+cp $1 $trs_file
+END
+chmod a+x dummy-driver
+
+mk_check ()
+{
+ st=0
+ $MAKE check >stdout || st=$?
+ cat stdout
+ # Our dummy driver make no testsuite progress report.
+ grep ': .*\.test' stdout && Exit 1
+ # Nor it writes to the log files.
+ test -s foo.log && Exit 1
+ test -s bar.log && Exit 1
+ return $st
+}
+
+# This must be different from the one defined in `test/defs', as that
+# assumes that the driver does proper testsuite progress reporting.
+count_test_results ()
+{
+ total=ERR pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR
+ eval "$@"
+ st=0
+ grep "^# TOTAL: *$total$" stdout || rc=1
+ grep "^# PASS: *$pass$" stdout || rc=1
+ grep "^# XFAIL: *$xfail$" stdout || rc=1
+ grep "^# SKIP: *$skip$" stdout || rc=1
+ grep "^# FAIL: *$fail$" stdout || rc=1
+ grep "^# XPASS: *$xpass$" stdout || rc=1
+ grep "^# ERROR: *$error$" stdout || rc=1
+ test $st -eq 0 || Exit 1
+}
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# Basic checks. Also that that "old-style" directives with format
+# "RESULT: test-name" are be ignored now.
+
+cat > foo.test <<END
+FAIL: foo.test
+:test-result: PASS
+END
+echo ERROR: bar.test > bar.test
+mk_check
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+cat > foo.test <<END
+PASS: foo.test
+:test-result: FAIL
+END
+echo SKIP: bar.test > bar.test
+: > bar.test
+mk_check && Exit 1
+count_test_results total=1 pass=0 fail=1 xpass=0 xfail=0 skip=0 error=0
+
+cat > foo.test <<END
+PASS: foo.test
+:test-result: FAIL
+END
+cat > bar.test <<END
+ERROR: foo.test
+:test-result: SKIP
+END
+mk_check && Exit 1
+count_test_results total=2 pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0
+
+cat > foo.test <<END
+:test-result: XFAIL
+:test-result: PASS
+:test-result: SKIP
+END
+cat > bar.test <<END
+:test-result: SKIP
+:test-result: PASS
+:test-result: SKIP
+:test-result: PASS
+:test-result: PASS
+END
+mk_check
+count_test_results total=8 pass=4 fail=0 xpass=0 xfail=1 skip=3 error=0
+
+# Check that all results expected to be supported are *really* supported.
+
+cat > foo.test <<END
+:test-result: PASS
+:test-result: SKIP
+:test-result: XFAIL
+:test-result: FAIL
+:test-result: XPASS
+:test-result: ERROR
+END
+: > bar.test
+mk_check && Exit 1
+count_test_results total=6 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=1
+
+cp foo.test bar.test
+mk_check && Exit 1
+count_test_results total=12 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=2
+
+# Check that we are liberal w.r.t. whitespace use.
+
+: > foo.test
+: > bar.test
+for RESULT in PASS FAIL XPASS XFAIL SKIP ERROR; do
+ sed -e 's/^ *//' -e 's/|//g' >> foo.test <<END
+ |:test-result:$RESULT|
+ |:test-result: $tab $RESULT|
+ |:test-result:$RESULT $tab|
+ |:test-result:$tab$tab $RESULT$tab $tab |
+END
+ echo " $tab $tab$tab :test-result: $RESULT" >> bar.test
+done
+cat foo.test # For debugging.
+cat bar.test # Likewise.
+mk_check && Exit 1
+count_test_results total=30 pass=5 fail=5 xpass=5 xfail=5 skip=5 error=5
+
+:
diff --git a/tests/test-missing.test b/tests/test-missing.test
new file mode 100755
index 0000000..a4ac179
--- /dev/null
+++ b/tests/test-missing.test
@@ -0,0 +1,72 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# parallel-tests:
+# - non-existent scripts listed in TESTS get diagnosed
+# See also related test 'test-missing2.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TESTS = ok.test zardoz.test
+TEST_LOG_COMPILER = true
+END
+
+: > ok.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+$MAKE check >output 2>&1 && { cat output; Exit 1; }
+cat output
+test -f ok.log
+grep '^PASS: ok\.test' output
+$FGREP 'zardoz.log' output
+test ! -f test-suite.log
+
+TESTS='zardoz2.test' $MAKE -e check >output 2>&1 \
+ && { cat output; Exit 1; }
+cat output
+$FGREP 'zardoz2.log' output
+test ! -f test-suite.log
+
+TEST_LOGS='zardoz3.log' $MAKE -e check >output 2>&1 \
+ && { cat output; Exit 1; }
+cat output
+$FGREP 'zardoz3.log' output
+test ! -f test-suite.log
+
+# The errors should persist even after `test-suite.log'
+# has been created.
+
+: > zardoz.test
+$MAKE check
+rm -f zardoz.test
+
+$MAKE check >output 2>&1 && { cat output; Exit 1; }
+cat output
+$FGREP 'zardoz.log' output
+test ! -f test-suite.log
+
+:
diff --git a/tests/parallel-tests-unreadable-log.test b/tests/test-missing2.test
similarity index 56%
rename from tests/parallel-tests-unreadable-log.test
rename to tests/test-missing2.test
index e8e101b..bc2979a 100755
--- a/tests/parallel-tests-unreadable-log.test
+++ b/tests/test-missing2.test
@@ -14,7 +14,10 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check that the testsuite driver copes well with unreadable test logs.
+# parallel-tests:
+# - non-existent scripts listed in TESTS get diagnosed, even when
+# all the $(TEST_LOGS) have a dummy dependency.
+# See also related test 'test-missing.test'.
parallel_tests=yes
. ./defs || Exit 1
@@ -24,37 +27,28 @@ AC_OUTPUT
END
cat > Makefile.am << 'END'
-TESTS = foo.test bar.test
-XFAIL_TESTS = bar.test
+TESTS = foobar1.test foobar2.test
+$(TEST_LOGS):
END
-cat > foo.test << 'END'
-#! /bin/sh
-exit 0
-END
-cat > bar.test << 'END'
-#! /bin/sh
-exit 1
-END
-chmod a+x foo.test bar.test
-
$ACLOCAL
$AUTOCONF
$AUTOMAKE -a
-# The testsuite driver will use this variable, so ensure it sanitizes
-# it and do not allow in spurious values from the environment.
-line=PASS; export line
-
./configure
-$MAKE foo.log
-$MAKE bar.log
-chmod a-r foo.log bar.log
-test ! -r foo.log || Exit 77
-$MAKE test-suite.log >stdout && { cat stdout; Exit 1; }
-cat stdout
-grep '^# ERROR: *2$' stdout
-grep '^# ERROR: *2$' test-suite.log
+$MAKE foobar1.log foobar2.log || Exit 99
+test ! -f foobar1.log || Exit 99
+test ! -f foobar1.trs || Exit 99
+test ! -f foobar2.log || Exit 99
+test ! -f foobar2.trs || Exit 99
+
+$MAKE check >output 2>&1 && { cat output; Exit 1; }
+cat output
+grep 'test-suite\.log.*foobar1\.log' output
+grep 'test-suite\.log.*foobar1\.trs' output
+grep 'test-suite\.log.*foobar2\.log' output
+grep 'test-suite\.log.*foobar2\.trs' output
+test ! -f test-suite.log
:
diff --git a/tests/test-trs-basic.test b/tests/test-trs-basic.test
new file mode 100755
index 0000000..33367d3
--- /dev/null
+++ b/tests/test-trs-basic.test
@@ -0,0 +1,138 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel harness features:
+# - creation and removal of `.trs' auxiliary files
+# - check some internals regarding the use of `.trs' files.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<END
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_EXTENSIONS = .sh .test
+TESTS = foo.test bar.sh sub/zardoz.test
+TEST_LOG_COMPILER = $(SHELL)
+SH_LOG_COMPILER = $(SHELL)
+
+## Used to check some internal details.
+tb:
+ echo $(am__TEST_BASES) > $@
+END
+
+mkdir sub
+echo 'exit $FOO_STATUS' > foo.test
+: > bar.sh
+: > sub/zardoz.test
+
+FOO_STATUS=0; export FOO_STATUS
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+#
+# Check some internal details first.
+#
+
+$MAKE tb
+test x"`cat tb`" = x"foo bar sub/zardoz"
+rm -f tb
+TESTS='foo.test foo2.sh foo3.exe foo4' $MAKE -e tb
+test x"`cat tb`" = x"foo foo2 foo3.exe foo4"
+rm -f tb
+
+#
+# The `test-suite.stamp' file and the `.trs' files get created by
+# "make check" and removed by "make clean" and "make mostlyclean".
+#
+
+: > unrelated.trs
+: > sub/foo.trs
+
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+test -f sub/zardoz.trs
+$MAKE clean
+test ! -f foo.trs
+test ! -f bar.trs
+test ! -f sub/zardoz.trs
+# Unrelated `.trs' files shouldn't be removed.
+test -f unrelated.trs
+test -f sub/foo.trs
+
+# The files should be properly created in case of testsuite failure too.
+FOO_STATUS=1 $MAKE check && Exit 1
+test -f foo.trs
+test -f bar.trs
+test -f sub/zardoz.trs
+$MAKE mostlyclean
+test ! -f foo.trs
+test ! -f bar.trs
+test ! -f sub/zardoz.trs
+# Unrelated `.trs' files shouldn't be removed.
+test -f unrelated.trs
+test -f sub/foo.trs
+
+#
+# Try with a subset of TESTS.
+#
+
+TESTS=foo.test $MAKE -e check
+test -f foo.trs
+test ! -f bar.trs
+test ! -f sub/zardoz.trs
+$MAKE clean
+test ! -f foo.trs
+TESTS='foo.test bar.sh' $MAKE -e check
+test -f foo.trs
+test -f bar.trs
+test ! -f sub/zardoz.trs
+# "make clean" shouldn't remove `.trs' files for tests not in $(TESTS).
+TESTS=bar.sh $MAKE -e clean
+test -f foo.trs
+test ! -f bar.trs
+
+$MAKE clean
+
+#
+# Try with a subset of TEST_LOGS.
+#
+
+TEST_LOGS=sub/zardoz.log $MAKE -e check
+test ! -f foo.trs
+test ! -f bar.trs
+test -f sub/zardoz.trs
+$MAKE clean
+test ! -f sub/zardoz.trs
+TEST_LOGS='foo.log bar.log' $MAKE -e check
+test -f foo.trs
+test -f bar.trs
+test ! -f sub/zardoz.trs
+# "make clean" shouldn't remove `.trs' files for tests whose log
+# is not in $(TEST_LOGS).
+TEST_LOGS=foo.log $MAKE -e clean
+test ! -f foo.trs
+test -f bar.trs
+test ! -f sub/zardoz.trs
+
+:
diff --git a/tests/test-trs-recover.test b/tests/test-trs-recover.test
new file mode 100755
index 0000000..7fff3ae
--- /dev/null
+++ b/tests/test-trs-recover.test
@@ -0,0 +1,166 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel harness features:
+# - recovery from deleted `.trs' files, in various scenarios
+# This test is complex and tricky, but that's acceptable since we are
+# testing semantics that are potentially complex and tricky.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<END
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TESTS = foo.test bar.test baz.test
+TEST_LOG_COMPILER = $(SHELL)
+END
+
+echo 'exit $TEST_STATUS' > foo.test
+echo 'exit $TEST_STATUS' > bar.test
+: > baz.test
+
+TEST_STATUS=0; export TEST_STATUS
+
+# Slower and possible overkill in some situations, but also clearer
+# and safer.
+update_stamp () { $sleep && touch stamp && $sleep; }
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+: Create the required log files.
+$MAKE check
+
+: Recreate by hand.
+rm -f foo.trs bar.trs baz.trs
+$MAKE foo.trs
+test -f foo.trs
+test ! -f bar.trs
+test ! -f baz.trs
+
+: Recreate by hand, several at the same time.
+rm -f foo.trs bar.trs baz.trs
+$MAKE foo.trs bar.trs
+test -f foo.trs
+test -f bar.trs
+test ! -f baz.trs
+
+: Recreate by hand, with a failing test.
+rm -f foo.trs bar.trs
+TEST_STATUS=1 $MAKE bar.trs baz.trs >stdout || { cat stdout; Exit 1; }
+cat stdout
+test ! -f foo.trs
+test -f bar.trs
+test -f baz.trs
+grep '^FAIL: bar\.test' stdout
+$EGREP '^(baz|foo)\.test' stdout && Exit 1
+
+: Recreate with a sweeping "make check", and ensure that also up-to-date
+: '.trs' files are remade.
+update_stamp
+rm -f foo.trs bar.trs
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+is_newest baz.trs stamp
+
+: Recreate with a sweeping "make check" with failing tests. Again,
+: ensure that also up-to-date '.trs' files are remade -- this time we
+: grep the "make check" output verify that.
+rm -f foo.trs bar.trs
+TEST_STATUS=1 $MAKE check >stdout && { cat stdout; Exit 1; }
+test -f foo.trs
+test -f bar.trs
+grep '^FAIL: foo\.test' stdout
+grep '^FAIL: bar\.test' stdout
+grep '^PASS: baz\.test' stdout
+
+: Recreate with a "make check" with redefined TESTS.
+rm -f foo.trs bar.trs baz.trs
+TESTS=foo.test $MAKE -e check
+test -f foo.trs
+test ! -f bar.trs
+test ! -f baz.trs
+
+: Recreate with a "make check" with redefined TEST_LOGS.
+rm -f foo.trs bar.trs baz.trs
+TEST_LOGS=bar.log $MAKE -e check
+test ! -f foo.trs
+test -f bar.trs
+test ! -f baz.trs
+
+: Interactions with "make recheck" are OK.
+rm -f foo.trs bar.trs baz.log baz.trs
+$MAKE recheck >stdout || { cat stdout; Exit 1; }
+cat stdout
+test -f foo.trs
+test -f bar.trs
+test ! -f baz.trs
+test ! -f baz.log
+grep '^PASS: foo\.test' stdout
+grep '^PASS: bar\.test' stdout
+grep 'baz\.test' stdout && Exit 1
+count_test_results total=2 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+: Setup for the next check.
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+test -f baz.trs
+
+: Recreate by remaking the global test log, and ensure that up-to-date
+: '.trs' files are *not* remade.
+update_stamp
+rm -f foo.trs bar.trs test-suite.log
+$MAKE test-suite.log >stdout || { cat stdout; Exit 1; }
+cat stdout
+grep '^PASS: foo\.test' stdout
+grep '^PASS: bar\.test' stdout
+grep 'baz\.test' stdout && Exit 1
+stat *.trs *.log stamp || : # For debugging.
+# Check that make has updated what it needed to, but no more.
+test -f foo.trs
+test -f bar.trs
+is_newest stamp baz.trs
+is_newest test-suite.log foo.trs bar.trs
+
+: Setup for the next check.
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+test -f baz.trs
+
+: Interactions with lazy test reruns are OK.
+rm -f foo.trs
+update_stamp
+touch bar.test
+RECHECK_LOGS= $MAKE -e check >stdout || { cat stdout; Exit 1; }
+cat stdout
+# Check that make has updated what it needed to, but no more.
+test -f foo.trs
+is_newest bar.trs bar.test
+is_newest stamp baz.trs
+grep '^PASS: foo\.test' stdout
+grep '^PASS: bar\.test' stdout
+grep 'baz\.test' stdout && Exit 1
+
+:
diff --git a/tests/test-trs-recover2.test b/tests/test-trs-recover2.test
new file mode 100755
index 0000000..bc94b3b
--- /dev/null
+++ b/tests/test-trs-recover2.test
@@ -0,0 +1,133 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel harness features:
+# - recovery from unreadable `.trs' files, in various scenarios
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<END
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TESTS = foo.test bar.test
+TEST_LOG_COMPILER = true
+END
+
+: > foo.test
+: > bar.test
+
+TEST_STATUS=0; export TEST_STATUS
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+: > t
+chmod a-r t
+test ! -r t || Exit 77
+rm -f t
+
+: Create the required log files.
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+
+: Recreate with a sweeping "make check".
+chmod a-r bar.trs
+$MAKE check
+test -f foo.trs
+test -r foo.trs
+test -f bar.trs
+test -r bar.trs
+
+: Again, this time with one .trs file missing and the other
+: one unreadable.
+rm -f foo.trs
+chmod a-r bar.trs
+$MAKE check
+test -f foo.trs
+test -r foo.trs
+test -f bar.trs
+test -r bar.trs
+
+: Again, but using "make recheck" this time.
+rm -f foo.trs
+chmod a-r bar.trs
+$MAKE recheck >stdout || { cat stdout; Exit 1; }
+cat stdout
+test -f foo.trs
+test -r foo.trs
+test -f bar.trs
+test -r bar.trs
+grep '^PASS: foo\.test' stdout
+grep '^PASS: bar\.test' stdout
+
+: More complex interactions with "make recheck" are OK.
+chmod a-r bar.log bar.trs
+$MAKE recheck >stdout || { cat stdout; Exit 1; }
+cat stdout
+test -f bar.trs
+test -r bar.trs
+grep '^PASS: bar\.test' stdout
+grep 'foo\.test' stdout && Exit 1
+count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+: Recreate by remaking the global test log.
+chmod a-r foo.trs
+rm -f test-suite.log
+$MAKE test-suite.log >stdout || { cat stdout; Exit 1; }
+cat stdout
+test -f foo.trs
+test -r foo.trs
+grep '^PASS: foo\.test' stdout
+grep 'bar\.test' stdout && Exit 1
+# Also test that have only run before should be counted in the final
+# testsuite summary.
+grep '^# TOTAL: *2$' stdout
+
+: Setup for the next check.
+: > baz.test
+sed 's/^TESTS =.*/& baz.test/' Makefile > t
+diff t Makefile && Exit 99
+mv -f t Makefile
+$MAKE check
+test -f foo.trs
+test -f bar.trs
+test -f baz.trs
+
+: Interactions with lazy test reruns are OK.
+chmod a-r foo.trs
+$sleep
+touch stamp
+$sleep
+touch bar.test
+RECHECK_LOGS= $MAKE -e check >stdout || { cat stdout; Exit 1; }
+cat stdout
+test -r foo.trs
+is_newest bar.trs bar.test
+grep '^PASS: foo\.test' stdout
+grep '^PASS: bar\.test' stdout
+grep 'baz\.test' stdout && Exit 1
+# Also test that have only run before should be counted in the final
+# testsuite summary.
+grep '^# TOTAL: *3$' stdout
+
+:
diff --git a/tests/trivial-test-driver b/tests/trivial-test-driver
index d334a93..7b3ef10 100644
--- a/tests/trivial-test-driver
+++ b/tests/trivial-test-driver
@@ -36,10 +36,12 @@ set -u
test_name=INVALID.NAME
log_file=BAD.LOG
+trs_file=BAD.TRS
while test $# -gt 0; do
case $1 in
--test-name) test_name=$2; shift;;
--log-file) log_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
# Ignored.
--expect-failure) shift;;
--color-tests) shift;;
@@ -52,16 +54,18 @@ while test $# -gt 0; do
shift
done
-## Run the test script, get test cases results, display them on console.
+## Log file header.
+{
+ echo "RUN: $test_name"
+ echo "RUN: $test_name" | sed 's/./=/g'
+ echo
+} > $log_file
-tmp_output=$log_file-output.tmp
-tmp_results=$log_file-results.tmp
-tmp_status=$log_file-status.tmp
+## Run the test script, get test cases results, display them on console.
-"$@" 2>&1 | tee $tmp_output | (
+"$@" 2>&1 | tee -a $log_file | (
i=0 st=0
- exec 5> $tmp_results
- : > $tmp_status
+ exec 5> $trs_file
while read line; do
result=
case $line in
@@ -81,38 +85,18 @@ tmp_status=$log_file-status.tmp
echo >&5
fi
done
- test $st -eq 0 || echo fail > $tmp_status
+ if test $st -eq 0; then
+ recheck=no
+ copy_in_global_log=no
+ else
+ recheck=yes
+ copy_in_global_log=yes
+ fi
+ echo ":recheck: $recheck" >&5
+ echo ":copy-in-global-log: $copy_in_global_log" >&5
+ exec 5>&-
) | awk '{ print $0 ", testcase " NR }'
-if test ! -s $tmp_status; then
- global_result=PASS
- recheck=no
- copy_in_global_log=no
-else
- global_result=FAIL
- recheck=yes
- copy_in_global_log=yes
-fi
-
-## Write the log file.
-
-{
- echo "$global_result: $test_name"
- echo "$global_result: $test_name" | sed 's/./=/g'
- # Blank lines required for having the output be valid RST.
- echo
- echo ":recheck: $recheck"
- echo
- echo ":copy-in-global-log: $copy_in_global_log"
- echo
- cat $tmp_results
- echo
- echo --------------------
- echo
- cat $tmp_output
-} > $log_file
-rm -f $tmp_output $tmp_results $tmp_status
-
## And we're done.
exit 0
--
1.7.2.3
- [PATCH 2/2] test harness: use new `.trs' files to hold test metadata (was: Re: [PATCH 1/2] test harness: allow more metadata in log files),
Stefano Lattarini <=