[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH 1/2] test harness: allow more metadata in log files
|
From: |
Stefano Lattarini |
|
Subject: |
[PATCH 1/2] test harness: allow more metadata in log files |
|
Date: |
Fri, 29 Jul 2011 16:33:44 +0200 |
|
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
Following up on this:
<https://lists.gnu.org/archive/html/automake-patches/2011-07/msg00070.html>
On Tuesday 19 July 2011, Stefano Lattarini wrote:
> On Monday 18 July 2011, Ralf Wildenhues wrote:
> > * Stefano Lattarini wrote on Fri, Jul 15, 2011 at 12:36:17AM CEST:
> > > The new code for parsing the testsuite-generated `.log' files,
> > > as introduced in commit `v1.11-872-gc96b881', considers each
> > > `:test-result:' field anywhere in a `.log' file as a declaration
> > > of a test result, and accounts for it as such in the testsuite
> > > summary. Unfortunately this could easily cause spurious test
> > > failures being reported in the testsuite summary. This happened
> > > in practice with the Automake's own testsuite; for example:
> > >
> > > $ make check TESTS='check12-p.test'; echo exit: $?
> > > ...
> > > PASS: check12-p.test
> > > =====================================
> > > 4 of 5 tests failed
> > > See tests/test-suite.log
> > > Please report to address@hidden
> > > =====================================
> > > make[2]: *** [test-suite.log] Error 1
> > > make: *** [check-am] Error 2
> > > exit: 2
> > >
> > > This change introduces a new special `:test-result:' "END", that,
> > > when seen, prevents the rest of the log file from being parsed.
> > >
> > > For more information, refer to the thread:
> > > <http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html>
> > >
> > > * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log
> > > file as soon as the special ":test-result:END" directive is seen.
> > > Related changes and enhancements.
> > > * lib/test-driver: Protect the rest of the log after the result
> > > lined with a ":test-result:END" directive.
> > > * tests/parallel-tests-no-spurious-summary.test: New test.
> > > * tests/test-driver-end-test-results.test: Likewise.
> > > * tests/Makefile.am (TESTS): Update.
> >
> > I'm still not sold on this. It is not as robust as a test protocol
> > could be; also I haven't seen this approach being used in any other test
> > suite environments. Whether some line is considered having a result or
> > not depends on (possibly far-away) context, on whether aggregation with
> > other results has happened. When passed through email, misquoting can
> > change not only the interpretation of the misquoted text (which could be
> > expected) but possibly also of later, correctly quoted (or not quoted)
> > text.
> >
> Good points. Hmm... the more I think about this issue, the more I'm
> convinced that we are (ok, I am ;-) starting to unduly use the `.log'
> files for two ortoghonal purposes:
> 1. keeping a verbose and detailed log of a test script run; and
> 2. keeping a sort of "dumb database" of test results, which is used
> to determine the final testsuite summary (and also by "make recheck"
> to decide which tests to re-run).
> Maybe it's time we begin to use two different set of files for this two
> different roles? E.g., for each `foo.test', we generate a `foo.log' file
> and a, say, `foo.tdat' (Test DATa file) containing the test results and
> possibly other metadata used by our harness? This might also allow for
> easier extensibility of such metadata in the future.
>
I'm pursuing this strategy now; it requires some trickery, but it seems
to work quite well, and not to cause any codebase bloating. Also, the
documentation of the new behaviour will be clearer and smaller, and the
addition of new complexity in the testsuite harness will be paid off with
a decrease in complexity of the test drivers (a win in the long term, as
we plan to add more test drivers in the future).
Here is a refactoring/preparatory patch, which makes the metadata in the
log files more flexible and more sophisticated; while doing so, it IMHO
also shows that a new set of generated files holding only test metadata
would be more natural and flexible.
Given what I've said, please do not judge this patch in isolation;
consider it only a "transitional step" towards the new APIs.
Thanks,
Stefano
From 3d7e15129c1a0805d1c4e8e12f16c690738e3282 Mon Sep 17 00:00:00 2001
Message-Id: <address@hidden>
From: Stefano Lattarini <address@hidden>
Date: Tue, 26 Jul 2011 14:51:20 +0200
Subject: [PATCH] test harness: allow more metadata in log files
This change reworks and improves the parallel test harness to use
more specialized reStructuredText fields in the log files (instead
of relying on specially-placed of "magic lines" and more indirect
semantical formatting); the new fields are the following:
- ":recheck:": tell whether the associated test will have to be
re-run by "make recheck";
- ":copy-in-global-log:": tell whether the content of the log
file should be copied in the "global log" `test-suite.log';
- ":end-metadata:", which inhibits the scanning of the rest of
the log file (for what concerns test metadata).
Also, the special `:test-result:' value "END" has been removed,
superseded by the new `:end-metadata:' field.
* doc/automake.texi (Log files generation and test results
recording): Document the new API and semantics. Remove or fix
some obsolete comments.
* lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html):
Adjust comments and code.
* lib/tap-driver (finish): Adjust, with the help of ...
(must_recheck, copy_in_global_log): ... these new functions.
* lib/test-driver (Main code): Adjust, with the help of ...
($recheck, $gcopy): ... these new variables.
* tests/trivial-test-driver: Update to obey the new APIs.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/tap-passthrough.test: Relax the test, avoiding to check
what is written in `test-suite.log'; such check has been moved ...
* tests/tap-global-log.test: ... in this new test, and extended.
* tests/test-driver-metadata-no-leading-space.test: New test.
* tests/test-driver-end-test-results.test: Removed, it checked
the old APIs; superseded by ...
* tests/test-driver-end-metadata.test: ... this new test.
* tests/tap-log.test: Improve syncing with ...
* tests/test-log.test: ... this new test.
* tests/parallel-tests.test: Remove some duplication w.r.t. this
last new test. Updated heading comments.
* tests/Makefile.am (TESTS): Update.
---
ChangeLog | 40 +++++
doc/automake.texi | 126 ++++++++--------
lib/Automake/tests/Makefile.in | 40 +++--
lib/am/check.am | 72 +++++-----
lib/tap-driver | 33 ++++-
lib/test-driver | 26 +++--
tests/Makefile.am | 5 +-
tests/Makefile.in | 45 ++++---
tests/parallel-tests.test | 40 ++----
tests/tap-global-log.test | 140 ++++++++++++++++++
tests/tap-log.test | 23 ++--
tests/tap-passthrough.test | 28 +---
tests/test-driver-end-metadata.test | 120 +++++++++++++++
tests/test-driver-end-test-results.test | 95 ------------
tests/test-driver-global-log.test | 121 +++++++++-------
tests/test-driver-metadata-no-leading-space.test | 94 ++++++++++++
tests/test-driver-recheck.test | 125 ++++++++++------
tests/test-log.test | 168 ++++++++++++++++++++++
tests/trivial-test-driver | 9 ++
19 files changed, 952 insertions(+), 398 deletions(-)
create mode 100755 tests/tap-global-log.test
create mode 100755 tests/test-driver-end-metadata.test
delete mode 100755 tests/test-driver-end-test-results.test
create mode 100755 tests/test-driver-metadata-no-leading-space.test
create mode 100755 tests/test-log.test
diff --git a/ChangeLog b/ChangeLog
index 272e4db..634b2d4 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,45 @@
2011-07-27 Stefano Lattarini <address@hidden>
+ test harness: allow more metadata in log files
+ This change reworks and improves the parallel test harness to use
+ more specialized reStructuredText fields in the log files (instead
+ of relying on specially-placed of "magic lines" and more indirect
+ semantical formatting); the new fields are the following:
+ - ":recheck:": tell whether the associated test will have to be
+ re-run by "make recheck";
+ - ":copy-in-global-log:": tell whether the content of the log
+ file should be copied in the "global log" `test-suite.log';
+ - ":end-metadata:", which inhibits the scanning of the rest of
+ the log file (for what concerns test metadata).
+ Also, the special `:test-result:' value "END" has been removed,
+ superseded by the new `:end-metadata:' field.
+ * doc/automake.texi (Log files generation and test results
+ recording): Document the new API and semantics. Remove or fix
+ some obsolete comments.
+ * lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html):
+ Adjust comments and code.
+ * lib/tap-driver (finish): Adjust, with the help of ...
+ (must_recheck, copy_in_global_log): ... these new functions.
+ * lib/test-driver (Main code): Adjust, with the help of ...
+ ($recheck, $gcopy): ... these new variables.
+ * tests/trivial-test-driver: Update to obey the new APIs.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/tap-passthrough.test: Relax the test, avoiding to check
+ what is written in `test-suite.log'; such check has been moved ...
+ * tests/tap-global-log.test: ... in this new test, and extended.
+ * tests/test-driver-metadata-no-leading-space.test: New test.
+ * tests/test-driver-end-test-results.test: Removed, it checked
+ the old APIs; superseded by ...
+ * tests/test-driver-end-metadata.test: ... this new test.
+ * tests/tap-log.test: Improve syncing with ...
+ * tests/test-log.test: ... this new test.
+ * tests/parallel-tests.test: Remove some duplication w.r.t. this
+ last new test. Updated heading comments.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-27 Stefano Lattarini <address@hidden>
+
tests: fix a spurious failure with non-bash shells
* tests/tap-fancy2.test: Remove an unportable use of backslashes
with the `echo' builtin, which was causing some shells (among them
diff --git a/doc/automake.texi b/doc/automake.texi
index e1a1ca3..2e5e2ba 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -9372,41 +9372,26 @@ consistency and a more pleasant user experience.
The test driver must correctly generate the file specified by the
@option{--log-file} option (of course even when the tested program
-fails or crashes). This file is quite free-form, but still it has
-to conform to the following conventions, in order to work with the
address@hidden harness and take advantage of its features.
+fails or crashes). This file, while being quite free-form, @emph{must}
+register some metadata through the use of custom reStructuredText fields,
+in order to work with the @option{parallel-tests} harness and take
+advantage of its features. Note that such fields will be recognized only
+when used @emph{exactly} at the beginning of a line -- not even leading
+whitespace is permitted.
address@hidden @bullet
address@hidden
address@hidden Global test script result
address@hidden Test result, global
-The @emph{global test result} for the whole test script must be placed
-in the first line of the @file{.log} file.
address@hidden The following semantics is checked by tests
'test-driver-recheck.test'
address@hidden and 'test-driver-global-log.test'. Please keep them in sync
whenever
address@hidden the following is changed.
-If this line does not begin with either @code{PASS:}, @code{XFAIL:} or
address@hidden:}, the test script will be considered failed, and it will be
-re-run by @code{make recheck}. Also, if this line does not begin with
-either @code{PASS:} or @code{XFAIL:}, the content of the @file{.log} file
-will be copied into the global @file{test-suite.log} (in order to help in
-debugging and bug-report analysis).
address@hidden @code
address@hidden
address@hidden Single test case result
address@hidden Test result, single
address@hidden @code{:test-result:} reStructuredText field
address@hidden reStructuredText field, @code{:test-result:}
-One of the main features of the new testsuite harness is the ability to
-support test protocols that allow a single test script to run more
-test cases, @emph{each with its distinct result}. In order for the
-testsuite summary to be correct in this case, the test driver must
-register @emph{each} such result in the generated @file{.log} file, using
-the @code{:test-result:} reStructuredText field. Otherwise, only the
-global test result will be considered (as it is the case of the default
address@hidden driver). Note that such a field will be
-recognized only when used @emph{exactly} at the beginning of a line:
-leading whitespace will not be ignored.
address@hidden :test-result:
address@hidden Register test result
address@hidden Register test case result
address@hidden Test result, registering
address@hidden Test case result, registering
address@hidden @code{:test-result:}
address@hidden reStructuredText field, @code{:test-result:}
+The test driver must use this filed to register the results of @emph{each}
+test case run by a test script file. Several @code{:test-result:} fields
+can be present in the same log file; this is done in order to support test
+protocols that allow a single test script to run more test cases.
@c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
The only recognized test results are currently @code{PASS}, @code{XFAIL},
@@ -9415,48 +9400,63 @@ when declared with @code{:test-result:}, can be
optionally followed by
text holding the name and/or a brief description of the corresponding
test; the @option{parallel-tests} harness will ignore such extra text when
generating @file{test-suite.log} and preparing the testsuite summary.
-Also, @code{:test-result:} can be used with a special ``pseudo-result''
address@hidden, that will instruct the testsuite harness to stop scanning
-the rest of the log file (this feature is used internally by the default
address@hidden driver, and can be useful to avoid spurious
-testsuite results).
-Let's see a small example. If a @file{.log} file contains the following
-lines (exactly in this order, but possibly interleaved with other
address@hidden:test-results:}'' lines):
address@hidden Keep in sync with 'test-driver-recheck.test'.
address@hidden @code{:recheck:}
address@hidden :recheck:
address@hidden reStructuredText field, @code{:recheck:}
+If this field is present and defined to @code{no}, then the corresponding
+test script will @emph{not} be run upon a @command{make recheck}. What
+happens when two or more @code{:recheck:} fields are present in a
address@hidden file is undefined behaviour.
+
address@hidden Keep in sync with 'test-driver-global-log.test'.
address@hidden @code{:copy-in-global-log:}
address@hidden :copy-in-global-log:
address@hidden reStructuredText field, @code{:copy-in-global-log:}
+If this field is present and defined to @code{no}, then the content
+of the @file{.log} file will @emph{not} be copied into the global
address@hidden We allow to forsake such copying because, while
+it can be useful in debugging and analysis of bug report, it can also be
+just a waste of space in normal situations, e.g., when a test script is
+successful. What happens when two or more @code{:copy-in-global-log:}
+fields are present in a @file{.log} file is undefined behaviour.
+
address@hidden Keep in sync with 'test-driver-end-metadata.test'.
address@hidden @code{:end-metadata:}
address@hidden :end-metadata:
address@hidden reStructuredText field, @code{:end-metadata:}
+This special field causes all the following text in the @file{.log}
+file not to be scanned by the testsuite harness in search for other
+test metadata. This feature is used internally by the default
address@hidden driver, and can be useful to e.g., avoid
+spurious testsuite results.
+
address@hidden table
+
address@hidden
+Let's see a small example. Assume a @file{.log} file contains the
+following lines (in this order, but possibly interleaved with other
+``non-metadata'' lines):
@example
:test-result: PASS server starts
:test-result: PASS HTTP/1.1 request
:test-result: FAIL HTTP/1.0 request
+:recheck: yes
:test-result: SKIP HTTPS request (TLS library wasn't available)
:test-result: PASS server stops
-:test-result: END
+:global-log-copy: no
+:metadata-end:
:test-result: XPASS (this shouldn't be parsed as a test result)
@end example
@noindent
-it means that the corresponding test script contributes with @emph{five}
-test results to the testsuite summary (three of these tests being
-successful, one failed, and one skipped).
-
address@hidden FIXME: currently, the @code{:test-result:} field is recognized
anywhere
address@hidden in the @file{.log} file; this is possibly prone to generating
spurious
address@hidden results, in case of verbose tests. We should allow a special
address@hidden @code{:test-result:} that stops any further parsing (maybe a line
address@hidden @code{:test-result:END} will do?).
-
address@hidden itemize
-
-Finally, note that the declaration of the global test result in the first
-line, apart from being needed for backward-compatibility with the default
address@hidden driver, can be useful also for test protocols
-that allow more test results per test script: using it, a custom test
-driver is allowed to decide what the global outcome of the test script
-is in case of conflicting test results within the script. For example,
-if a test script reports 8 successful test cases and 2 skipped test
-cases, some drivers might report that globally as a SKIP, while others
-as a PASS.
+Then the corresponding test script will be re-run by @command{make check},
+will contribute with @emph{five} test results to the testsuite summary
+(three of these tests being successful, one failed, and one skipped), and
+the @file{.log} file content will @emph{not} be copied in the global log
+file @file{test-suite.log}.
@node Testsuite progress output
@subsubsection Testsuite progress output
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 3777b5d..9b45aa2 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -331,8 +331,10 @@ cscope cscopelist:
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
- rst_magic=":test-result:"; \
+ tab=' '; \
nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
+ IFS=" $$tab$$nl"; oIFS=$$IFS; \
+ ws="[ $$tab]"; \
list='$(TEST_LOGS)'; \
list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
@@ -340,12 +342,19 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
exec 5<&0; \
for f in $$list2; do \
exec 0<$$f; \
+ IFS=''; \
while read line; do \
- case $$line in $$rst_magic*);; *) continue;; esac; \
- res=`expr "x$$line" : "x$$rst_magic[ ]*\\\\(.*\\\\)$$"`; \
- case $$res in END*) break;; esac; \
- results2="$$results2$$nl$$res"; \
+ IFS=$$oIFS; \
+ case $$line in \
+ :end-metadata:*) \
+ break;; \
+ :test-result:*) \
+ res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
+ results2="$$results2$$nl$$res";; \
+ esac; \
+ IFS=''; \
done; \
+ IFS=$$oIFS; \
exec 0<&5; \
done; \
exec 5<&-; \
@@ -399,11 +408,9 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo ".. contents:: :depth: 2"; \
echo; \
for f in $$list; do \
- test -r $$f && read line < $$f || line=; \
- case $$line in \
- PASS:*|XFAIL:*);; \
- *) echo; cat $$f;; \
- esac; \
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
+ || { echo && cat $$f; } \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
@@ -462,13 +469,14 @@ check-html:
$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_HTML) || exit 4; \
exit $$rv
recheck recheck-html:
- @target=`echo $@ | sed 's,^re,,'`; \
+ @ws='[ ]'; \
+ target=`echo $@ | sed 's,^re,,'`; \
list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
- if test -r $$f && read line < $$f; then \
- case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
- else echo $$f; fi; \
+ list=`for f in $$list; do \
+ test -f $$f || continue; \
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
+ || echo $$f; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS)
TEST_LOGS="'"$$list"'"'
diff --git a/lib/am/check.am b/lib/am/check.am
index ffb70ba..272fc0a 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -126,14 +126,14 @@ $(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
-## The custom reStructuredText field used to register the outcome of
-## a test. This is for supporting test protocols, such as TAP and
-## SubUnit, where a single test script can run multiple tests, each
-## with its own outcome.
- rst_magic=":test-result:"; \
+ tab=' '; \
## The following line will define the `$nl' shell variable to a literal
## newline. Idiom taken from the Autoconf manual.
nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
+## Sanitize internal field separator for the shell: "space tab newline".
+ IFS=" $$tab$$nl"; oIFS=$$IFS; \
+## Regular expression to match a whitespace.
+ ws="[ $$tab]"; \
## All test logs.
list='$(TEST_LOGS)'; \
## Readable test logs.
@@ -148,19 +148,27 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
exec 5<&0; \
for f in $$list2; do \
exec 0<$$f; \
+## We want to support special reStructuredText fields only when they're
+## placed at the beginning of a line (as documented in the manual), so
+## we have to temporarily clear the $IFS variable to prevent automatic
+## whitespace stripping by the shell.
+ IFS=''; \
while read line; do \
-## Ignore lines that do not declare test results.
- case $$line in $$rst_magic*);; *) continue;; esac; \
-## The test result declared by this line goes in $res.
- res=`expr "x$$line" : "x$$rst_magic[ ]*\\\\(.*\\\\)$$"`; \
-## A special result of "END" means that the rest of the log is not to be
-## scanned. Be a little lax in matching such special result, to allow
-## for e.g., trailing whitespace or punctuation.
- case $$res in END*) break;; esac; \
-## Otherwise register the result.
- results2="$$results2$$nl$$res"; \
+ IFS=$$oIFS; \
+ case $$line in \
+## Special field that tells not to scan the rest of the log file.
+ :end-metadata:*) \
+ break;; \
+## A test test result -- register it.
+ :test-result:*) \
+ res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
+ results2="$$results2$$nl$$res";; \
+ esac; \
+## For the next loop.
+ IFS=''; \
done; \
-## Restore original standard input.
+## Restore original standard input and internal field separator.
+ IFS=$$oIFS; \
exec 0<&5; \
done; \
## We don't need this anymore.
@@ -225,14 +233,11 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo; \
echo ".. contents:: :depth: 2"; \
echo; \
-## Here we assume that the test driver writes a proper "summarizing
-## result" for the test script on the first line.
for f in $$list; do \
- test -r $$f && read line < $$f || line=; \
- case $$line in \
- PASS:*|XFAIL:*);; \
- *) echo; cat $$f;; \
- esac; \
+## FIXME: two forks per test -- this is horrendously inefficient!
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
+ || { echo && cat $$f; } \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
@@ -326,20 +331,17 @@ AM_RECURSIVE_TARGETS += check-html
## Rechecking failures. ##
## -------------------- ##
-## Rerun all FAILed or XPASSed tests.
+## Rerun all tests that experienced an error or an unexpected failure.
recheck recheck-html:
- @target=`echo $@ | sed 's,^re,,'`; \
+ @ws='[ ]'; \
+ target=`echo $@ | sed 's,^re,,'`; \
list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
- if test -r $$f && read line < $$f; then \
-## Here we assume that the test driver writes a proper summary for the
-## test script on the first line. See the comments in the rules of
-## $(TEST_SUITE_LOG) above for why we consider this acceptable and even
-## advisable.
- case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
-## A test whose log is unreadable is to be considered failed.
- else echo $$f; fi; \
+ list=`for f in $$list; do \
+ test -f $$f || continue; \
+## FIXME: two forks per test -- this is horrendously inefficient!
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
+ || echo $$f; \
done | tr '\012\015' ' '`; \
## This apparently useless munging helps to avoid a nasty bug (a
## segmentation fault!) on Solaris XPG4 make.
diff --git a/lib/tap-driver b/lib/tap-driver
index 69533b4..9075d35 100755
--- a/lib/tap-driver
+++ b/lib/tap-driver
@@ -138,6 +138,19 @@ TEST_RESULTS :
return @test_results;
}
+ # Whether the test script should be re-run by "make recheck".
+ sub must_recheck ()
+ {
+ return grep { !/^(?:XFAIL|PASS|SKIP)$/ } (keys %test_results);
+ }
+
+ # Whether the content of the log file associated to this test should
+ # be copied into the "global" test-suite.log.
+ sub copy_in_global_log ()
+ {
+ return grep { not $_ eq "PASS" } (keys %test_results);
+ }
+
# FIXME: this can certainly be improved ...
sub get_global_test_result ()
{
@@ -168,25 +181,31 @@ sub finish ()
{
open LOG, ">", $log_file or die "opening $log_file: $!\n";
- # We need to declare a global test result in order to have
- # "make recheck" working.
+ # Some extra trailing empty lines outputted below are required
+ # to support reStructuredText -> HTML conversion.
+
my $global_result = get_global_test_result;
my $global_result_line = "$global_result: $test_script_name";
print LOG "$global_result_line\n";
print LOG "=" x length ($global_result_line) . "\n";
print LOG "\n";
+ print LOG ":recheck: " . (must_recheck ? "yes" : "no") . "\n";
+ print LOG "\n";
+
+ print LOG ":copy-in-global-log: " .
+ (copy_in_global_log ? "yes" : "no") . "\n";
+ print LOG "\n";
+
foreach (get_test_results)
{
print LOG ":test-result: $_\n";
- # Add extra trailing empty lines to support reStructuredText -> HTML
- # conversion.
print LOG "\n";
}
- # So that the output from the test script won't be parsed for test
- # results (which would potentially expose us to false positives).
- print LOG ":test-result: END\n";
+ # So that the output from the test script won't be parsed for
+ # testsuite metadata.
+ print LOG ":end-metadata:\n";
print LOG "\n";
close TMP or die "closing $log_file-t: $!\n";
diff --git a/lib/test-driver b/lib/test-driver
index aa39c5b..45b3afe 100755
--- a/lib/test-driver
+++ b/lib/test-driver
@@ -1,7 +1,7 @@
#! /bin/sh
# test-driver - basic driver script for the `parallel-tests' mode.
-scriptversion=2011-06-29.13; # UTC
+scriptversion=2011-07-27.12; # UTC
# Copyright (C) 2011 Free Software Foundation, Inc.
#
@@ -107,12 +107,12 @@ if test $enable_hard_errors = no && test $estatus -eq 99;
then
fi
case $estatus:$expect_failure in
- 0:yes) col=$red; res=XPASS;;
- 0:*) col=$grn; res=PASS ;;
- 77:*) col=$blu; res=SKIP ;;
- 99:*) col=$mgn; res=ERROR;;
- *:yes) col=$lgn; res=XFAIL;;
- *:*) col=$red; res=FAIL ;;
+ 0:yes) col=$red res=XPASS recheck=yes gcopy=yes;;
+ 0:*) col=$grn res=PASS recheck=no gcopy=no;;
+ 77:*) col=$blu res=SKIP recheck=no gcopy=yes;;
+ 99:*) col=$mgn res=ERROR recheck=yes gcopy=yes;;
+ *:yes) col=$lgn res=XFAIL recheck=no gcopy=yes;;
+ *:*) col=$red res=FAIL recheck=yes gcopy=yes;;
esac
# Report outcome to console.
@@ -121,12 +121,20 @@ echo "${col}${res}${std}: $test_name"
# Now write log file.
{
echo "$res: $test_name (exit: $estatus)" | rst_section
+ # Blank lines required for having the output be valid RST.
+ echo
echo ":test-result: $res (exit status: $estatus)"
echo
- echo ":test-result:END"
+ echo ":recheck: $recheck"
+ echo
+ echo ":copy-in-global-log: $gcopy"
+ echo
+ # So that the output from the test script won't be parsed for
+ # testsuite metadata.
+ echo ":end-metadata:"
echo
# Use a reStructuredText transition to better separate the test
- # outcome report from its registered output.
+ # metadata the test registered output.
echo ------------
echo
cat $tmpfile
diff --git a/tests/Makefile.am b/tests/Makefile.am
index f322054..9a0c222 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -743,7 +743,8 @@ testsuite-summary-color.test \
testsuite-summary-count.test \
testsuite-summary-count-many.test \
testsuite-summary-reference-log.test \
-test-driver-end-test-results.test \
+test-driver-end-metadata.test \
+test-driver-metadata-no-leading-space.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
test-driver-custom-xfail-tests.test \
@@ -757,6 +758,7 @@ test-driver-global-log.test \
test-driver-recheck.test \
test-driver-strip-vpath.test \
test-driver-fail.test \
+test-log.test \
parse.test \
percent.test \
percent2.test \
@@ -1001,6 +1003,7 @@ tap-exit.test \
tap-signal.test \
tap-fancy.test \
tap-fancy2.test \
+tap-global-log.test \
tap-global-result.test \
tap-html.test \
tap-log.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 0ce3dd9..13e850c 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -988,7 +988,8 @@ testsuite-summary-color.test \
testsuite-summary-count.test \
testsuite-summary-count-many.test \
testsuite-summary-reference-log.test \
-test-driver-end-test-results.test \
+test-driver-end-metadata.test \
+test-driver-metadata-no-leading-space.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
test-driver-custom-xfail-tests.test \
@@ -1002,6 +1003,7 @@ test-driver-global-log.test \
test-driver-recheck.test \
test-driver-strip-vpath.test \
test-driver-fail.test \
+test-log.test \
parse.test \
percent.test \
percent2.test \
@@ -1246,6 +1248,7 @@ tap-exit.test \
tap-signal.test \
tap-fancy.test \
tap-fancy2.test \
+tap-global-log.test \
tap-global-result.test \
tap-html.test \
tap-log.test \
@@ -1441,8 +1444,10 @@ cscope cscopelist:
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
- rst_magic=":test-result:"; \
+ tab=' '; \
nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
+ IFS=" $$tab$$nl"; oIFS=$$IFS; \
+ ws="[ $$tab]"; \
list='$(TEST_LOGS)'; \
list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
@@ -1450,12 +1455,19 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
exec 5<&0; \
for f in $$list2; do \
exec 0<$$f; \
+ IFS=''; \
while read line; do \
- case $$line in $$rst_magic*);; *) continue;; esac; \
- res=`expr "x$$line" : "x$$rst_magic[ ]*\\\\(.*\\\\)$$"`; \
- case $$res in END*) break;; esac; \
- results2="$$results2$$nl$$res"; \
+ IFS=$$oIFS; \
+ case $$line in \
+ :end-metadata:*) \
+ break;; \
+ :test-result:*) \
+ res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
+ results2="$$results2$$nl$$res";; \
+ esac; \
+ IFS=''; \
done; \
+ IFS=$$oIFS; \
exec 0<&5; \
done; \
exec 5<&-; \
@@ -1509,11 +1521,9 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo ".. contents:: :depth: 2"; \
echo; \
for f in $$list; do \
- test -r $$f && read line < $$f || line=; \
- case $$line in \
- PASS:*|XFAIL:*);; \
- *) echo; cat $$f;; \
- esac; \
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
+ || { echo && cat $$f; } \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
@@ -1572,13 +1582,14 @@ check-html:
$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_HTML) || exit 4; \
exit $$rv
recheck recheck-html:
- @target=`echo $@ | sed 's,^re,,'`; \
+ @ws='[ ]'; \
+ target=`echo $@ | sed 's,^re,,'`; \
list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
- if test -r $$f && read line < $$f; then \
- case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
- else echo $$f; fi; \
+ list=`for f in $$list; do \
+ test -f $$f || continue; \
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
+ || echo $$f; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS)
TEST_LOGS="'"$$list"'"'
diff --git a/tests/parallel-tests.test b/tests/parallel-tests.test
index b67922f..2d9e2a2 100755
--- a/tests/parallel-tests.test
+++ b/tests/parallel-tests.test
@@ -14,14 +14,14 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check parallel-tests features:
-# - VERBOSE
-# - clean
-# - TEST_SUITE_LOG
-# - dependencies between tests
-# - TESTS
-# - TEST_LOGS
-# - RECHECK_LOGS
+# Basic checks on parallel-tests support:
+# - console output
+# - log files, and what goes in 'test-suite.log'
+# - make clean
+# - dependencies between tests
+# - TESTS redefinition at runtime
+# - TEST_LOGS redefinition at runtime
+# - RECHECK_LOGS redefinition at runtime
parallel_tests=yes
. ./defs || Exit 1
@@ -31,7 +31,6 @@ AC_OUTPUT
END
cat > Makefile.am << 'END'
-TEST_SUITE_LOG = mylog.log
TESTS = foo.test bar.test baz.test
XFAIL_TESTS = bar.test
foo.log: bar.log
@@ -69,23 +68,20 @@ $AUTOMAKE -a
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
count_test_results total=3 pass=1 fail=1 skip=0 xfail=0 xpass=0 error=1
-test -f mylog.log
-cat mylog.log
-test `grep -c '^FAIL:' mylog.log` -eq 1
-test `grep -c '^ERROR:' mylog.log` -eq 1
-$EGREP '^(X?PASS|XFAIL|SKIP)' mylog.log && Exit 1
+test -f test-suite.log
+cat test-suite.log
+test `grep -c '^FAIL:' test-suite.log` -eq 1
+test `grep -c '^ERROR:' test-suite.log` -eq 1
+$EGREP '^(X?PASS|XFAIL|SKIP)' test-suite.log && Exit 1
test -f baz.log
test -f bar.log
test -f foo.log
-# clean should remove all log files (but not more).
-: > unrelated.log
$MAKE clean
test ! -f baz.log
test ! -f bar.log
test ! -f foo.log
-test ! -f mylog.log
-test -f unrelated.log
+test ! -f test-suite.log
# Check dependencies: baz.test needs to run before bar.test,
# but foo.test is not needed.
@@ -100,7 +96,7 @@ grep '^ERROR: bar\.test$' stdout
test -f baz.log
test -f bar.log
test ! -f foo.log
-test -f mylog.log
+test -f test-suite.log
# Upon a lazy rerun, foo.test should be run, but the others shouldn't.
# Note that the lazy rerun still exits with a failure, due to the previous
@@ -139,12 +135,6 @@ grep '^# PASS: *1$' stdout
grep '^# FAIL: *1$' stdout
grep '^# ERROR: *1$' stdout
-# Test VERBOSE.
-env VERBOSE=yes $MAKE -e check > stdout && { cat stdout; Exit 1; }
-cat stdout
-grep 'this is.*bar.test' stdout
-grep 'this is.*baz.test' stdout
-
$MAKE clean
env TEST_LOGS=baz.log $MAKE -e check > stdout && { cat stdout; Exit 1; }
cat stdout
diff --git a/tests/tap-global-log.test b/tests/tap-global-log.test
new file mode 100755
index 0000000..da5aeb8
--- /dev/null
+++ b/tests/tap-global-log.test
@@ -0,0 +1,140 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - which log files get copied in the global log?
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > ok.test << 'END'
+1..5
+ok 1
+ok 2
+ok 3
+not seen in global log
+ok 4
+ok 5
+END
+
+cat > top << 'END'
+1..6
+ok 1
+ok 2
+ok 3
+END
+
+cat > bot << 'END'
+ok 5
+ok 6
+END
+
+cat top - bot > skip.test << 'END'
+ok # SKIP
+::skip::
+END
+
+cat top - bot > todo.test << 'END'
+not ok # TODO
+::todo::
+END
+
+cat top - bot > fail.test << 'END'
+not ok
+::fail::
+END
+
+cat top - bot > xpass.test << 'END'
+ok # TODO
+::xpass::
+END
+
+cat top - bot > bail.test << 'END'
+::bail::
+Bail out!
+END
+
+cat top - bot > error.test << 'END'
+::error::
+1..7
+END
+
+# Created with "dd if=/dev/urandom count=1 | base64-encode"
+cat > hodgepodge <<'END'
+1+0 records in
+1+0 records out
+512 bytes (512 B) copied, 0.000241092 s, 2.1 MB/s
+gdivnV4VhL4DOzhE3zULJuun3PwqqQqMdATVcZbIQkNgyRvNBoqqHMBQEs7QsjDbp2nK+Szz
+EcelGyvXmHrW7yImaom6Yrg95k31VCmp/pGDRnTDwDrcOPJiv9jDReEmTAQkPuqLO+mFNly+
+DDHM9fNbzGumstsQ3wq3DOXz1pCV3JXwhjeaHSboeEbmr55bX0XHLSKaecctA0rXDXEyZWZ/
+ODlawSrAXzw0H7a+xBwjnNXZ3zYiwk3x+WQrPqNjb+qXiLLTxAKzx2/KnaFhxkPlte5jPRNB
+FciDolL+H/10LsCdSzLOrGnY2zH6vL2JMZfxjnb73zWFcdWWE01LTD7wpN5O1MP3+N47lcVe
+lWbkD04cJvhwxLElYSO24B743GG5EyGYt9SeZRE6xbgwq3fVOS8KqjwGxwi4adSBTtw0CV8W
+S/6n8dck1vBvjA+qpk0zMSYSqc3+jzW9UiGTmTEIwfw80p/lGNsfjcNBJ86nFkWUnHmrsi8k
+Dv57sK70mTg239g08f5Uvdga/5UreMBSgB0hUj5sbq57r7B1fsVr0Kag468la8zKy3ZEZ0gX
+++sbaJ9WGHhnKvjooeH+4Y6HwAFsdINde++FlCmp4EuNKKEEuXbSKLaOTy3+6pJ2DYdvRCL+
+frZwxH4hcrw8qh+8IakB02viewZS/qT57v4=
+END
+
+exec 5>misc.test
+echo 'ok # SKIP' >&5
+cat hodgepodge >&5
+echo 'not ok # TODO' >&5
+echo 'ok' >&5
+exec 5>&-
+
+cat > skipall.test << 'END'
+1..0 # SKIP all
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+END
+echo TESTS = *.test >> Makefile.am
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# We don't care about the exit status of "make check" here.
+$MAKE check || :
+cat test-suite.log
+
+grep 'ok\.test|not seen' test-suite.log && Exit 1
+
+for s in skip todo fail xpass bail error; do
+ $FGREP "::$s::" test-suite.log
+done
+
+grep '^1\.\.0 # SKIP all$' test-suite.log
+
+case `cat test-suite.log` in
+ *"`cat hodgepodge`"*) ;;
+ *) Exit 1;;
+esac
+
+:
diff --git a/tests/tap-log.test b/tests/tap-log.test
index 37b5956..e5bab2f 100755
--- a/tests/tap-log.test
+++ b/tests/tap-log.test
@@ -17,9 +17,10 @@
# TAP support:
# - log file creation
# - log file removal
-# - stdout and stderr of a TAP-generating test script go in its log file
+# - stdout and stderr of a script go in its log file
# - TEST_SUITE_LOG redefinition, at either automake or make time
# - VERBOSE environment variable support
+# Keep in sync with 'test-log.test'.
parallel_tests=yes
. ./defs || Exit 1
@@ -94,7 +95,7 @@ chmod a+x *.test
$ACLOCAL
$AUTOCONF
-$AUTOMAKE
+$AUTOMAKE -a
./configure
@@ -111,17 +112,18 @@ for result in pass fail xfail xpass skip error; do
done
test $st -eq 0 || Exit 1
cat my.log # For debugging.
-for result in fail xpass skip error; do
+for result in xfail fail xpass skip error; do
cat $result.log # For debugging.
$FGREP "$pmarker $result $pmarker" my.log || st=1
$FGREP "$cmarker $result $cmarker" my.log || st=1
done
-test `$FGREP -c "$pmarker" my.log` -eq 4
-test `$FGREP -c "$cmarker" my.log` -eq 4
+test `$FGREP -c "$pmarker" my.log` -eq 5
+test `$FGREP -c "$cmarker" my.log` -eq 5
-# Successful test scripts shouldn't be mentioned in the global log.
-$EGREP '(^pass|[^x]pass|xfail)\.test' my.log && Exit 1
-# But failing and skipped one should.
+# Passed test scripts shouldn't be mentioned in the global log.
+$EGREP '(^pass|[^x]pass)\.test' my.log && Exit 1
+# But failing (expectedly or not) and skipped ones should.
+$FGREP 'xfail.test' my.log
$FGREP 'skip.test' my.log
$FGREP 'fail.test' my.log
$FGREP 'xpass.test' my.log
@@ -137,6 +139,7 @@ test ! -f xfail.log
test ! -f xpass.log
test ! -f skip.log
test ! -f error.log
+# "make mostlyclean" shouldn't remove unrelated log files.
test -f error2.log
test -f test-suite.log
test -f global.log
@@ -155,7 +158,7 @@ log=`cat global.log`
case $out in *"$log"*) ;; *) Exit 1;; esac
touch error2.log test-suite.log my.log
-$MAKE mostlyclean
+$MAKE clean
ls -l # For debugging.
test ! -f global.log
test ! -f pass.log
@@ -164,11 +167,11 @@ test ! -f xfail.log
test ! -f xpass.log
test ! -f skip.log
test ! -f error.log
+# "make clean" shouldn't remove unrelated log files.
test -f error2.log
test -f test-suite.log
test -f my.log
rm -f *.log
-
:
diff --git a/tests/tap-passthrough.test b/tests/tap-passthrough.test
index ab9b3e5..b4a7c12 100755
--- a/tests/tap-passthrough.test
+++ b/tests/tap-passthrough.test
@@ -62,15 +62,11 @@ ok 4 - x
# some diagnostic${tab}
not ok # TODO low priority
ok # SKIP who cares?
-This is not in test-suite.log, since ok.test does not fail.
$weirdchars
END
-st=0
-TESTS=ok.test $MAKE -e check || st=$?
+TESTS=ok.test $MAKE -e check || { cat ok.log; Exit 1; }
cat ok.log
-cat test-suite.log
-test $st -eq 0 || Exit 1
for rx in \
'1\.\.6' \
@@ -83,15 +79,11 @@ for rx in \
"# some diagnostic${tab}" \
'not ok # TODO low priority' \
'ok # SKIP who cares?' \
- 'This is not in test-suite\.log, since ok\.test does not fail\.' \
; do
grep "^$rx$" ok.log
done
$FGREP "$weirdchars" ok.log
-$FGREP 'ok.test' test-suite.log && Exit 1
-$FGREP 'not in test-suite.log' test-suite.log && Exit 1
-
#
# Mixed failing/successful tests.
#
@@ -104,7 +96,7 @@ END
cat > ok.test <<END
1..1
ok
-local only
+only one success here
END
cat > ko.test <<END
@@ -150,14 +142,11 @@ cat ko.log
cat bail.log
cat skip.log
cat err.log
-cat test-suite.log
test $st -gt 0 || Exit 1
grep '^1\.\.1$' tiny.log
grep '^ok$' tiny.log
-
-grep '^local only$' ok.log
-grep 'local only' test-suite.log && Exit 1
+grep '^only one success here$' ok.log
for rx in \
'1\.\.5' \
@@ -171,17 +160,11 @@ for rx in \
'ok # TODO dunno' \
; do
grep "^$rx$" ko.log
- grep "^$rx$" test-suite.log
done
$FGREP "$weirdchars" ko.log
-$FGREP "$weirdchars" test-suite.log
-for log in bail.log test-suite.log; do
- grep '^Bail out! Test is taking too long!$' $log
-done
-for log in skip.log test-suite.log; do
- grep '^1\.\.0 # Skipped: WWW::Mechanize not installed$' $log
-done
+grep '^Bail out! Test is taking too long!$' bail.log
+grep '^1\.\.0 # Skipped: WWW::Mechanize not installed$' skip.log
for rx in \
'^1\.\.3$' \
@@ -196,7 +179,6 @@ for rx in \
'^ERROR:.* err\.test 23 .*OUT[ -]OF[ -]ORDER.*expecting 2' \
; do
grep "$rx" err.log
- grep "$rx" test-suite.log
done
:
diff --git a/tests/test-driver-end-metadata.test
b/tests/test-driver-end-metadata.test
new file mode 100755
index 0000000..42e7e57
--- /dev/null
+++ b/tests/test-driver-end-metadata.test
@@ -0,0 +1,120 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers and parallel-tests harness: check the documented
+# semantics for inhibiting the scanning of the remaining portion of a
+# `.log' file (for what concerns metadata). Currently, this is done
+# with the use of the reStructuredText field `:end-metadata:'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = ./passthrough-driver
+TESTS = foo.test bar.test baz.test
+END
+
+cat > passthrough-driver <<'END'
+#!/bin/sh
+set -e; set -u;
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+echo "$test_name: RUN"
+cp $1 $log_file
+END
+chmod a+x passthrough-driver
+
+cat > foo.test <<END
+:end-metadata:
+:test-result: FAIL
+:copy-in-global-log: no
+:recheck: no
+foo foo foo
+END
+
+cat > bar.test <<END
+:test-result: PASS
+:recheck: no
+bar bar bar
+:copy-in-global-log: no
+:end-metadata: $tab
+:test-result: ERROR
+:end-metadata:
+END
+
+# `:end-metadata:' directives in other files shouldn't influence this one.
+cat > baz.test <<END
+:test-result: PASS
+:recheck: no
+:copy-in-global-log: no
+baz baz baz
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+st=0
+$MAKE check >stdout || st=1
+# For debugging.
+cat stdout
+cat foo.log
+cat bar.log
+cat test-suite.log
+
+# Check that only the `:test-result:' in baz.test and the first one in
+# foo.test have been considered.
+test $st -eq 0
+grep '^# TOTAL: *2$' stdout
+grep '^# PASS: *2$' stdout
+for result in SKIP XPASS FAIL XFAIL ERROR; do
+ grep "^# $result: *0$" stdout
+done
+
+# Check that a `:end-metadata:' directive does not prevent following text
+# to be copied into the log file.
+$FGREP 'foo foo foo' foo.log
+$FGREP 'bar bar bar' bar.log
+$FGREP 'baz baz baz' baz.log # More of a sanity check really.
+
+# Check that only the `:copy-in-global-log:' in bar.test has been
+# considered.
+$FGREP 'foo foo foo' test-suite.log
+$FGREP 'bar bar bar' test-suite.log && Exit 1
+$FGREP 'baz baz baz' test-suite.log && Exit 1
+
+# Check that only the `:recheck:' in bar.test has been
+# considered.
+$MAKE recheck >stdout || { cat stdout; Exit 1; }
+cat stdout
+grep '^foo\.test: RUN$' stdout
+grep 'ba[rz]\.test.*RUN' stdout && Exit 1
+
+:
diff --git a/tests/test-driver-end-test-results.test
b/tests/test-driver-end-test-results.test
deleted file mode 100755
index 7008700..0000000
--- a/tests/test-driver-end-test-results.test
+++ /dev/null
@@ -1,95 +0,0 @@
-#! /bin/sh
-# Copyright (C) 2011 Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# parallel-tests harness: check that a special ":test-result:END"
-# directive prevents the code generating the testsuite summary and
-# the test-suite.log from parsing the rest of the log file.
-
-parallel_tests=yes
-. ./defs || Exit 1
-
-cat >> configure.in << 'END'
-AC_OUTPUT
-END
-
-cat > Makefile.am << 'END'
-TEST_LOG_DRIVER = ./dummy-driver
-TESTS = a.test b.test c.test d.test
-END
-
-cat > dummy-driver <<'END'
-#!/bin/sh
-set -e
-while test $# -gt 0; do
- case $1 in
- --log-file) log_file=$2; shift;;
- --test-name|--expect-failure|--color-tests|--enable-hard-errors) shift;;
- --) shift; break;;
- *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
- esac
- shift
-done
-cp $1 $log_file
-END
-chmod a+x dummy-driver
-
-cat > a.test <<'END'
-PASS: a.test
-:test-result:END
-:test-result:FAIL
-END
-
-cat > b.test <<END
-PASS: b.test
-:test-result:PASS
-:test-result:END
-:test-result:FAIL
-:test-result:ERROR
-END
-
-cat > c.test <<END
-PASS: c.test
-:test-result:SKIP
-:test-result: END$tab $tab$tab
-:test-result:XPASS
-END
-
-cat > d.test <<END
-PASS: d.test
-:test-result:PASS
-:test-result:PASS
-:test-result: $tab END.
-:test-result:FAIL
-END
-
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE
-
-./configure
-
-$MAKE check >output 2>&1 || { cat output; Exit 1; }
-cat output
-cat a.log; cat b.log; cat c.log; cat d.log # For debugging.
-
-grep '^# TOTAL: *4$' output
-grep '^# PASS: *3$' output
-grep '^# SKIP: *1$' output
-for result in XPASS FAIL XFAIL ERROR; do
- grep "^# $result: *0$" output
-done
-
-:
diff --git a/tests/test-driver-global-log.test
b/tests/test-driver-global-log.test
index 35b6fe1..7596e94 100755
--- a/tests/test-driver-global-log.test
+++ b/tests/test-driver-global-log.test
@@ -14,8 +14,10 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check that the output of testcases having a global outcome that is not
-# "PASS" or "XFAIL" is copied in the global 'test-suite.log'.
+# Custom test drivers and parallel-tests harness: check the documented
+# semantics for deciding when the content of a test log file should be
+# copied in the global test-suite.log file. Currently, this is done
+# with the use of the reStructuredText field `:copy-in-global-log:'.
parallel_tests=yes
. ./defs || Exit 1
@@ -24,36 +26,69 @@ cat >> configure.in << 'END'
AC_OUTPUT
END
-# The :test-result: fields should be irrelevant for the decision
-# of whether a test output is to be copied in the test-suite.log.
-cat > pass.test <<END
-PASS: pass.test
+# The `:test-result:' and `:recheck:' fields and the first line of the
+# log file should be be irrelevant for the decision of whether a test
+# output is to be copied in the `test-suite.log'.
+
+cat > no-1.test <<END
+FAIL: no-1.test
:test-result: SKIP
+:copy-in-global-log: no
:test-result: FAIL
-:test-result: XFAIL
:test-result: XPASS
-:test-result: ERROR
-not seen (pass)
+not seen 1
+END
+
+for RES in XPASS FAIL XFAIL SKIP ERROR UNKNOWN; do
+ unindent > $RES.test <<END
+ $RES: $RES.test
+ :test-result: $RES
+ :copy-in-global-log: no
+ not seen $RES
END
+done
-cat > xfail.test <<'END'
-XFAIL: xfail.test
-not seen (xfail)
+# In the last line, with leading and trailing whitespace.
+cat > no-2.test <<END
+:test-result: FAIL
+not seen 2
+:recheck: yes
+:copy-in-global-log:$tab $tab no $tab
END
-echo SKIP: skip.test > skip.test
-echo FAIL: fail.test > fail.test
-echo XPASS: xpass.test > xpass.test
-echo ERROR: error.test > error.test
-echo :test-result: PASS > fake-pass.test
-echo "$tab $tab$tab" > empty.test
-
-cat > Makefile.am << 'END'
-TEST_LOG_DRIVER = ./dummy-driver
-TESTS = pass.test skip.test fail.test xfail.test xpass.test error.test \
- fake-pass.test empty.test
+# In the first line, with no whitespace.
+cat > no-3.test <<END
+:copy-in-global-log:no
+not seen 3
+:test-result: FAIL
END
+cat > yes-1.test <<END
+PASS: yes-1.test
+:test-result: PASS
+:copy-in-global-log: yes
+seen yes 1
+END
+
+# A lacking `:copy-in-global-log:' implies that the content of
+# the log file should be copied.
+cat > yes-2.test <<END
+PASS: y.test
+:test-result: PASS
+seen yes 2
+END
+
+# Corner cases.
+echo ':copy-in-global-log:' > corn-1.test
+echo " $tab $tab$tab" > corn-2.test
+cat > corn-3.test <<'END'
+seen corn 31
+:copy-in-global-log:address@hidden
+seen corn 32
+END
+
+echo TEST_LOG_DRIVER = ./dummy-driver > Makefile.am
+echo TESTS = *.test >> Makefile.am
cat > dummy-driver <<'END'
#!/bin/sh
@@ -78,35 +113,19 @@ $AUTOMAKE
./configure
-$MAKE check && { cat test-suite.log; Exit 1; }
+# We don't care about the exit status of "make check" here, that
+# should be checked in other tests.
+$MAKE check || :
cat test-suite.log
-grep '^SKIP: skip\.test$' test-suite.log
-grep '^FAIL: fail.test$' test-suite.log
-grep '^XPASS: xpass.test$' test-suite.log
-grep '^ERROR: error.test$' test-suite.log
-grep '^:test-result: PASS$' test-suite.log
-grep "^$tab $tab$tab$" test-suite.log
-$EGREP 'not seen' test-suite.log && Exit 1
-test `grep -c ':test-result:' test-suite.log` -eq 1
-
-cat > pass.test <<END
-PASS: pass.test
-:test-result: SKIP
-:test-result: XFAIL
-not seen (pass)
-END
-cat > xfail.test <<END
-XFAIL: xfail.test
-:test-result: SKIP
-:test-result: PASS
-not seen (xfail)
-END
-
-TESTS='pass.test skip.test xfail.test' $MAKE -e check \
- || { cat test-suite.log; Exit 1; }
-cat test-suite.log
-grep '^SKIP: skip\.test$' test-suite.log
-$EGREP ':test-result:|not seen' test-suite.log && Exit 1
+grep '^seen yes 1$' test-suite.log
+grep '^seen yes 2$' test-suite.log
+grep '^seen corn 31$' test-suite.log
+grep '^seen corn 32$' test-suite.log
+grep '^:copy-in-global-log:$' test-suite.log
+grep "^ $tab $tab$tab$" test-suite.log
+$FGREP 'not seen' test-suite.log && Exit 1
+$EGREP '^(XPASS|FAIL|SKIP|ERROR|UNKNOWN)' test-suite.log && Exit 1
+test `grep -c ':test-result:' test-suite.log` -eq 2
:
diff --git a/tests/test-driver-metadata-no-leading-space.test
b/tests/test-driver-metadata-no-leading-space.test
new file mode 100755
index 0000000..4c3f026
--- /dev/null
+++ b/tests/test-driver-metadata-no-leading-space.test
@@ -0,0 +1,94 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers and parallel-tests harness: check that, in the
+# log files, the reStructuredText fields for metadata declaration are
+# recognized only when placed at the beginning of a line.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = ./passthrough-driver
+TESTS = foo.test
+END
+
+cat > passthrough-driver <<'END'
+#!/bin/sh
+set -e; set -u;
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+echo "$test_name: RUN"
+cp $1 $log_file
+END
+chmod a+x passthrough-driver
+
+cat > foo.test <<END
+ $tab :test-result: FAIL
+ $tab :end-metadata:
+:test-result: PASS
+ $tab :copy-in-global-log: no
+ $tab :recheck: no
+foo foo foo
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+st=0
+$MAKE check >stdout || st=1
+# For debugging.
+cat stdout
+cat foo.log
+cat test-suite.log
+
+# Check that only the second `:test-result:' has been processed.
+test $st -eq 0
+grep '^# TOTAL: *1$' stdout
+grep '^# PASS: *1$' stdout
+for result in SKIP XPASS FAIL XFAIL ERROR; do
+ grep "^# $result: *0$" stdout
+done
+
+# Check that `:copy-in-global-log:' hasn't been processed.
+grep 'foo foo foo' test-suite.log
+
+# Check that `:recheck:' hasn't been processed.
+: > older
+$sleep
+$MAKE recheck >stdout || { cat stdout; Exit 1; }
+cat stdout
+grep '^foo\.test: RUN$' stdout
+grep 'foo foo foo' test-suite.log
+is_newest test-suite.log older
+is_newest foo.log older
+
+:
diff --git a/tests/test-driver-recheck.test b/tests/test-driver-recheck.test
index f9482b1..a8654c1 100755
--- a/tests/test-driver-recheck.test
+++ b/tests/test-driver-recheck.test
@@ -14,8 +14,10 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Test the "make recheck" semantics for custom test driver, as
-# documented in detail in the Automake manual.
+# Test the "make recheck" semantics for custom test drivers, as documented
+# in the Automake manual. Currently, that is based on the reStructuredText
+# field `:recheck:'.
+
parallel_tests=yes
. ./defs || Exit 1
@@ -30,54 +32,86 @@ TEST_EXTENSIONS =
TESTS =
END
-rechecked=''
-notrechecked=''
-
-for R in PASS XFAIL SKIP; do
- echo $R: > $R-1
- echo " $R:" > $R-2
- echo $R:foo > $R-3
- echo "$tab $R: bar baz" > $R-4
- # The :test-result: fields should be irrelevant for the decision of
- # whether "make recheck" should or should not re-run a test.
- unindent > $R-5 <<END
- $R:
- :test-result: FAIL
- :test-result: XPASS
- :test-result: ERROR
+#
+# Tests to be re-run by "make recheck"
+#
+
+: > y-1
+echo "foo bar" > y-2
+echo ":recheck:" > y-3
+echo ":recheck:yes" > y-4
+echo ":recheck: who cares" > y-5
+echo ":recheck: $tab y" > y-6
+echo ":recheck: yeah!$tab$tab " > y-7
+cat > y-10 <<END
+foo
+:recheck: ???
END
- notrechecked="$notrechecked $R-1 $R-2 $R-3 $R-4 $R-5"
- echo $R > BAD-$R-1
- echo $R. > BAD-$R-2
- echo :$R: > BAD-$R-3
- echo $R asd > BAD-$R-4
- rechecked="$rechecked BAD-$R-1 BAD-$R-2 BAD-$R-3 BAD-$R-4"
-done
+cat > y-11 <<END
+:recheck: YES
+foo
+END
+cat > y-12 <<END
+foo
+:recheck:yes
-for R in FAIL XPASS ERROR UNKNOWN; do
- echo $R: > $R-1
- echo $R:foo > $R-2
- echo $R: bar baz > $R-3
- echo $R > $R-4
- echo $R asd > $R-5
- # The :test-result: fields should be irrelevant for the decision of
- # whether "make recheck" should or should not re-run a test.
- unindent > $R-6 <<END
- :test-result: PASS
- :test-result: SKIP
- :test-result: XFAIL
+bar
+zardoz
END
- (echo $R: && cat $R-6) > $R-7
- rechecked="$rechecked $R-1 $R-2 $R-3 $R-4 $R-5 $R-6 $R-7"
-done
-: > EMPTY
-echo " $tab $tab" > WHITE
-rechecked="$rechecked EMPTY WHITE"
+# The :test-result: fields and the fist line of the log should be
+# irrelevant for the decision of whether "make recheck" should or
+# should not re-run a test.
+
+echo ":test-result: PASS" > y-100
+
+echo "PASS: y-101"
+
+cat > y-102 <<END
+PASS: y-102
+===========
+
+:test-result: PASS
+END
+
+#
+# Tests *not* to be re-run by "make recheck"
+#
+
+echo ":recheck:no" > n-1
+echo ":recheck: no " > n-2
+echo ":recheck: $tab no" > n-3
+echo ":recheck: no $tab$tab " > n-4
+cat > n-5 <<END
+foo
+:recheck:no
+END
+cat > n-6 <<END
+:recheck: no
+foo
+END
+cat > n-7 <<END
+foo
+:recheck: no$tab$tab
+
+bar
+zardoz
+END
+
+# The :test-result: fields should be irrelevant for the decision of
+# whether "make recheck" should or should not re-run a test.
+cat > n-100 <<END
+:test-result: FAIL
+:test-result: XPASS
+:test-result: ERROR
+:test-result: UNKNOWN
+:recheck: no
+END
-tests="$rechecked $notrechecked"
+rechecked=`echo y-[0-9]*`
-for t in $tests; do echo $t; done | sed 's/.*/TESTS += &/' >> Makefile.am
+for t in [yn]-[0-9]*; do echo $t; done \
+ | sed 's/.*/TESTS += &/' >> Makefile.am
cat Makefile.am # For debugging.
@@ -119,8 +153,7 @@ for iteration in 1 2; do
$MAKE recheck
ls -l
for t in $rechecked; do test -f $t.run; done
- for t in $notrechecked; do test ! -r $t.run; done
- rm -f *.run
+ find . -name 'n-*.run' | grep . && Exit 1
done
:
diff --git a/tests/test-log.test b/tests/test-log.test
new file mode 100755
index 0000000..1385386
--- /dev/null
+++ b/tests/test-log.test
@@ -0,0 +1,168 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features:
+# - log file creation
+# - log file removal
+# - stdout and stderr of a test script go in its log file
+# - TEST_SUITE_LOG redefinition, at either automake or make time
+# - VERBOSE environment variable support
+# Keep in sync with 'tap-log.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<END
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TESTS = pass.test skip.test xfail.test fail.test xpass.test error.test
+XFAIL_TESTS = xpass.test xfail.test
+TEST_SUITE_LOG = global.log
+END
+
+# Custom markers, for use in grepping checks.
+cmarker=::: # comment marker
+pmarker=%%% # plain maker
+
+cat > pass.test <<END
+#! /bin/sh
+echo "$pmarker pass $pmarker" >&2
+echo "# $cmarker pass $cmarker" >&2
+exit 0
+END
+
+cat > skip.test <<END
+#! /bin/sh
+echo "$pmarker skip $pmarker"
+echo "# $cmarker skip $cmarker"
+exit 1
+END
+
+cat > xfail.test <<END
+#! /bin/sh
+echo "$pmarker xfail $pmarker" >&2
+echo "# $cmarker xfail $cmarker" >&2
+exit 1
+END
+
+cat > fail.test <<END
+#! /bin/sh
+echo 1..1
+echo "$pmarker fail $pmarker"
+echo "# $cmarker fail $cmarker"
+exit 1
+END
+
+cat > xpass.test <<END
+#! /bin/sh
+echo "$pmarker xpass $pmarker" >&2
+echo "# $cmarker xpass $cmarker" >&2
+exit 0
+END
+
+cat > error.test <<END
+#! /bin/sh
+echo "$pmarker error $pmarker"
+echo "# $cmarker error $cmarker"
+exit 99
+END
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+TEST_SUITE_LOG=my.log $MAKE -e check && Exit 1
+ls -l # For debugging.
+test ! -f test-suite.log
+test ! -f global.log
+test -f my.log
+st=0
+for result in pass fail xfail xpass skip error; do
+ cat $result.log # For debugging.
+ $FGREP "$pmarker $result $pmarker" $result.log || st=1
+ $FGREP "$cmarker $result $cmarker" $result.log || st=1
+done
+test $st -eq 0 || Exit 1
+cat my.log # For debugging.
+for result in xfail fail xpass skip error; do
+ cat $result.log # For debugging.
+ $FGREP "$pmarker $result $pmarker" my.log || st=1
+ $FGREP "$cmarker $result $cmarker" my.log || st=1
+done
+test `$FGREP -c "$pmarker" my.log` -eq 5
+test `$FGREP -c "$cmarker" my.log` -eq 5
+
+# Passed test scripts shouldn't be mentioned in the global log.
+$EGREP '(^pass|[^x]pass)\.test' my.log && Exit 1
+# But failing (expectedly or not) and skipped ones should.
+$FGREP 'xfail.test' my.log
+$FGREP 'skip.test' my.log
+$FGREP 'fail.test' my.log
+$FGREP 'xpass.test' my.log
+$FGREP 'error.test' my.log
+
+touch error2.log test-suite.log global.log
+TEST_SUITE_LOG=my.log $MAKE -e mostlyclean
+ls -l # For debugging.
+test ! -f my.log
+test ! -f pass.log
+test ! -f fail.log
+test ! -f xfail.log
+test ! -f xpass.log
+test ! -f skip.log
+test ! -f error.log
+# "make mostlyclean" shouldn't remove unrelated log files.
+test -f error2.log
+test -f test-suite.log
+test -f global.log
+
+rm -f *.log
+
+VERBOSE=yes $MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+cat global.log
+test ! -f my.log
+test ! -f test-suite.log
+# Check that VERBOSE causes the global testsuite log to be
+# emitted on stdout.
+out=`cat stdout`
+log=`cat global.log`
+case $out in *"$log"*) ;; *) Exit 1;; esac
+
+touch error2.log test-suite.log my.log
+$MAKE clean
+ls -l # For debugging.
+test ! -f global.log
+test ! -f pass.log
+test ! -f fail.log
+test ! -f xfail.log
+test ! -f xpass.log
+test ! -f skip.log
+test ! -f error.log
+# "make clean" shouldn't remove unrelated log files.
+test -f error2.log
+test -f test-suite.log
+test -f my.log
+
+rm -f *.log
+
+:
diff --git a/tests/trivial-test-driver b/tests/trivial-test-driver
index 4b43506..d334a93 100644
--- a/tests/trivial-test-driver
+++ b/tests/trivial-test-driver
@@ -86,8 +86,12 @@ tmp_status=$log_file-status.tmp
if test ! -s $tmp_status; then
global_result=PASS
+ recheck=no
+ copy_in_global_log=no
else
global_result=FAIL
+ recheck=yes
+ copy_in_global_log=yes
fi
## Write the log file.
@@ -95,6 +99,11 @@ fi
{
echo "$global_result: $test_name"
echo "$global_result: $test_name" | sed 's/./=/g'
+ # Blank lines required for having the output be valid RST.
+ echo
+ echo ":recheck: $recheck"
+ echo
+ echo ":copy-in-global-log: $copy_in_global_log"
echo
cat $tmp_results
echo
--
1.7.2.3
- [GSoC] Some patches for testsuite harness improvements and TAP support introduction, Stefano Lattarini, 2011/07/14
- [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/14
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/18
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/19
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/19
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/20
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/20
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/21
- [PATCH 1/2] test harness: allow more metadata in log files,
Stefano Lattarini <=
- Re: [PATCH 1/2] test harness: allow more metadata in log files, Stefano Lattarini, 2011/07/31
[PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR', Stefano Lattarini, 2011/07/14
[PATCH 3/5] {test-protocols} parallel-tests: simplify testsuite summary, Stefano Lattarini, 2011/07/14
[PATCH 4/5] {test-protocols} tests defs: new auxiliary function 'count_test_results', Stefano Lattarini, 2011/07/14