[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Automake-commit] [SCM] GNU Automake branch, test-protocols, updated. v1
|
From: |
Stefano Lattarini |
|
Subject: |
[Automake-commit] [SCM] GNU Automake branch, test-protocols, updated. v1.11-896-g1402741 |
|
Date: |
Wed, 29 Jun 2011 13:55:48 +0000 |
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU Automake".
http://git.sv.gnu.org/gitweb/?p=automake.git;a=commitdiff;h=1402741313cf6c2fe8611eb294f9a8111c994a32
The branch, test-protocols has been updated
via 1402741313cf6c2fe8611eb294f9a8111c994a32 (commit)
via 3605250b571309a41f52ba110c1f14d2a2ca25f7 (commit)
via 47f596af02a6229dbc87ec1e3317a7a471e0a160 (commit)
via 3e334a272fa601bebb5896e25cfb63f34822a275 (commit)
via 69bb25618c4f3a14e5b405822d726620e955b07b (commit)
via b1109e2f9164e850dea0fea1f009d3265e51e5c8 (commit)
via c9e39919f0afea01e55795050d4978f530e086e2 (commit)
via d49776091458ca8affb5defa3c766c354bb09efe (commit)
via 404882ab2ac0a9447f6eccb2fa218df70582f488 (commit)
via 9667c39a2cd9c1af4be172169031975febe78657 (commit)
via 5755b7bcbd3892eb6aa31542fc7e2435a0de11ab (commit)
via 1f2e5599a81965c3d7da88995f70455e8fcb8e30 (commit)
via c1040a74c10a9a011a04ad0d347b61465c5ee194 (commit)
via 9478566087f83d1fac2c9a7629898a3d39a05fdd (commit)
via cb3e1296ac8a01fcf4b38d10584f218722655e1c (commit)
via 72d36af268fa7bee24466e21d2d543d8d1ce7b1c (commit)
via 9b61494445c234e505a2efea8e1c0384591edd2f (commit)
via e8bcc4800b17c3fd64b951e0136e7769693edbc8 (commit)
via 4e9474e6e9b18723d3fb02858824682dbd0e03f0 (commit)
via e4a70cff310e29066e388aa18aeb07242e003eec (commit)
via f31a38aed9c8ffbd33d797aa2df5b5864860bb17 (commit)
via 5c237a6208c74eb445613e5237b4749d62f18b28 (commit)
via a9eef973b5ea47cc3495f1a8307d4f7b85aea46f (commit)
via 42a465249b6d3425efbf23df8f2b62fc0e088cfb (commit)
from c96b881f41b180f6176064cc00fe7af620c24c63 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit 1402741313cf6c2fe8611eb294f9a8111c994a32
Author: Stefano Lattarini <address@hidden>
Date: Wed Jun 22 21:59:34 2011 +0200
docs: document custom test drivers and protocols
* doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
use suggested here is not portable to 'parallel-tests'.
(Simple Tests using parallel-tests): Document new restrictions on
the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
(Custom Test Drivers): New section and node.
(Overview of Custom Test Drivers Support): New subsection.
(Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
(APIs for Custom Test Drivers): Likewise.
(Options): Update description of color-tests.
* lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
moved into the manual.
(recheck, recheck-html): Minor adjustments to better conform to the
documentation (this should cause no semantic changes w.r.t. the
former behaviour); minor improvements and extensions to existing
comments.
* tests/test-driver-create-log-dir.test: New test.
* tests/test-driver-strip-vpath.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/Makefile.am (TESTS): Update.
commit 3605250b571309a41f52ba110c1f14d2a2ca25f7
Merge: c96b881 47f596a
Author: Stefano Lattarini <address@hidden>
Date: Wed Jun 29 10:06:20 2011 +0200
Merge branch 'master' into test-protocols
* master:
docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
docs: avoid a footnote, some related rewordings and improvements
Revert "docs: parallel-tests is not experimental anymore"
docs: minor cosmetic fixes
help: improve text about automatically-distributed files
refactor: split 'usage' subroutine in automake
tests: fix bug in 'autodist.test'
parallel-tests: stricter checks on DISABLE_HARD_ERRORS support
docs: parallel-tests is not experimental anymore
check: document and test $(TEST_SUITE_LOG) overriding
commit 47f596af02a6229dbc87ec1e3317a7a471e0a160
Merge: b1109e2 3e334a2
Author: Stefano Lattarini <address@hidden>
Date: Wed Jun 29 09:53:31 2011 +0200
Merge branch 'am-tests-environment'
* am-tests-environment:
docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
commit 3e334a272fa601bebb5896e25cfb63f34822a275
Author: Stefano Lattarini <address@hidden>
Date: Wed Jun 29 09:44:14 2011 +0200
docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
* doc/automake.texi (Simple Tests using parallel-tests): Ditto, and
related adjustments.
Suggestion by Ralf Wildenhues.
commit 69bb25618c4f3a14e5b405822d726620e955b07b
Author: Stefano Lattarini <address@hidden>
Date: Thu Jun 23 19:05:44 2011 +0200
docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
* doc/automake.texi (Simple Tests using parallel-tests): The
old example on AM_TESTS_ENVIRONMENT relied on unportable shell
features, and in particular didn't work with various Korn
Shells (see also commit `v1.11-925-g29ca903'). Give another
example, simpler this time, but still inspired to real-world
usage (the GNU coreutils testsuite).
-----------------------------------------------------------------------
Summary of changes:
ChangeLog | 110 ++++++
automake.in | 95 +++---
doc/automake.texi | 455 +++++++++++++++++++++---
lib/Automake/tests/Makefile.in | 2 +-
lib/am/check.am | 15 +-
tests/Makefile.am | 9 +
tests/Makefile.in | 11 +-
tests/autodist-no-duplicate.test | 52 +++
tests/autodist-subdir.test | 27 +--
tests/autodist.test | 31 +--
tests/parallel-tests-harderror.test | 98 +++++
tests/parallel-tests-log-override-1.test | 112 ++++++
tests/parallel-tests-log-override-2.test | 88 +++++
tests/parallel-tests-log-override-recheck.test | 92 +++++
tests/parallel-tests.test | 18 +-
tests/test-driver-create-log-dir.test | 74 ++++
tests/test-driver-global-log.test | 110 ++++++
tests/test-driver-recheck.test | 126 +++++++
tests/test-driver-strip-vpath.test | 96 +++++
19 files changed, 1451 insertions(+), 170 deletions(-)
create mode 100755 tests/autodist-no-duplicate.test
create mode 100755 tests/parallel-tests-harderror.test
create mode 100755 tests/parallel-tests-log-override-1.test
create mode 100755 tests/parallel-tests-log-override-2.test
create mode 100755 tests/parallel-tests-log-override-recheck.test
create mode 100755 tests/test-driver-create-log-dir.test
create mode 100755 tests/test-driver-global-log.test
create mode 100755 tests/test-driver-recheck.test
create mode 100755 tests/test-driver-strip-vpath.test
diff --git a/ChangeLog b/ChangeLog
index d6dbd13..6203f89 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,93 @@
+2011-06-29 Stefano Lattarini <address@hidden>
+
+ docs: document custom test drivers and protocols
+ * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
+ use suggested here is not portable to 'parallel-tests'.
+ (Simple Tests using parallel-tests): Document new restrictions on
+ the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
+ (Custom Test Drivers): New section and node.
+ (Overview of Custom Test Drivers Support): New subsection.
+ (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
+ (APIs for Custom Test Drivers): Likewise.
+ (Options): Update description of color-tests.
+ * lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
+ moved into the manual.
+ (recheck, recheck-html): Minor adjustments to better conform to the
+ documentation (this should cause no semantic changes w.r.t. the
+ former behaviour); minor improvements and extensions to existing
+ comments.
+ * tests/test-driver-create-log-dir.test: New test.
+ * tests/test-driver-strip-vpath.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-06-29 Stefano Lattarini <address@hidden>
+
+ docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
+ * doc/automake.texi (Simple Tests using parallel-tests): Ditto, and
+ related adjustments.
+ Suggestion by Ralf Wildenhues.
+
+2011-06-29 Stefano Lattarini <address@hidden>
+
+ docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
+ * doc/automake.texi (Simple Tests using parallel-tests): The
+ old example on AM_TESTS_ENVIRONMENT relied on unportable shell
+ features, and in particular didn't work with various Korn
+ Shells (see also commit `v1.11-925-g29ca903'). Give another
+ example, simpler this time, but still inspired to real-world
+ usage (the GNU coreutils testsuite).
+
+2011-06-23 Stefano Lattarini <address@hidden>
+
+ docs: avoid a footnote, some related rewordings and improvements
+ * doc/automake.texi (Dist): Reword the part about automatically
+ distributed files to avoid a footnote. Since we are at it, extend
+ a bit, and add an example and a reference to a relevant test case.
+
+2011-06-23 Stefano Lattarini <address@hidden>
+
+ docs: minor cosmetic fixes
+ * doc/automake.texi: Break few overly long lines, throughout the
+ file.
+ ("Simple Tests"): Move @vindex for XFAIL_TESTS to the correct
+ position, i.e., before and not after the paragraph where it is
+ introduced.
+ ("Options" @item ansi2knr): Use @pxref instead of @xref. This
+ fixes a texinfo warning.
+ ("Other things Automake recognizes" @item AM_C_PROTOTYPES): Use
+ @pxref instead of @ref.
+
+2011-06-23 Stefano Lattarini <address@hidden>
+
+ help: improve text about automatically-distributed files
+ This change fixes automake bug#7819.
+ * automake.in (usage): Distinguish between files that are always
+ automatically distributed when found, and those which are only
+ "under certain conditions".
+ * doc/automake.texi (Basics of Distribution): Update accordingly.
+ * tests/autodist-subdir.test: Update.
+ * tests/autodist-no-duplicate.test: Likewise.
+ * tests/autodist.test: Likewise.
+ (configure.in): Remove useless call to AM_MAINTAINER_MODE.
+
+2011-06-23 Stefano Lattarini <address@hidden>
+
+ refactor: split 'usage' subroutine in automake
+ This change is related to automake bug#7819.
+ * automake.in (print_autodist_files): New subroutine,
+ extracted from ...
+ (usage): ... this, which now uses it.
+ * tests/autodist-no-duplicate.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-06-23 Stefano Lattarini <address@hidden>
+
+ tests: fix bug in 'autodist.test'
+ * tests/autodist.test: Avoid spurious failure due to no
+ `defs-static' file being found in the parent directory.
+
2011-06-21 Stefano Lattarini <address@hidden>
parallel-tests: allow each test to have multiple results
@@ -439,6 +529,15 @@
($AUTORECONF): New variable, defaulting to address@hidden@'.
* doc/Makefile.am ($(srcdir)/amhello-1.0.tar.gz): Update.
+2011-05-22 Stefano Lattarini <address@hidden>
+
+ parallel-tests: stricter checks on DISABLE_HARD_ERRORS support
+ * tests/parallel-tests-harderror.test: New test, doing more
+ in-depth checks on DISABLE_HARD_ERRORS.
+ * tests/parallel-tests.test: Remove tests on DISABLE_HARD_ERRORS,
+ now redundant.
+ * tests/Makefile.am (TESTS): Update.
+
2011-05-22 Ralf Wildenhues <address@hidden>
Stefano Lattarini <address@hidden>
@@ -524,6 +623,17 @@
have `.o' suffix and executables have no default suffix; instead,
use `$(OBJEXT)' and `$(EXEEXT)'.
+2011-05-11 Stefano Lattarini <address@hidden>
+
+ check: document and test $(TEST_SUITE_LOG) overriding
+ * doc/automake.texi (Simple Tests using parallel-tests): Explain
+ how and why TEST_SUITE_LOG can be overridden at runtime.
+ * tests/parallel-tests-log-override-1.test: New test, check that
+ the newly documented idiom and similar usages are truly supported.
+ * tests/parallel-tests-log-override-2.test: Likewise.
+ * tests/parallel-tests-log-override-recheck.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
2011-05-07 Stefano Lattarini <address@hidden>
tests: fix spurious failure of txinfo21.test on FreeBSD
diff --git a/automake.in b/automake.in
index 340d4e3..2ad8dc2 100644
--- a/automake.in
+++ b/automake.in
@@ -8398,6 +8398,50 @@ sub generate_makefile ($$)
################################################################
+# Helper function for usage().
+sub print_autodist_files (@)
+{
+ my @lcomm = sort (&uniq (@_));
+
+ my @four;
+ format USAGE_FORMAT =
+ @<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<
+ $four[0], $four[1], $four[2], $four[3]
+.
+ local $~ = "USAGE_FORMAT";
+
+ my $cols = 4;
+ my $rows = int(@lcomm / $cols);
+ my $rest = @lcomm % $cols;
+
+ if ($rest)
+ {
+ $rows++;
+ }
+ else
+ {
+ $rest = $cols;
+ }
+
+ for (my $y = 0; $y < $rows; $y++)
+ {
+ @four = ("", "", "", "");
+ for (my $x = 0; $x < $cols; $x++)
+ {
+ last if $y + 1 == $rows && $x == $rest;
+
+ my $idx = (($x > $rest)
+ ? ($rows * $rest + ($rows - 1) * ($x - $rest))
+ : ($rows * $x));
+
+ $idx += $y;
+ $four[$x] = $lcomm[$idx];
+ }
+ write;
+ }
+}
+
+
# Print usage information.
sub usage ()
{
@@ -8431,51 +8475,12 @@ Library files:
";
Automake::ChannelDefs::usage;
- my ($last, @lcomm);
- $last = '';
- foreach my $iter (sort ((@common_files, @common_sometimes)))
- {
- push (@lcomm, $iter) unless $iter eq $last;
- $last = $iter;
- }
-
- my @four;
- print "\nFiles which are automatically distributed, if found:\n";
- format USAGE_FORMAT =
- @<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<
- $four[0], $four[1], $four[2], $four[3]
-.
- $~ = "USAGE_FORMAT";
-
- my $cols = 4;
- my $rows = int(@lcomm / $cols);
- my $rest = @lcomm % $cols;
-
- if ($rest)
- {
- $rows++;
- }
- else
- {
- $rest = $cols;
- }
-
- for (my $y = 0; $y < $rows; $y++)
- {
- @four = ("", "", "", "");
- for (my $x = 0; $x < $cols; $x++)
- {
- last if $y + 1 == $rows && $x == $rest;
-
- my $idx = (($x > $rest)
- ? ($rows * $rest + ($rows - 1) * ($x - $rest))
- : ($rows * $x));
-
- $idx += $y;
- $four[$x] = $lcomm[$idx];
- }
- write;
- }
+ print "\nFiles automatically distributed if found " .
+ "(always):\n";
+ print_autodist_files @common_files;
+ print "\nFiles automatically distributed if found " .
+ "(under certain conditions):\n";
+ print_autodist_files @common_sometimes;
print '
Report bugs to <@PACKAGE_BUGREPORT@>.
diff --git a/doc/automake.texi b/doc/automake.texi
index dd3a340..a26f2a5 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -84,7 +84,8 @@ section entitled ``GNU Free Documentation License.''
@c Put the macros in the function index.
@syncodeindex ac fn
address@hidden Put everything else into one index (arbitrarily chosen to be the
concept index).
address@hidden Put everything else into one index (arbitrarily chosen to be the
address@hidden concept index).
@syncodeindex op cp
@syncodeindex tr cp
@syncodeindex cm cp
@@ -318,9 +319,23 @@ Support for test suites
* Simple Tests:: Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
+* Custom Test Drivers:: Writing and using custom test drivers
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
+Custom Test Drivers
+
+* Overview of Custom Test Drivers Support::
+* Declaring Custom Test Drivers::
+* API for Custom Test Drivers::
+
+API for Custom Test Drivers
+
+* Command-line arguments for test drivers::
+* Log files generation and test results recording::
+* Testsuite progress output::
+* HTML generation from testsuite logs::
+
Changing Automake's Behavior
* Options generalities:: Semantics of Automake option
@@ -3052,8 +3067,9 @@ this way, e.g., @code{AC_PATH_XTRA} defines
@code{X_CFLAGS} and
if @code{AC_PATH_XTRA} is called.
@item AM_C_PROTOTYPES
-This is required when using the deprecated de-ANSI-fication feature; see
address@hidden @emph{It will be removed} in the next major Automake release.
+This is required when using the deprecated de-ANSI-fication feature;
address@hidden @emph{It will be removed} in the next major Automake
+release.
@item AM_CONDITIONAL
This introduces an Automake conditional (@pxref{Conditionals}).
@@ -5871,8 +5887,8 @@ needs to be avoided: when a target should not be built
from sources.
We already saw such an example in @ref{true}; this happens when all
the constituents of a target have already been compiled and just need
to be combined using a @code{_LDADD} variable. Then it is necessary
-to define an empty @code{_SOURCES} variable, so that @command{automake} does
not
-compute a default.
+to define an empty @code{_SOURCES} variable, so that @command{automake}
+does not compute a default.
@example
bin_PROGRAMS = target
@@ -6855,8 +6871,9 @@ If no directory prefix is given, the files are assumed to
be in the
current directory.
Note that automatic de-ANSI-fication will not work when the package is
-being built for a different host architecture. That is because
@command{automake}
-currently has no way to build @command{ansi2knr} for the build machine.
+being built for a different host architecture. That is because
address@hidden currently has no way to build @command{ansi2knr}
+for the build machine.
@c FIXME: this paragraph might be better moved to an `upgrading' section.
@cindex @code{LTLIBOBJS} and @code{ansi2knr}
@@ -6950,10 +6967,10 @@ bin_PROGRAMS = liver$(EXEEXT)
The targets Automake generates are likewise given the @samp{$(EXEEXT)}
extension.
-The variables @code{TESTS} and @code{XFAIL_TESTS} (@pxref{Simple Tests}) are
also
-rewritten if they contain filenames that have been declared as programs
-in the same @file{Makefile}. (This is mostly useful when some programs
-from @code{check_PROGRAMS} are listed in @code{TESTS}.)
+The variables @code{TESTS} and @code{XFAIL_TESTS} (@pxref{Simple Tests})
+are also rewritten if they contain filenames that have been declared as
+programs in the same @file{Makefile}. (This is mostly useful when some
+programs from @code{check_PROGRAMS} are listed in @code{TESTS}.)
However, Automake cannot apply this rewriting to @command{configure}
substitutions. This means that if you are conditionally building a
@@ -8333,14 +8350,19 @@ is run. The default setting is @option{--best}.
@cmindex include
For the most part, the files to distribute are automatically found by
Automake: all source files are automatically included in a distribution,
-as are all @file{Makefile.am}s and @file{Makefile.in}s. Automake also
+as are all @file{Makefile.am} and @file{Makefile.in} files. Automake also
has a built-in list of commonly used files that are automatically
included if they are found in the current directory (either physically,
-or as the target of a @file{Makefile.am} rule). This list is printed by
address@hidden --help}. Also, files that are read by @command{configure}
+or as the target of a @file{Makefile.am} rule); this list is printed by
address@hidden --help}. Note that some files in this list are actually
+distributed only if other certain conditions hold (for example,
address@hidden The following example is covered by autodist-config-headers.test.
+the @file{config.h.top} and @file{config.h.bot} files are automatically
+distributed only if, e.g., @samp{AC_CONFIG_HEADERS([config.h])} is used
+in @file{configure.ac}). Also, files that are read by @command{configure}
(i.e.@: the source files corresponding to the files specified in various
Autoconf macros such as @code{AC_CONFIG_FILES} and siblings) are
-automatically distributed. Files included in @file{Makefile.am}s (using
+automatically distributed. Files included in a @file{Makefile.am} (using
@code{include}) or in @file{configure.ac} (using @code{m4_include}), and
helper scripts installed with @samp{automake --add-missing} are also
distributed.
@@ -8633,6 +8655,7 @@ are very similar.
@menu
* Simple Tests:: Listing programs and scripts in @code{TESTS}
* Simple Tests using parallel-tests:: More powerful test driver
+* Custom Test Drivers:: Writing and using custom test drivers
* DejaGnu Tests:: Interfacing with the external testing framework
* Install Tests:: Running tests on installed packages
@end menu
@@ -8662,7 +8685,9 @@ given test program exits with a status of 77, then its
result is ignored
in the final count. This feature allows non-portable tests to be
ignored in environments where they don't make sense.
address@hidden tests and color-tests}
@vindex AM_COLOR_TESTS
address@hidden Colorized testsuite output
If the Automake option @code{color-tests} is used (@pxref{Options})
and standard output is connected to a capable terminal, then the test
results and the summary are colored appropriately. The user can disable
@@ -8691,19 +8716,21 @@ TESTS_ENVIRONMENT = $(PERL) -Mstrict -w
TESTS = foo.pl bar.pl baz.pl
@end example
-Note that the @option{parallel-tests} driver provides a more elegant
-way to achieve the same effect, freeing the @code{TESTS_ENVIRONMENT}
-variable for the user to override (@pxref{Simple Tests using
-parallel-tests}).
+It's important to note that this last use of @code{TESTS_ENVIRONMENT}
+is invalid when the @option{parallel-tests} option is active; however,
+the @option{parallel-tests} driver provides a more elegant way to
+achieve the same effect, with the further benefit of freeing the
address@hidden variable for the user
+(@pxref{Simple Tests using parallel-tests}).
@cindex Tests, expected failure
@cindex Expected test failure
address@hidden XFAIL_TESTS
You may define the variable @code{XFAIL_TESTS} to a list of tests
(usually a subset of @code{TESTS}) that are expected to fail. This will
reverse the result of those tests.
address@hidden XFAIL_TESTS
Automake ensures that each file listed in @code{TESTS} is built before
any tests are run; you can list both source and derived programs (or
@@ -8739,7 +8766,9 @@ for exceptional failures. Similar to the simple test
driver,
the @code{check_*} variables are honored, and the environment variable
@env{srcdir} is set during test execution. Also, @code{TESTS_ENVIRONMENT}
is still honored, but is complemented by a new developer-reserved variable
address@hidden (described below).
address@hidden (described below), and its allowed uses are
+somewhat restricted (other features of the @option{parallel-tests} driver
+compensate for this, though).
This test driver is still experimental and may undergo changes in order
to satisfy additional portability requirements.
@@ -8812,26 +8841,45 @@ and @samp{./wrapper-script -d baz} to produce
@file{foo.log},
but should be reserved for the user.
@vindex AM_TESTS_ENVIRONMENT
-The @code{AM_TESTS_ENVIRONMENT} variable can be used to run initialization
-code and set environment variables for the tests' runs. The user can
-still employ the @code{TESTS_ENVIRONMENT} variable to override settings
-from @code{AM_TESTS_ENVIRONMENT}. Note that, for implementation reasons,
-if the @code{AM_TESTS_ENVIRONMENT} variable is set, its contents
address@hidden be terminated by a semicolon.
-
address@hidden
-# The tests below are expected to use the file descriptor passed in
-# the environment variable 'warn_fileno' to print warnings (e.g.,
-# about skipped and failed tests). If this variable were to be set
-# to `2' (i.e. default stderr), the warnings would be redirected by
-# the automake parallel-tests driver into the .log files. But the
-# AM_TESTS_ENVIRONMENT definition below will cause the reasons for
-# skip/failure to be printed to the console instead. The user
-# can still override this by setting TESTS_ENVIRONMENT to e.g.
-# `warn_fileno=2' at make runtime, which will cause the warnings
-# to be sent to the .log files again.
-TESTS = test1.sh test2.sh ...
-AM_TESTS_ENVIRONMENT = exec 9>&2; warn_fileno=9; export warn_fileno;
+The @code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables can
+be used to run initialization code and set environment variables for the
+test scripts. The former variable is developer-reserved, and can be
+defined in the @file{Makefile.am}, while the latter is reserved for the
+user, which can employ it to extend or override the settings in the
+former; for this to work portably, however, the contents of a non-empty
address@hidden @emph{must} be terminated by a semicolon.
+
address@hidden
+AM_TESTS_ENVIRONMENT = \
+## Some environment initializations are kept in a separate shell file
+## `tests-env.sh', which can make it easier to also run tests from
+## the command line.
+ . $(srcdir)/tests-env.sh; \
+## On Solaris, prefer more POSIX-compliant versions of the standard
+## tools by default.
+ if test -d /usr/xpg4/bin; then \
+ PATH=/usr/xpg4/bin:$$PATH; export PATH; \
+ fi;
address@hidden $$ restore font-lock
address@hidden example
+
+It's important to note that, differently from what we've seen for the
+serial testsuite driver (@pxref{Simple Tests using parallel-tests}), the
address@hidden and @code{TESTS_ENVIRONMENT} variables
address@hidden be use to define a custom test driver; the
address@hidden and @code{LOG_FLAGS} (or their extension-specific
+counterparts) should be used instead:
+
address@hidden
+## This is WRONG!
+AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib' $(PERL) -Mstrict -w
address@hidden example
+
address@hidden
+## Do this instead.
+AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib'; export PERL5LIB;
+LOG_COMPILER = $(PERL)
+AM_LOG_FLAGS = -Mstrict -w
@end example
@trindex mostlyclean
@@ -8872,6 +8920,20 @@ use a command like this to run only a subset of the
tests:
env TESTS="foo.test bar.test" make -e check
@end example
+Note however that the command above will unconditionally overwrite the
address@hidden file, thus clobbering the recorded results
+of any previous testsuite run. This might be undesirable for packages
+whose testsuite takes long time to execute. Luckily, this problem can
+easily be avoided by overriding also @code{TEST_SUITE_LOG} at runtime;
+for example,
+
address@hidden
+env TEST_SUITE_LOG=partial.log TESTS="..." make -e check
address@hidden example
+
+will write the result of the partial testsuite runs to the
address@hidden, without touching @file{test-suite.log}.
+
@item
You can set the @code{TEST_LOGS} variable. By default, this variable is
computed at @command{make} run time from the value of @code{TESTS} as
@@ -8882,6 +8944,9 @@ set x subset*.log; shift
env TEST_LOGS="foo.log $*" make -e check
@end example
+The comments made above about @code{TEST_SUITE_LOG} overriding applies
+here too.
+
@item
@vindex RECHECK_LOGS
@cindex lazy test execution
@@ -8959,6 +9024,306 @@ semantics of FreeBSD and OpenBSD @command{make}
conflict with this).
In case of doubt you may want to require to use GNU @command{make},
or work around the issue with inference rules to generate the tests.
address@hidden Custom Test Drivers
address@hidden Custom Test Drivers
+
address@hidden
+* Overview of Custom Test Drivers Support::
+* Declaring Custom Test Drivers::
+* API for Custom Test Drivers::
address@hidden menu
+
address@hidden Overview of Custom Test Drivers Support
address@hidden Overview of Custom Test Drivers Support
+
+Starting form Automake version 1.12, the @option{parallel-tests} harness
+allows the package authors to use third-party custom test drivers, in case
+the
address@hidden FIXME: this should become "default ones" once we have TAP and
Subunit
+default one is inadequate for their purposes.
address@hidden FIXME: add this once we have TAP and Subunit
address@hidden "or do not support their testing protocol of choice."
+
+A custom test driver is expected to properly run the test programs
+passed to it, including the command-line arguments passed to it,
+to analyze their execution and outcome, to create the @file{.log}
+files associated to these test runs, and to display the test results
+on the console. It is responsibility of the author of the test driver
+to ensure that it implements all the above steps meaningfully and
+correctly; Automake isn't and can't be of any help here. On the other
+hand, the Automake-provided code for testsuite summary generation offers
+support for test drivers allowing several test results per test script,
+if they take care to register such results properly (@pxref{Log files
+generation and test results recording}).
+
+The exact details of how test scripts' results are to be determined and
+analyzed is left to the individual drivers. Some drivers might only
+consider the test script exit status (this is done for example by the
+default test driver used by the @option{parallel-tests} harness, described
+in the previous section). Other drivers might implement more complex and
+advanced test protocols, which might require them to parse and interpreter
+the output emitted by the test script they're running (examples of such
+protocols are TAP and SubUnit).
+
+It's very important to note that, even when using custom test drivers,
+most of the @option{parallel-tests} infrastructure described in the
+previous section remains in place; this includes:
+
address@hidden
address@hidden
+concurrency through the use of @command{make}'s option @option{-j};
address@hidden
+per-test @file{.log} files, and generation of a summary @file{.log}
+file from them;
address@hidden
address@hidden target and lazy reruns of tests;
address@hidden
+inter-test dependencies;
address@hidden
+support for @code{check_*} variables (@code{check_PROGRAMS},
address@hidden, ...);
address@hidden
+use of @code{VERBOSE} environment variable to get verbose output on
+testsuite failures;
address@hidden
+definition and honoring of @code{TESTS_ENVIRONMENT} and
address@hidden variables;
address@hidden
+definition of generic and extension-specific @code{LOG_COMPILER} and
address@hidden variables.
address@hidden itemize
+
address@hidden
+On the other hand, the exact semantics of how (and if)
address@hidden, @code{XFAIL_TESTS}, and hard errors are supported
+and handled is left to the individual test drivers.
+
address@hidden TODO: We should really add a working example in the doc/
directory,
address@hidden TODO: and reference if from here.
+
address@hidden Declaring Custom Test Drivers
address@hidden Declaring Custom Test Drivers
+
address@hidden _LOG_DRIVER
address@hidden _LOG_DRIVER_FLAGS
address@hidden LOG_DRIVER
address@hidden LOG_DRIVER_FLAGS
address@hidden @var{ext}_LOG_DRIVER
address@hidden @var{ext}_LOG_DRIVER_FLAGS
address@hidden address@hidden
address@hidden AM_LOG_DRIVER_FLAGS
+Custom testsuite drivers are declared by defining the make variables
address@hidden or @address@hidden (where @var{ext} must
+be declared in @code{TEST_EXTENSIONS}). They must be defined to
+programs or scripts that will be used to drive the execution, logging,
+and outcome report of the tests with corresponding extensions (or of
+those with no registered extension in the case of @code{LOG_DRIVER}).
+Clearly, multiple distinct test drivers can be declared in the same
address@hidden Note moreover that the @code{LOG_DRIVER} variables
+are @emph{not} a substitute for the @code{LOG_COMPILER} variables: the
+two sets of variables can, and often do, usefully and legitimately
+coexist.
+
address@hidden TODO: We should really be able to point to a clarifying example
here!
+
+The developer-reserved variable @code{AM_LOG_DRIVER_FLAGS} and the
+user-reserved variable @code{LOG_DRIVER_FLAGS} can be used to define
+flags that will be passed to each invocation of @code{LOG_DRIVER}
+(with the user-defined flags obviously taking precedence over the
+developer-reserved ones). Similarly, for each extension @var{ext}
+declared in @code{TEST_EXTENSIONS}, flags listed in
address@hidden@var{ext}_LOG_DRIVER_FLAGS} and
address@hidden@var{ext}_LOG_DRIVER_FLAGS} will be passed to
+invocations of @address@hidden
+
address@hidden API for Custom Test Drivers
address@hidden API for Custom Test Drivers
+
+Note that @emph{the APIs described here are still somewhat experimental},
+and might undergo changes and tightenings in the future, to accommodate
+for new features or to satisfy additional portability requirements.
+
+The main characteristic of these APIs is that they are designed to share
+as much infrastructure, semantics, and implementation details as possible
+with the @option{parallel-tests} harness and its default driver. So
+everything said in the previous section should apply here too, unless
+explicitly stated otherwise.
+
address@hidden
+* Command-line arguments for test drivers::
+* Log files generation and test results recording::
+* Testsuite progress output::
+* HTML generation from testsuite logs::
address@hidden menu
+
address@hidden Command-line arguments for test drivers
address@hidden Command-line arguments for test drivers
+
+A custom driver can rely on various command-line options and arguments
+being passed to it automatically by the Automake's @option{parallel-tests}
+harness. It is @emph{mandatory} that it understands all of them (even
+if the exact interpretation of the associated semantics can legitimately
+change between a test driver and another, and even be a no-op in some
+drivers).
+
address@hidden
+Here is the list of options:
+
address@hidden @option
address@hidden address@hidden
+The name of the test, with VPATH prefix (if any) removed. This can have a
+suffix and a directory component (as in e.g., @file{sub/foo.test}), and is
+mostly meant to be used in console reports about testsuite advancements and
+results (@pxref{Testsuite progress output}).
address@hidden address@hidden
+The log file the test driver must create. If it has a directory component
+(as in e.g., @file{sub/foo.log}), the Automake harness will ensure that
+such directory exist @emph{before} the test driver is called.
address@hidden address@hidden|address@hidden
+Whether the console output should be colorized or not (@pxref{Simple
+tests and color-tests}, to learn when this option gets activated and
+when it doesn't).
address@hidden address@hidden|address@hidden
+Whether the tested program is expected to fail.
address@hidden address@hidden|address@hidden
+Whether ``hard errors'' in the tested program should be treated differently
+from normal failures or not (the default should be @code{yes}). The exact
+meaning of ``hard error'' is highly dependent from the test protocols or
+conventions in use.
address@hidden --
+Explicitly terminate the list of options.
address@hidden table
+
+The first of the remaining arguments passed to the test driver is the
+program to be run, and the other arguments are command-line options
+and arguments for this program.
+
+Note that the exact semantics attached to the @option{--color-tests},
address@hidden and @option{--enable-hard-errors} options are
+left up to the individual test drivers. Still, having a behaviour
+compatible or at least similar to that provided by the default
address@hidden driver is advised, as that would offer a better
+consistency and a more pleasant user experience.
+
address@hidden Log files generation and test results recording
address@hidden Log files generation and test results recording
+
+The test driver must correctly generate the file specified by the
address@hidden option (of course even when the tested program
+fails or crashes). This file is quite free-form, but still it has
+to conform to the following conventions, in order to work with the
address@hidden harness and take advantage of its features.
+
address@hidden @bullet
address@hidden
address@hidden Global test script result
address@hidden Test result, global
+The @emph{global test result} for the whole test script must be placed
+in the first line of the @file{.log} file.
address@hidden The following semantics is checked by tests
'test-driver-recheck.test'
address@hidden and 'test-driver-global-log.test'. Please keep them in sync
whenever
address@hidden the following is changed.
+If this line does not begin with either @code{PASS:}, @code{XFAIL:} or
address@hidden:}, the test script will be considered failed, and it will be
+re-run by @code{make recheck}. Also, if this line does not begin with
+either @code{PASS:} or @code{XFAIL:}, the content of the @file{.log} file
+will be copied into the global @file{test-suite.log} (in order to help in
+debugging and bug-report analysis).
+
address@hidden
address@hidden Single test case result
address@hidden Test result, single
address@hidden @code{:test-result:} reStructuredText field
address@hidden reStructuredText field, @code{:test-result:}
+One of the main features of the new testsuite harness is the ability to
+support test protocols that allow a single test script to run more
+test cases, @emph{each with its distinct result}. In order for the
+testsuite summary to be correct in this case, the test driver must
+register @emph{each} such result in the generated @file{.log} file, using
+the @code{:test-result:} reStructuredText field. Otherwise, only the
+global test result will be considered (as it is the case of the default
address@hidden driver).
+
address@hidden Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
+The only recognized test results are currently @code{PASS}, @code{XFAIL},
address@hidden, @code{FAIL} and @code{XPASS}. These results, when declared
+with @code{:test-result:}, can be optionally followed by text holding
+the name and/or a brief description of the corresponding test; the
address@hidden harness will ignore such extra text when
+generating @file{test-suite.log} and preparing the testsuite summary.
+
+For example, if a @file{.log} file contains the following lines:
+
address@hidden
+:test-result: PASS server starts
+:test-result: PASS HTTP/1.1 request
+:test-result: FAIL HTTP/1.0 request
+:test-result: SKIP HTTPS request (TLS library wasn't available)
+:test-result: PASS server stops
address@hidden example
+
address@hidden
+it means that the corresponding test script contributes with five test
+results to the testsuite summary (three of these tests being successful,
+one failed, and one skipped).
+
address@hidden FIXME: currently, the @code{:test-result:} field is recognized
anywhere
address@hidden in the @file{.log} file; this is possibly prone to generating
spurious
address@hidden results, in case of verbose tests. We should allow a special
address@hidden @code{:test-result:} that stops any further parsing (maybe a line
address@hidden @code{:test-result:END} will do?).
+
address@hidden itemize
+
+Finally, note that the declaration of the global test result in the first
+line, apart from being needed for backward-compatibility with the default
address@hidden driver, can be useful also for test protocols
+that allow more test results per test script: using it, a custom test
+driver is allowed to decide what the global outcome of the test script
+is in case of conflicting test results within the script. For example,
+if a test script reports 8 successful test cases and 2 skipped test
+cases, some drivers might report that globally as a SKIP, while others
+as a PASS.
+
address@hidden Testsuite progress output
address@hidden Testsuite progress output
+
+A custom test driver also has the task of displaying, on the standard
+output, the test results as soon as they become available. Depending on
+the protocol in use, it can also display the reasons for failures and
+skips, and, more generally, any useful diagnostic output (but remember
+that each line on the screen is precious, so that cluttering the screen
+with overly verbose information is bad idea). The exact format of this
+progress output is left up to the test driver; in fact, a custom test
+driver might @emph{theoretically} even decide not to do any such report,
+leaving it all to the testsuite summary (that would be a very lousy idea,
+of course, and serves only to illustrate the flexibility that is
+granted here).
+
+Remember that consistency is good; so, if possible, try to be consistent
+with the output of the built-in Automake test drivers, providing a similar
+``look & feel''. In particular, the testsuite progress output should be
+colorized when the @option{--color-tests} is passed to the driver. On the
+other end, if you are using a known and widespread test protocol with
+well-established implementations, being consistent with those
+implementations' output might be a good idea too.
+
address@hidden TODO: Give an example, maybe inspired to py.test-style output.
address@hidden TODO: That is a good idea because it shows a test driver that
allows
address@hidden TODO: for different levels of verbosity in the progress output
(could
address@hidden TODO: be implemented either using a driver cmdline flag, or an
address@hidden TODO: environment variable, or both).
+
address@hidden HTML generation from testsuite logs
address@hidden HTML generation from testsuite logs
+
+If HTML testsuite output (with @code{check-html}) is to be supported,
+the generated @file{.log} files must contain syntactically valid
+reStructuredText (among the other things, this means that every
address@hidden:test-result:} line must be followed by a blank line). If this
+is not the case, the HTML generation will not work, although all the
+other functionalities of the Automake testsuite harness should remain
+untouched, and continue to work correctly.
@node DejaGnu Tests
@section DejaGnu Tests
@@ -9196,7 +9561,7 @@ implies options @option{readme-alpha} and
@option{check-news}.
@itemx @address@hidden/ansi2knr}
@cindex Option, @option{ansi2knr}
@opindex ansi2knr
-Turn on the deprecated de-ANSI-fication feature (@xref{ANSI}). Note
+Turn on the deprecated de-ANSI-fication feature (@pxref{ANSI}). Note
that that feature and this option @emph{will be removed} in the next
major Automake release.
@@ -9215,8 +9580,10 @@ in the first few lines of the @file{NEWS} file.
@item @option{color-tests}
@cindex Option, @option{color-tests}
@opindex color-tests
-Cause output of the simple test suite (@pxref{Simple Tests}) to be
-colorized on capable terminals.
+Cause output of the simple and parallel test suites (see @ref{Simple Tests}
+and @ref{Simple Tests using parallel-tests}) and of properly-written custom
+test drivers (@pxref{Custom Test Drivers}) to be colorized on capable
+terminals.
@item @option{dejagnu}
@cindex Option, @option{dejagnu}
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 9c344f5..e1867fb 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -464,7 +464,7 @@ recheck recheck-html:
list=`for f in $$list; do \
test -f $$f || continue; \
if test -r $$f && read line < $$f; then \
- case $$line in FAIL*|XPASS*) echo $$f;; esac; \
+ case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
else echo $$f; fi; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
diff --git a/lib/am/check.am b/lib/am/check.am
index 062c59c..b02d6c5 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -205,16 +205,8 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo; \
echo ".. contents:: :depth: 2"; \
echo; \
-## Here we assume that the test driver writes a proper summary for the
-## test script on the first line. Requiring this is not a limitation,
-## but a feature, since this way a custom test driver is allowed to decide
-## what the outcome is in case of conflicting testcase results in a test
-## script. For example, if a test script reports 8 successful testcases
-## and 2 skipped testcases, some drivers might report that globally as a
-## SKIP, while others as a PASS.
-## FIXME: This should be documented in the automake manual. The above
-## FIXME: explanation is indeed more appropriate for the manual than for
-## FIXME: comments in code.
+## Here we assume that the test driver writes a proper "summarizing
+## result" for the test script on the first line.
for f in $$list; do \
test -r $$f && read line < $$f || line=; \
case $$line in \
@@ -317,7 +309,8 @@ recheck recheck-html:
## test script on the first line. See the comments in the rules of
## $(TEST_SUITE_LOG) above for why we consider this acceptable and even
## advisable.
- case $$line in FAIL*|XPASS*) echo $$f;; esac; \
+ case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
+## A test whose log is unreadable is to be considered failed.
else echo $$f; fi; \
done | tr '\012\015' ' '`; \
## This apparently useless munging helps to avoid a nasty bug (a
diff --git a/tests/Makefile.am b/tests/Makefile.am
index f1ccb7b..0b83b0c 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -176,6 +176,7 @@ autodist-acconfig-no-subdir.test \
autodist-aclocal-m4.test \
autodist-config-headers.test \
autodist-configure-no-subdir.test \
+autodist-no-duplicate.test \
autodist-stamp-vti.test \
autohdr.test \
autohdr2.test \
@@ -723,6 +724,10 @@ parallel-tests-unreadable-log.test \
parallel-tests-subdir.test \
parallel-tests-interrupt.test \
parallel-tests-reset-term.test \
+parallel-tests-harderror.test \
+parallel-tests-log-override-1.test \
+parallel-tests-log-override-2.test \
+parallel-tests-log-override-recheck.test \
parallel-tests-empty-testlogs.test \
parallel-test-driver-install.test \
test-driver-custom-no-extra-driver.test \
@@ -733,6 +738,10 @@ test-driver-custom-multitest-recheck.test \
test-driver-custom-multitest-recheck2.test \
test-driver-custom-html.test \
test-driver-custom-no-html.test \
+test-driver-create-log-dir.test \
+test-driver-global-log.test \
+test-driver-recheck.test \
+test-driver-strip-vpath.test \
test-driver-fail.test \
parse.test \
percent.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index f06562b..193fc95 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -436,6 +436,7 @@ autodist-acconfig-no-subdir.test \
autodist-aclocal-m4.test \
autodist-config-headers.test \
autodist-configure-no-subdir.test \
+autodist-no-duplicate.test \
autodist-stamp-vti.test \
autohdr.test \
autohdr2.test \
@@ -983,6 +984,10 @@ parallel-tests-unreadable-log.test \
parallel-tests-subdir.test \
parallel-tests-interrupt.test \
parallel-tests-reset-term.test \
+parallel-tests-harderror.test \
+parallel-tests-log-override-1.test \
+parallel-tests-log-override-2.test \
+parallel-tests-log-override-recheck.test \
parallel-tests-empty-testlogs.test \
parallel-test-driver-install.test \
test-driver-custom-no-extra-driver.test \
@@ -993,6 +998,10 @@ test-driver-custom-multitest-recheck.test \
test-driver-custom-multitest-recheck2.test \
test-driver-custom-html.test \
test-driver-custom-no-html.test \
+test-driver-create-log-dir.test \
+test-driver-global-log.test \
+test-driver-recheck.test \
+test-driver-strip-vpath.test \
test-driver-fail.test \
parse.test \
percent.test \
@@ -1505,7 +1514,7 @@ recheck recheck-html:
list=`for f in $$list; do \
test -f $$f || continue; \
if test -r $$f && read line < $$f; then \
- case $$line in FAIL*|XPASS*) echo $$f;; esac; \
+ case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
else echo $$f; fi; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
diff --git a/tests/autodist-no-duplicate.test b/tests/autodist-no-duplicate.test
new file mode 100755
index 0000000..81464c0
--- /dev/null
+++ b/tests/autodist-no-duplicate.test
@@ -0,0 +1,52 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check that there are no duplicates in the list of files automatically
+# distributed by automake.
+
+. ./defs || Exit 1
+
+set -e
+
+re='Files .*automatically distributed.*if found'
+
+# The automake manual states that the list of automatically-distributed
+# files should be given by `automake --help'.
+
+list1=`$AUTOMAKE --help \
+ | sed -n "/^$re.*always/,/^ *$/p" \
+ | sed 1d`
+list1=`echo $list1`
+
+list2=`$AUTOMAKE --help \
+ | sed -n "/^$re.*under certain conditions/,/^ *$/p" \
+ | sed 1d`
+list2=`echo $list2`
+
+test -n "$list1"
+test -n "$list2"
+
+st=0
+for i in 1 2; do
+ eval list=\${list$i}
+ for f in $list; do echo $f; done | sort > files$i.lst
+ uniq files$i.lst > files$i.uniq
+ cat files$i.lst
+ cat files$i.uniq
+ diff files$i.lst files$i.uniq || st=1
+done
+
+Exit $st
diff --git a/tests/autodist-subdir.test b/tests/autodist-subdir.test
index 09dcc82..6c0a1d5 100755
--- a/tests/autodist-subdir.test
+++ b/tests/autodist-subdir.test
@@ -37,33 +37,8 @@ $AUTOCONF
# The automake manual states that the list of automatically-distributed
# files should be given by `automake --help'.
list=`$AUTOMAKE --help \
- | sed -n '/^Files .*automatically distributed.*if found/,/^ *$/p' \
+ | sed -n '/^Files.*automatically distributed.*if found.*always/,/^
*$/p' \
| sed 1d`
-list=`for f in $list; do
- case $f in
- configure|configure.in|configure.ac)
- # See test 'autodist-configure-no-subdir.test'.
- ;;
- aclocal.m4)
- # This file should be distributed only when it is a real
- # dependency for configure. Anyway, not a check to be
- # performed in this test.
- ;;
- acconfig.h)
- # Works only when it really exists, not when it is a
- # target in Makefile.am.
- ;;
- stamp-vti)
- # Works only when using info_TEXINFOS and version.texi.
- ;;
- config.h.bot|config.h.top)
- # Works only when the AC_CONFIG_HADERS macro is used.
- ;;
- *)
- echo $f
- ;;
- esac
- done`
# Normalize whitespace, just in case.
list=`echo $list`
diff --git a/tests/autodist.test b/tests/autodist.test
index 681be09..f6fa6fd 100755
--- a/tests/autodist.test
+++ b/tests/autodist.test
@@ -23,15 +23,13 @@
# Ensure we are run from the right directory.
# (The last thing we want is to delete some random user files.)
-test -f ../defs-static
+test -f ../defs-static && test -f ../defs || Exit 99
rm -f *
cat > configure.in <<END
AC_INIT([$me], [1.0])
AC_CONFIG_AUX_DIR([.])
AM_INIT_AUTOMAKE
-## Will avoid useless regeneration of aclocal.m4, configure, etc.
-AM_MAINTAINER_MODE
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
END
@@ -42,33 +40,8 @@ $AUTOCONF
# The automake manual states that the list of automatically-distributed
# files should be given by `automake --help'.
list=`$AUTOMAKE --help \
- | sed -n '/^Files .*automatically distributed.*if found/,/^ *$/p' \
+ | sed -n '/^Files.*automatically distributed.*if found.*always/,/^
*$/p' \
| sed 1d`
-list=`for f in $list; do
- case $f in
- configure|configure.in|configure.ac)
- # See test 'autodist-configure-no-subdir.test'.
- ;;
- aclocal.m4)
- # This file should be distributed only when it is a real
- # dependency for configure. Anyway, not a check to be
- # performed in this test.
- ;;
- acconfig.h)
- # Works only when it really exists, not when it is a
- # target in Makefile.am.
- ;;
- stamp-vti)
- # Works only when using info_TEXINFOS and version.texi.
- ;;
- config.h.bot|config.h.top)
- # Works only when the AC_CONFIG_HADERS macro is used.
- ;;
- *)
- echo $f
- ;;
- esac
- done`
# Normalize whitespace, just in case.
list=`echo $list`
diff --git a/tests/parallel-tests-harderror.test
b/tests/parallel-tests-harderror.test
new file mode 100755
index 0000000..8680e29
--- /dev/null
+++ b/tests/parallel-tests-harderror.test
@@ -0,0 +1,98 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features: DISABLE_HARD_ERRORS
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+set -e
+
+cat >> configure.in << 'END'
+AC_CONFIG_FILES([sub/Makefile])
+AC_OUTPUT
+END
+
+mkdir sub
+
+cat > Makefile.am << 'END'
+SUBDIRS = . sub
+TESTS = foo.test
+XFAIL_TESTS = foo.test
+EXTRA_DIST = $(TESTS)
+END
+
+cat > sub/Makefile.am << 'END'
+TESTS = bar.test
+XFAIL_TESTS = $(TESTS)
+EXTRA_DIST = $(TESTS)
+END
+
+cat > foo.test <<'END'
+#! /bin/sh
+exit 99
+END
+chmod a+x foo.test
+cp foo.test sub/bar.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+# DISABLE_HARD_ERRORS is not defined as a make variable, so that it
+# should be possible to define either from the environment or from
+# command-line, also when recursive make invocations are involved,
+# and also without using the `-e' make flag; this should work also
+# for non-GNU make. Moreover, it shouldn't be necessary to define
+# DISABLE_HARD_ERRORS to "yes" to really disable hard errors: any
+# non-empty value should do.
+$MAKE check DISABLE_HARD_ERRORS=yes
+DISABLE_HARD_ERRORS=x $MAKE check
+
+# But an empty values for DISABLE_HARD_ERRORS means that hard errors
+# are not to be counted like normal failures.
+
+$MAKE check DISABLE_HARD_ERRORS='' && Exit 1
+cat test-suite.log
+grep '^FAIL: foo\.test .*exit.*99' test-suite.log
+
+cd sub
+# The `-e' is wanted here.
+DISABLE_HARD_ERRORS='' $MAKE -e check && Exit 1
+cat test-suite.log
+grep '^FAIL: bar\.test .*exit.*99' test-suite.log
+cd ..
+
+# Check the distributions.
+$MAKE DISABLE_HARD_ERRORS=y distcheck
+
+# Finally, DISABLE_HARD_ERRORS should work also when the developer
+# sets it directly in Makefile.am or Makefile. And its effects
+# should remain local to that specific Makefile, obviously.
+
+echo 'DISABLE_HARD_ERRORS = yes' >> Makefile.am
+$AUTOMAKE Makefile
+./config.status Makefile
+VERBOSE=yes $MAKE check && Exit 1
+grep '^FAIL' test-suite.log && Exit 1
+grep '^FAIL: bar\.test .*exit.*99' sub/test-suite.log
+
+echo 'DISABLE_HARD_ERRORS = zardoz' >> sub/Makefile
+VERBOSE=yes $MAKE check
+
+:
diff --git a/tests/parallel-tests-log-override-1.test
b/tests/parallel-tests-log-override-1.test
new file mode 100755
index 0000000..6e98380
--- /dev/null
+++ b/tests/parallel-tests-log-override-1.test
@@ -0,0 +1,112 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features: runtime redefinition of $(TEST_SUITE_LOG).
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+set -e
+
+cat >> configure.in <<'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am <<'END'
+TESTS = pass.test skip.test xfail.test
+XFAIL_TESTS = xfail.test
+EXTRA_DIST = $(TESTS)
+END
+
+cat > pass.test <<'END'
+#! /bin/sh
+exit 0
+END
+
+cat > skip.test <<'END'
+#! /bin/sh
+echo "% test skipped %"
+exit 77
+END
+
+cat > xfail.test <<'END'
+#! /bin/sh
+echo "# expected failure #"
+exit 1
+END
+
+chmod a+x *.test
+
+test_log_edit ()
+{
+ sed -e "s|^ *$me 1\.0:.*$| $me 1.0: ???|" \
+ -e "s|^=====*|=======================|" $*
+}
+
+test_log_expected ()
+{
+ test_log_edit orig > exp
+ test_log_edit $1 > got
+ diff exp got || Exit 1
+ rm -f exp got
+}
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+$MAKE check
+ls -l
+cat test-suite.log
+cp test-suite.log orig
+
+$MAKE clean
+test -f test-suite.log && Exit 99 # Sanity check.
+
+# Check that we can override the testsuite log file at runtime.
+TEST_SUITE_LOG=zardoz.log $MAKE -e check
+ls -l
+test ! -f test-suite.log
+cat zardoz.log
+test_log_expected zardoz.log
+# Sanity check the distribution too (this also does minimal checks on
+# VPATH support).
+TEST_SUITE_LOG=zardoz.log $MAKE -e distcheck
+
+# Check that cleanup rules remove the correct file even when
+# user overrides are in place.
+cp orig test-suite.log
+TEST_SUITE_LOG=zardoz.log $MAKE -e clean
+ls -l
+test ! -f zardoz.log
+diff orig test-suite.log
+
+# Check that the default testsuite log doesn't get unduly modified.
+# Also check that the testsuite log file doesn't need to be named
+# accordingly to the `*.log' pattern.
+chmod a-w test-suite.log
+TEST_SUITE_LOG=TheLogFile $MAKE -e check
+ls -l
+diff orig test-suite.log
+test_log_expected TheLogFile
+TEST_SUITE_LOG=TheLogFile $MAKE -e clean
+ls -l
+test ! -f TheLogFile
+diff orig test-suite.log
+
+:
diff --git a/tests/parallel-tests-log-override-2.test
b/tests/parallel-tests-log-override-2.test
new file mode 100755
index 0000000..649360c
--- /dev/null
+++ b/tests/parallel-tests-log-override-2.test
@@ -0,0 +1,88 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features: runtime redefinition of:
+# - $(TEST_SUITE_LOG) and $(TESTS)
+# - $(TEST_SUITE_LOG) and $(TEST_LOGS)
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+set -e
+
+cat >> configure.in <<'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am <<'END'
+TESTS = pass.test pass2.test skip.test skip2.test fail.test
+END
+
+cat > pass.test <<'END'
+#! /bin/sh
+exit 0
+END
+
+cp pass.test pass2.test
+
+cat > skip.test <<'END'
+#! /bin/sh
+echo "% skipped test %"
+exit 77
+END
+
+cp skip.test skip2.test
+
+cat > fail.test <<'END'
+#! /bin/sh
+exit 1
+END
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+for test_list_override in \
+ 'TESTS=pass.test skip.test' \
+ 'TEST_LOGS=pass.log skip.log'
+do
+ env TEST_SUITE_LOG=partial.log "$test_list_override" \
+ $MAKE -e check >stdout || { cat stdout; Exit 1; }
+ cat stdout
+ ls -l
+ cat pass.log
+ cat skip.log
+ cat partial.log
+ test ! -f test-suite.log
+ test ! -f pass2.log
+ test ! -f skip2.log
+ test ! -f fail.log
+ grep '^PASS: .*pass\.test' stdout
+ grep '^SKIP: .*skip\.test' stdout
+ $FGREP 'skip.test' partial.log
+ $FGREP '% skipped test %' partial.log
+ for t in pass2 skip2 fail; do
+ $FGREP "$t.test" stdout && Exit 1
+ $FGREP "$t.test" partial.log && Exit 1
+ done
+ rm -f *.log
+done
+
+:
diff --git a/tests/parallel-tests-log-override-recheck.test
b/tests/parallel-tests-log-override-recheck.test
new file mode 100755
index 0000000..8a50d29
--- /dev/null
+++ b/tests/parallel-tests-log-override-recheck.test
@@ -0,0 +1,92 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features: runtime redefinition of $(TEST_SUITE_LOG)
+# for the recheck target.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+set -e
+
+cat >> configure.in <<'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am <<'END'
+TESTS = foo.test bar.test baz.test
+END
+
+cat > foo.test <<'END'
+#! /bin/sh
+echo "this is $0"
+exit 0
+END
+
+cat > bar.test <<'END'
+#! /bin/sh
+echo "this is $0"
+exit 99
+END
+
+cat > baz.test <<'END'
+#! /bin/sh
+echo "this is $0"
+exit ${BAZ_EXIT_STATUS-1}
+END
+
+chmod a+x *.test
+
+unset BAZ_EXIT_STATUS || :
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+chmod a-rw test-suite.log
+TEST_SUITE_LOG=my.log $MAKE -e recheck >stdout \
+ && { cat stdout; Exit 1; }
+cat stdout
+ls -l
+grep '^2 of 2 .*failed' stdout
+for x in stdout my.log; do
+ $FGREP foo.test $x && Exit 1
+ $FGREP bar.test $x
+ $FGREP baz.test $x
+done
+
+chmod a-rw my.log
+BAZ_EXIT_STATUS=0 TEST_SUITE_LOG=my2.log $MAKE -e recheck >stdout \
+ && { cat stdout; Exit 1; }
+cat stdout
+ls -l
+grep '^1 of 2 .*failed' stdout
+$FGREP foo.test stdout && Exit 1
+$FGREP bar.test stdout
+$FGREP baz.test stdout
+$FGREP foo.test my2.log && Exit 1
+$FGREP bar.test my2.log
+$FGREP baz.test my2.log && Exit 1
+
+chmod u+r test-suite.log my.log
+$FGREP baz.test test-suite.log
+$FGREP baz.test my.log
+
+:
diff --git a/tests/parallel-tests.test b/tests/parallel-tests.test
index da961e1..f046c3f 100755
--- a/tests/parallel-tests.test
+++ b/tests/parallel-tests.test
@@ -19,7 +19,6 @@
# - clean
# - TEST_SUITE_LOG
# - dependencies between tests
-# - DISABLE_HARD_ERRORS
# - TESTS
# - TEST_LOGS
# - RECHECK_LOGS
@@ -66,12 +65,13 @@ $AUTOCONF
$AUTOMAKE -a
./configure
-# No hard errors: all tests should be run, there should be one failure.
-env DISABLE_HARD_ERRORS=yes $MAKE -e check >stdout && { cat stdout; Exit 1; }
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-test `grep -c '^FAIL' stdout` -eq 1
+# There should be two errors: bar.test is a hard error.
+test `grep -c '^FAIL' stdout` -eq 2
test -f mylog.log
-test `grep -c '^FAIL' mylog.log` -eq 1
+test `grep -c '^FAIL' mylog.log` -eq 2
test -f baz.log
test -f bar.log
test -f foo.log
@@ -87,19 +87,11 @@ test ! -f foo.log
test ! -f mylog.log
test -f unrelated.log
-$MAKE clean
-$MAKE check >stdout && { cat stdout; Exit 1; }
-cat stdout
-# Now, there should be two errors: bar.test is a hard error.
-test `grep -c '^FAIL' stdout` -eq 2
-test `grep -c '^FAIL' mylog.log` -eq 2
-
# Check dependencies: baz.test needs to run before bar.test,
# but foo.test is not needed.
# Note that this usage has a problem: the summary will only
# take bar.log into account, because the $(TEST_SUITE_LOG) rule
# does not "see" baz.log. Hmm.
-$MAKE clean
env TESTS='bar.test' $MAKE -e check && Exit 1
test -f baz.log
test -f bar.log
diff --git a/tests/test-driver-create-log-dir.test
b/tests/test-driver-create-log-dir.test
new file mode 100755
index 0000000..5713b92
--- /dev/null
+++ b/tests/test-driver-create-log-dir.test
@@ -0,0 +1,74 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers: if a log file has a directory component (as in
+# e.g., `sub/foo.log'), the Automake test harness must ensure that
+# directory exists before calling any custom test driver.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+# The extra `.' components below are meant.
+TESTS = sub/foo sub/subsub/bar.test ././sub1/./baz
+$(TESTS):
+LOG_DRIVER = $(srcdir)/checkdir-driver
+TEST_LOG_DRIVER = $(LOG_DRIVER)
+EXTRA_DIST = checkdir-driver
+
+check-local: $(TEST_SUITE_LOG)
+ test -d sub
+ test -d sub1
+ test -d sub/subsub
+ test -f sub/foo.log
+ test -f sub/subsub/bar.log
+ test -f sub1/baz.log
+END
+
+cat > checkdir-driver <<'END'
+#! /bin/sh
+set -e
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name|--expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+echo "log: $log_file" # For debugging.
+case $log_file in */*);; *) exit 1;; esac
+dir=`expr "$log_file" : '\(.*\)/[^/]*'`
+echo "dir: $dir" # For debugging.
+test -d "$dir" || exit 1
+echo dummy > "$log_file"
+END
+chmod a+x checkdir-driver
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+$MAKE check
+$MAKE distcheck
+
+:
diff --git a/tests/test-driver-global-log.test
b/tests/test-driver-global-log.test
new file mode 100755
index 0000000..cdaaff6
--- /dev/null
+++ b/tests/test-driver-global-log.test
@@ -0,0 +1,110 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check that the output of testcases having a global outcome that is not
+# "PASS" or "XFAIL" is copied in the global 'test-suite.log'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+# The :test-result: fields should be irrelevant for the decision
+# of whether a test output is to be copied in the test-suite.log.
+cat > pass.test <<END
+PASS: pass.test
+:test-result: SKIP
+:test-result: FAIL
+:test-result: XFAIL
+:test-result: XPASS
+:test-result: ERROR
+not seen (pass)
+END
+
+cat > xfail.test <<'END'
+XFAIL: xfail.test
+not seen (xfail)
+END
+
+echo SKIP: skip.test > skip.test
+echo FAIL: fail.test > fail.test
+echo XPASS: xpass.test > xpass.test
+echo :test-result: PASS > fake-pass.test
+echo "$tab $tab$tab" > empty.test
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = ./dummy-driver
+TESTS = pass.test skip.test fail.test xfail.test xpass.test \
+ fake-pass.test empty.test
+END
+
+
+cat > dummy-driver <<'END'
+#!/bin/sh
+set -e
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+cp $1 $log_file
+END
+chmod a+x dummy-driver
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check && { cat test-suite.log; Exit 1; }
+cat test-suite.log
+grep '^SKIP: skip\.test$' test-suite.log
+grep '^FAIL: fail.test$' test-suite.log
+grep '^XPASS: xpass.test$' test-suite.log
+grep '^:test-result: PASS$' test-suite.log
+grep "^$tab $tab$tab$" test-suite.log
+$EGREP 'not seen' test-suite.log && Exit 1
+test `grep -c ':test-result:' test-suite.log` -eq 1
+
+cat > pass.test <<END
+PASS: pass.test
+:test-result: SKIP
+:test-result: XFAIL
+not seen (pass)
+END
+
+cat > xfail.test <<END
+XFAIL: xfail.test
+:test-result: SKIP
+:test-result: PASS
+not seen (xfail)
+END
+
+TESTS='pass.test skip.test xfail.test' $MAKE -e check \
+ || { cat test-suite.log; Exit 1; }
+cat test-suite.log
+grep '^SKIP: skip\.test$' test-suite.log
+$EGREP ':test-result:|not seen' test-suite.log && Exit 1
+
+:
diff --git a/tests/test-driver-recheck.test b/tests/test-driver-recheck.test
new file mode 100755
index 0000000..2ae75b9
--- /dev/null
+++ b/tests/test-driver-recheck.test
@@ -0,0 +1,126 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Test the "make recheck" semantics for custom test driver, as
+# documented in detail in the Automake manual.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+LOG_DRIVER = ./dummy-driver
+TEST_EXTENSIONS =
+TESTS =
+END
+
+rechecked=''
+notrechecked=''
+
+for R in PASS XFAIL SKIP; do
+ echo $R: > $R-1
+ echo " $R:" > $R-2
+ echo $R:foo > $R-3
+ echo "$tab $R: bar baz" > $R-4
+ # The :test-result: fields should be irrelevant for the decision of
+ # whether "make recheck" should or should not re-run a test.
+ unindent > $R-5 <<END
+ $R:
+ :test-result: FAIL
+ :test-result: XPASS
+ :test-result: ERROR
+END
+ notrechecked="$notrechecked $R-1 $R-2 $R-3 $R-4 $R-5"
+ echo $R > BAD-$R-1
+ echo $R. > BAD-$R-2
+ echo :$R: > BAD-$R-3
+ echo $R asd > BAD-$R-4
+ rechecked="$rechecked BAD-$R-1 BAD-$R-2 BAD-$R-3 BAD-$R-4"
+done
+
+for R in FAIL XPASS UNKNOWN; do
+ echo $R: > $R-1
+ echo $R:foo > $R-2
+ echo $R: bar baz > $R-3
+ echo $R > $R-4
+ echo $R asd > $R-5
+ # The :test-result: fields should be irrelevant for the decision of
+ # whether "make recheck" should or should not re-run a test.
+ unindent > $R-6 <<END
+ :test-result: PASS
+ :test-result: SKIP
+ :test-result: XFAIL
+END
+ (echo $R: && cat $R-6) > $R-7
+ rechecked="$rechecked $R-1 $R-2 $R-3 $R-4 $R-5 $R-6 $R-7"
+done
+
+: > EMPTY
+echo " $tab $tab" > WHITE
+rechecked="$rechecked EMPTY WHITE"
+
+tests="$rechecked $notrechecked"
+
+for t in $tests; do echo $t; done | sed 's/.*/TESTS += &/' >> Makefile.am
+
+cat Makefile.am # For debugging.
+
+cat > dummy-driver <<'END'
+#!/bin/sh
+set -e
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+: > $test_name.run
+cp $1 $log_file
+END
+chmod a+x dummy-driver
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# The ':test-result:' fields should be ignored by "make recheck",
+# but should cause the testsuite report to detect errors.
+$MAKE check && Exit 1
+ls -l
+for t in $tests; do test -f $t.run; done
+rm -f *.run
+
+# But now the tests that actually get re-run have only ':test-result:'
+# fields indicating success, so "make recheck" must pass. Still, the
+# next "make recheck" call should still re-run the same set of tests.
+for iteration in 1 2; do
+ $MAKE recheck
+ ls -l
+ for t in $rechecked; do test -f $t.run; done
+ for t in $notrechecked; do test ! -r $t.run; done
+ rm -f *.run
+done
+
+:
diff --git a/tests/test-driver-strip-vpath.test
b/tests/test-driver-strip-vpath.test
new file mode 100755
index 0000000..f1b316c
--- /dev/null
+++ b/tests/test-driver-strip-vpath.test
@@ -0,0 +1,96 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers: check that the test name passed to the test
+# driver has any VPATH prefix stripped.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+ocwd=`pwd` || fatal_ "cannot get current working directory"
+
+mkdir src build
+mv install-sh missing configure.in src
+rm -f depcomp
+
+cd src
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+# The directories of the `bar.test' and `baz.test' tests are deliberately
+# called as the source directory, to verify that the VPATH-stripping code
+# doesn't get too easily confused.
+# The $(empty) are for eliciting VPATH rewrites on make implementations
+# that support it (e.g., Solaris make), to improve coverage.
+empty =
+TESTS = $(empty) foo.test src/bar.test ./src/baz.test $(empty)
+$(TESTS):
+TEST_LOG_DRIVER = $(srcdir)/checkstrip-driver
+EXTRA_DIST = checkstrip-driver
+END
+
+cat > checkstrip-driver <<'END'
+#! /bin/sh
+set -e
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+echo "test name: $test_name" # For debugging.
+case $test_name in
+ foo.test|./foo.test|src/ba[rz].test|./src/ba[rz].test);;
+ *) exit 1;;
+esac
+echo dummy > "$log_file"
+END
+chmod a+x checkstrip-driver
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+cd ..
+
+mkdir build1
+cd build1
+../src/configure
+# "$MAKE -n" is for debugging, should highlight any VPATH rewrite.
+$MAKE -n check
+$MAKE check
+cd ..
+
+mkdir build2
+cd build2
+"$ocwd"/src/configure
+# "$MAKE -n" is for debugging, should highlight any VPATH rewrite.
+$MAKE -n check
+$MAKE check
+cd ..
+
+cd src
+./configure
+$MAKE distcheck
+
+:
hooks/post-receive
--
GNU Automake
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Automake-commit] [SCM] GNU Automake branch, test-protocols, updated. v1.11-896-g1402741,
Stefano Lattarini <=