[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH] {maint} tests init: don't automatically re-execute tests with a
|
From: |
Stefano Lattarini |
|
Subject: |
[PATCH] {maint} tests init: don't automatically re-execute tests with a POSIX shell |
|
Date: |
Fri, 6 Jul 2012 13:21:54 +0200 |
I've unwittingly broken support for that feature *again* in some of
my recent testsuite tweaking. In this case, the re-execution code
works correctly when the tests are executed with a POSIX shells, but
breaks when they are invoked by an old-style Bourne shells (e.g.,
/bin/sh on Solaris).
It's time to face it: that feature is too much brittle, and too seldom
used (because the Makefile takes care of running the tests with the
correct shell anyway, so that a breakage is only experienced when
running the tests by hand). It just don't remain working for long, not
when we often touch the testsuite setup (which we are going to do again
when we'll try to move part of our testsuite framework to Gnulib, or a
similar project).
So, instead of trying to be extra-smart and automatically re-execute the
tests with the correct shell, we now offer a simple wrapper script that
the user can employ to run the test scripts with the proper shell. And
while we are at it, we write this wrapper to also deal with TAP tests in
a better way, running them through the prove(1) utility, so that their
results are correctly recognized and reported.
* t/ax/test-runner.in: New file; the wrapper script we were talking about.
* configure.ac (AC_CONFIG_FILES): Process it into 't/ax/test-runner'.
* defs: Remove code for automatic re-execution of the scripts with the
correct shell. This file now just a very thin layer around 'defs-static'
and 't/ax/test-init.sh'.
* t/README: Adjust, and remove or fix some imprecise or outdated text in
the process (like "... test scripts are written with portability in mind,
so that they should run with any decent Bourne-compatible shell ..." ).
* Makefile.am (AM_TESTS_ENVIRONMENT): No need to export 'AM_TESTS_REEXEC'
to "no" anymore.
* t/self-check-explicit-skips.sh: Likewise.
* t/self-check-exit.tap: Likewise.
* t/self-check-me.tap: Likewise.
* t/self-check-dir.tap: Likewise.
* t/self-check-cleanup.tap: Likewise.
* t/self-check-reexec.tap: Remove as obsolete.
* t/list-of-tests.mk: Adjust.
Signed-off-by: Stefano Lattarini <address@hidden>
---
I will push this by this evening if there are no objections.
Regards,
Stefano
Makefile.am | 5 -
configure.ac | 1 +
defs | 38 +------
t/README | 50 ++++------
t/ax/test-runner.in | 62 ++++++++++++
t/list-of-tests.mk | 1 -
t/self-check-cleanup.tap | 1 -
t/self-check-dir.tap | 1 -
t/self-check-exit.tap | 2 -
t/self-check-explicit-skips.sh | 1 -
t/self-check-me.tap | 2 -
t/self-check-reexec.tap | 216 ----------------------------------------
12 files changed, 86 insertions(+), 294 deletions(-)
create mode 100644 t/ax/test-runner.in
delete mode 100755 t/self-check-reexec.tap
diff --git a/Makefile.am b/Makefile.am
index b745721..f55f6fb 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -345,11 +345,6 @@ AM_TESTS_ENVIRONMENT = \
; do \
eval test x"\$${$$v}" = x || unset $$v; \
done;
-# The 'AM_TESTS_REEXEC=no' setting tells the tests not to needlessly
-# re-execute themselves with the shell detected at configure time, since
-# we are already running them under it explicitly in our setup (see e.g.
-# the definition of TEST_LOG_COMPILER above).
-AM_TESTS_ENVIRONMENT += AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC;
# We want warning messages and explanations for skipped tests to go to
# the console if possible, so set up 'stderr_fileno_' properly.
AM_TESTS_FD_REDIRECT = 9>&2
diff --git a/configure.ac b/configure.ac
index 7d30762..8ec92ba 100644
--- a/configure.ac
+++ b/configure.ac
@@ -559,6 +559,7 @@ AC_SUBST([EXEEXT])
## ---------------------- ##
AC_CONFIG_FILES([Makefile])
+AC_CONFIG_FILES([t/ax/test-runner])
AC_CONFIG_LINKS([GNUmakefile:GNUmakefile])
AC_CONFIG_LINKS([defs:defs])
diff --git a/defs b/defs
index 143be66..df85746 100644
--- a/defs
+++ b/defs
@@ -15,44 +15,8 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# *** IMPORTANT NOTE ***
-# This file should execute correctly with any system's /bin/sh (which
-# might be just a Bourne shell, non POSIX-conforming, like on Solaris
-# up to version 11 at least).
-
# Source the shell static setup and variable definitions.
-. ./defs-static
-test $? -eq 0 || exit 99
-
-# Make sure we run with the shell detected at configure time or forced
-# by the user (unless the user forbids it). That is assumed to be a
-# proper POSIX shell.
-case ${AM_TESTS_REEXEC-yes} in
- n|no|false|0)
- ;;
- *)
- # Ensure we can find ourselves.
- if test ! -f "$argv0"; then
- echo "$me: unable to find myself: '$argv0'" >&2
- exit 99
- fi
- AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC
- # Cannot simply do "opts=$-", since the content of $- is not
- # portable among different shells. So try to propagate only
- # the portable and interesting options.
- case $- in
- *x*v*|*v*x) opts=-vx;;
- *v*) opts=-v;;
- *x*) opts=-x;;
- *) opts=;;
- esac
- echo exec $AM_TEST_RUNNER_SHELL $opts "$argv0" "$*"
- exec $AM_TEST_RUNNER_SHELL $opts "$argv0" ${1+"$@"}
- # This should be dead code, unless some strange error happened.
- echo "$me: failed to re-execute with $AM_TEST_RUNNER_SHELL" >&2
- exit 99
- ;;
-esac
+. ./defs-static; test $? -eq 0 || exit 99
# Source the actual test initialization and setup code, and return
# control to the test script that is sourcing us.
diff --git a/t/README b/t/README
index fe86229..4fe7b6c 100644
--- a/t/README
+++ b/t/README
@@ -70,45 +70,39 @@ About the tests
There are two kinds of tests in the Automake testsuite (both implemented
as shell scripts). The scripts with the '.sh' suffix are "simple"
tests, their outcome completely determined by their exit status. Those
- with the '.tap' suffix use the TAP protocol. If you want to run a test
- by hand, you can do so directly if it is a simple test:
+ with the '.tap' suffix use the TAP protocol.
+
+ If you want to run a test by hand, you should be able to do so using the
+ 'test-runner' script:
- ./t/nogzip.sh
+ ./t/ax/test-runner t/nogzip.sh
+ ./t/ax/test-runner t/add-missing.tap
- (it will be verbose by default), while if it is a TAP test you can pass
- it to your preferred TAP runner, as in e.g.:
-
- prove --verbose --merge ./t/add-missing.tap
-
- The tests can also be run directly in a VPATH build, as with:
-
- /path/to/srcdir/t/nogzip.sh
- prove --verbose --merge /path/to/srcdir/t/add-missing.tap
+ This will run the test using the correct shell, and should also work in
+ VPATH builds. Note that, to run the TAP tests this way, you'll need to
+ have the prove(1) utility available in $PATH.
Supported shells
----------------
- By default, the tests are run by the $SHELL detected at configure
- time. They also take care to re-execute themselves with that shell,
- unless told not to. So, to run the tests with a different shell, say
- '/path/to/another/sh', the user must use:
-
- AM_TESTS_REEXEC=no /path/to/another/sh ./t/foo.sh
- AM_TESTS_REEXEC=no prove -v -e /path/to/another/sh ./t/bar.tap
+ By default, the tests are run by a proper shell detected at configure
+ time. Here is how you can run the tests with a different shell, say
+ '/bin/my-sh':
- to run a test directly, and:
+ # Running through the makefile test driver.
+ make check AM_TEST_RUNNER_SHELL=/bin/my-sh (GNU make)
+ AM_TEST_RUNNER_SHELL=/bin/my-sh make -e check (non-GNU make)
- make check LOG_COMPILER=/path/to/sh (GNU make)
- LOG_COMPILER=/path/to/sh make -e check (non-GNU make)
+ # Run a test directly from the command line.
+ AM_TEST_RUNNER_SHELL=/bin/my-sh ./t/ax/test-runner t/foo.sh
- to run the test(s) through the makefile test driver.
- The test scripts are written with portability in mind, so that they
- should run with any decent Bourne-compatible shell. However, it is
- worth nothing that older versions of Zsh (pre-4.3) exhibited several
- bugs and incompatibilities with our uses, and are thus not supported
- for running Automake's test scripts.
+ The test scripts are written with portability in mind, and should run
+ with any decent POSIX shell. However, it is worth nothing that older
+ versions of Zsh (pre-4.3) exhibited several bugs and incompatibilities
+ with our uses, and are thus not supported for running Automake's test
+ scripts.
Reporting failures
diff --git a/t/ax/test-runner.in b/t/ax/test-runner.in
new file mode 100644
index 0000000..c632ec3
--- /dev/null
+++ b/t/ax/test-runner.in
@@ -0,0 +1,62 @@
address@hidden@
+#
+# Copyright (C) 2012 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Run an Automake test from the command line.
+# Usage:
+
+set -e; set -u
+
+: ${AM_TEST_RUNNER_SHELL='@AM_TEST_RUNNER_SHELL@'}
+: ${AM_PROVE_CMD='prove'}
+: ${AM_PROVEFLAGS='--merge --verbose'}
+: ${srcdir='@srcdir@'}
+
+error () { echo "$0: $*" >&2; exit 255; }
+
+# Some shell flags should be passed over to the test scripts.
+#while test $# -gt 0; do
+# case $1 in
+# --help) echo "Usage: $0 [SHELL-OPTIONS] TEST [TEST-OPTIONS]"; exit $?;;
+# -*)
+# esac
+#done
+
+test $# -gt 0 || error "missing argument"
+
+tst=$1; shift
+
+case $tst in
+ *.sh) wrapper () { exec "$@"; };;
+ *.tap) wrapper () { exec "$AM_PROVE_CMD" $AM_PROVEFLAGS -e "$@"; };;
+ *) error "test '$tst' has an unrecognized extension";;
+esac
+
+case $tst in
+ /*) ;;
+ *) if test -f ./$tst; then
+ tst=./$tst
+ # Support for VPATH build.
+ elif test -f $srcdir/$tst; then
+ tst=$srcdir/$tst
+ else
+ error "could not find test '$tst'"
+ fi
+ ;;
+esac
+
+wrapper "$tst" "$@"
+error "dead code reached"
diff --git a/t/list-of-tests.mk b/t/list-of-tests.mk
index 73c4127..0fda449 100644
--- a/t/list-of-tests.mk
+++ b/t/list-of-tests.mk
@@ -964,7 +964,6 @@ t/self-check-exit.tap \
t/self-check-explicit-skips.sh \
t/self-check-is_newest.tap \
t/self-check-me.tap \
-t/self-check-reexec.tap \
t/self-check-report.sh \
t/self-check-seq.tap \
t/self-check-is-blocked-signal.tap \
diff --git a/t/self-check-cleanup.tap b/t/self-check-cleanup.tap
index 2c5a9a9..faffd71 100755
--- a/t/self-check-cleanup.tap
+++ b/t/self-check-cleanup.tap
@@ -29,7 +29,6 @@ diff "$am_top_builddir"/defs-static defs-static \
&& fatal_ "failed to edit defs-static"
cp "$am_top_builddir"/defs .
-AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC
keep_testdirs=; unset keep_testdirs
if ln -s defs foo && test -h foo; then
diff --git a/t/self-check-dir.tap b/t/self-check-dir.tap
index 94555a6..2bc2ce0 100755
--- a/t/self-check-dir.tap
+++ b/t/self-check-dir.tap
@@ -24,7 +24,6 @@ am_create_testdir=no
plan_ 5
-AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC
keep_testdirs=; unset keep_testdirs
# This needs to be consistent with what $AM_TEST_RUNNER_SHELL
diff --git a/t/self-check-exit.tap b/t/self-check-exit.tap
index 4bc6710..0fbc9d8 100755
--- a/t/self-check-exit.tap
+++ b/t/self-check-exit.tap
@@ -29,8 +29,6 @@ plan_ 34
# so that false positives remain very unlikely.
set +e
-AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC
-
# It's especially important that the 'unset' is done before sourcing
# ./defs, i.e., when 'set -e' is not active yet, for the benefit of shells
# (like Bash 2.05 and Solaris 10 /bin/ksh) that returns a non-zero status
diff --git a/t/self-check-explicit-skips.sh b/t/self-check-explicit-skips.sh
index 95ed313..eb60dc2 100755
--- a/t/self-check-explicit-skips.sh
+++ b/t/self-check-explicit-skips.sh
@@ -24,7 +24,6 @@ am_create_testdir=no
set +e
unset am_explicit_skips stderr_fileno_
-AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC
# FIXME: this can be simplified when we improve our runtime checks on the
# testsuite shell to ensure it supports "VAR=val shell_func" correctly.
diff --git a/t/self-check-me.tap b/t/self-check-me.tap
index 717fe2e..9b286b5 100755
--- a/t/self-check-me.tap
+++ b/t/self-check-me.tap
@@ -22,8 +22,6 @@ am_create_testdir=no
plan_ 12
-AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC
-
# This test becomes cumbersome if we keep the 'errexit' shell flag set.
# And removing it is no big deal, as this test is a TAP-based one, so
# that false positives remain very unlikely.
diff --git a/t/self-check-reexec.tap b/t/self-check-reexec.tap
deleted file mode 100755
index f46797f..0000000
--- a/t/self-check-reexec.tap
+++ /dev/null
@@ -1,216 +0,0 @@
-#! /bin/sh
-# Copyright (C) 2011-2012 Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# Sanity check for the automake testsuite.
-# Check that automatic re-execution of test script with the
-# configure-time $AM_TEST_RUNNER_SHELL.
-
-am_create_testdir=empty
-. ./defs || exit 1
-
-plan_ 32
-
-unset AM_TESTS_REEXEC BASH_VERSION || :
-
-cwd=$(pwd) || fatal_ "getting current working directory"
-cp "$am_top_builddir"/defs . || fatal_ "fetching 'defs' from top_builddir"
-
-#
-# Search for required bash and non-bash shells.
-#
-
-for bash_shell in \
- "$SHELL" "$AM_TEST_RUNNER_SHELL" bash bash3 bash4 :
-do
- test "$bash_shell" = : && break
- $bash_shell --version || continue
- $bash_shell -c 'test -n "$BASH_VERSION"' || continue
- break
-done
-
-# This might not be optimal, but it's much better than writing wrapper
-# scripts acting as "fake" shells.
-for non_bash_shell in \
- /bin/sh /bin/ksh sh ksh ash dash pdksh "$SHELL" "$AM_TEST_RUNNER_SHELL" :
-do
- test "$non_bash_shell" = : && break
- $non_bash_shell -c 'exit 0' || continue
- # Be sure to reject also any shell that is Zsh "in disguise" (as can
- # be found on some Debian systems, where /bin/ksh can be symlinked to
- # /bin/zsh4). This is required because our testsuite does not support
- # older versions of Zsh, and that has caused spurious failures in the
- # past.
- $non_bash_shell -c 'test -n "$ZSH_VERSION$BASH_VERSION"' && continue
- break
-done
-
-echo "bash_shell='$bash_shell'"
-echo "non_bash_shell='$non_bash_shell'"
-
-# This would denote an internal error.
-if test "$bash_shell" = : && test "$non_bash_shell" = :; then
- fatal_ "we couldn't find a bash shell nor a non-bash one"
-fi
-
-#
-# Functions used throughout the test.
-#
-
-get_ddata ()
-{
- case $1 in
- ""|*/) dsep=;;
- *) dsep=/;;
- esac
- case $1 in
- "") dname="no dir";;
- /*) dname="absolute dir";;
- *) dname="dir '$1'";;
- esac
-}
-
-get_sh ()
-{
- case $1 in
- bash) sh=$bash_shell;;
- non-bash) sh=$non_bash_shell;;
- *) fatal_ "get_sh: invalid shell type '$1'";;
- esac
-}
-
-#
-# Check how to default, force or prevent a re-execution.
-#
-
-cat > need-bash.sh <<'END'
-#!/bin/false
-. ./defs
-# Ensure that the script gets re-executed with bash. Also ensure that
-# non-standard syntax used after the inclusion of './defs' doesn't cause
-# non-bash shells to fail.
-# Subshell required to prevent some shells (e.g., Solaris 10 /bin/sh)
-# from only complaining on stderr but then exiting with exit status 0.
-(foo=abac && test xbxc = ${foo//a/x} && test -n "$BASH_VERSION")
-END
-
-sh_var=AM_TEST_RUNNER_SHELL
-sed -e "s|^am_top_builddir=.*|am_top_builddir='$cwd'|" \
- -e "s|^$sh_var=.*$|$sh_var=bash; export $sh_var|" \
- < "$am_top_builddir"/defs-static >defs-static
-
-do_reexec ()
-{
- command_ok_ "re-exec if AM_TESTS_REEXEC=$1" \
- env AM_TESTS_REEXEC="$1" $non_bash_shell need-bash.sh
-}
-
-dont_reexec ()
-{
- command_ok_ "don't re-exec if AM_TESTS_REEXEC=$1" \
- not env AM_TESTS_REEXEC="$1" $non_bash_shell need-bash.sh
-}
-
-if test "$bash_shell" = :; then
- skip_row_ 10 -r "no bash shell found" AM_TESTS_REEXEC
-elif test "$non_bash_shell" = :; then
- skip_row_ 10 -r "no non-bash shell found" AM_TESTS_REEXEC
-else
- command_ok_ "re-exec if AM_TESTS_REEXEC unset" \
- $non_bash_shell need-bash.sh
- do_reexec ''
- do_reexec yes
- do_reexec y
- do_reexec true
- do_reexec 1
- dont_reexec no
- dont_reexec n
- dont_reexec false
- dont_reexec 0
-fi
-
-#
-# Check message about the re-execution. Also check that arguments passed
-# to a test script are preserved by a re-exec, even in "corner" cases.
-#
-
-cat > dummy.sh <<'END'
-#!/bin/sh
-. ./defs
-:
-END
-
-cat > checkargs.sh <<'END'
-. ./defs
-test $# -eq 3 && test x"$1" = x'a' && test x"$2" = x && test x"$3" = x"-e"
-END
-
-chmod a+x dummy.sh checkargs.sh
-
-mkdir sub
-cp dummy.sh checkargs.sh defs sub
-sed -e "s|^am_top_builddir=.*|am_top_builddir='$cwd'|" \
- < "$am_top_builddir"/defs-static > defs-static
-sed -e "s|^am_top_builddir=.*|am_top_builddir='$cwd/sub'|" \
- < "$am_top_builddir"/defs-static > sub/defs-static
-
-check_preserve_args ()
-{
- dir=$1; shift
- get_ddata "$dir"
- $sh "${dir}${dsep}checkargs.sh" a '' -e && r='ok' || r='not ok'
- result_ "$r" "$sh re-exec preserving args [$dname]"
-}
-
-check_reexec_message ()
-{
- dir=$1; shift
- get_ddata "$dir"
- $sh "${dir}${dsep}dummy.sh" "$@" \
- | grep "^exec $AM_TEST_RUNNER_SHELL ${dir}${dsep}dummy\\.sh $*\$" \
- && r='ok' || r='not ok'
- result_ "$r" "$sh display re-exec message [$dname] [args: $*]"
-}
-
-./dummy.sh a b \
- | grep "^exec $AM_TEST_RUNNER_SHELL \\./dummy\\.sh a b$" \
- && r='ok' || r='not ok'
-result_ "$r" "direct run display re-exec message [args: a b]"
-
-./checkargs.sh a '' -e && r='ok' || r='not ok'
-result_ "$r" "direct re-exec preserving args"
-
-for sh_type in non-bash bash; do
- get_sh $sh_type
- if test "$sh" = :; then
- skip_row_ 5 -r "no $sh_type shell available" "re-exec message"
- skip_row_ 5 -r "no $sh_type shell available" "re-exec preserving args"
- continue
- fi
- check_preserve_args ''
- check_reexec_message '' a b c
- check_preserve_args .
- check_reexec_message . a b c
- cd sub
- check_preserve_args ..
- check_reexec_message .. a b c
- cd ..
- check_preserve_args "$cwd"
- check_reexec_message "$cwd" a -b c-
- check_preserve_args sub
- check_reexec_message sub 1 2 3 4
-done
-
-:
--
1.7.9.5
- [PATCH] {maint} tests init: don't automatically re-execute tests with a POSIX shell,
Stefano Lattarini <=