[FYI] {testsuite-work} testsuite: allow user to ask for stricter skip semantics

[Stefano Lattarini]

Given how the Automake testsuite is currently structured, if a
command in a test script fails with status `77', the global test
outcome is considered a SKIP, because the value of `77' for `$?'
is passed to the exit trap.  Indeed, this happens in practice, as
an autoconf-generated configure script can exit with status `77'
if it fails to find, e.g., a required compiler.

While this behaviour is quite useful for avoiding spurious test
failures in the wild, it can also sometimes prevent the Automake
developers to easily see and declare the requirements of their
tests.

This change introduces a new variable `am_explicit_skips', meant to
be user-overridable, and which, when set to a "true" value (i.e.,
`yes' or `1'), require a direct call to `Exit 77' in order to make
the test outcome be considered a SKIP.

* tests/defs.in ($am__test_skipped): New variable, initialized
to `no'.
(Exit): Set `$am__test_skipped' to `yes' if passed an exit status
of 77.
(trap '...' 0): When `$am_explicit_skips' is set to a "true" value,
reset an exit status of `77' to `78' if $am__test_skipped is not
set to `yes'.
* tests/self-check-exit.test: Adjust: unset `am_explicit_skips'.
* tests/self-check-explicit-skips.test: New test.
* tests/Makefile.am (TESTS): Update.
---
 ChangeLog                            |   28 ++++++++++++++++++
 tests/Makefile.am                    |    1 +
 tests/Makefile.in                    |    1 +
 tests/defs                           |   22 +++++++++++++-
 tests/self-check-exit.test           |    2 +
 tests/self-check-explicit-skips.test |   52 ++++++++++++++++++++++++++++++++++
 6 files changed, 104 insertions(+), 2 deletions(-)
 create mode 100755 tests/self-check-explicit-skips.test

diff --git a/ChangeLog b/ChangeLog
index b61cfdc..cab13f2 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,33 @@
 2011-05-24  Stefano Lattarini  <address@hidden>
 
+       testsuite: allow user to ask for stricter skip semantics
+       Given how the Automake testsuite is currently structured, if a
+       command in a test script fails with status `77', the global test
+       outcome is considered a SKIP, because the value of `77' for `$?'
+       is passed to the exit trap.  Indeed, this happens in practice, as
+       an autoconf-generated configure script can exit with status `77'
+       if it fails to find, e.g., a required compiler.
+       While this behaviour is quite useful for avoiding spurious test
+       failures in the wild, it can also sometimes prevent the Automake
+       developers to easily see and declare the requirements of their
+       tests.
+       This change introduces a new variable `am_explicit_skips', meant to
+       be user-overridable, and which, when set to a "true" value (i.e.,
+       `yes' or `1'), require a direct call to `Exit 77' in order to make
+       the test outcome be considered a SKIP.
+       * tests/defs.in ($am__test_skipped): New variable, initialized
+       to `no'.
+       (Exit): Set `$am__test_skipped' to `yes' if passed an exit status
+       of 77.
+       (trap '...' 0): When `$am_explicit_skips' is set to a "true" value,
+       reset an exit status of `77' to `78' if $am__test_skipped is not
+       set to `yes'.
+       * tests/self-check-exit.test: Adjust: unset `am_explicit_skips'.
+       * tests/self-check-explicit-skips.test: New test.
+       * tests/Makefile.am (TESTS): Update.
+
+2011-05-24  Stefano Lattarini  <address@hidden>
+
        testsuite: user can force skipping of compiler-requiring tests
        * tests/defs (cc, c++, fortran, fortran77): Skip the test if
        the relevant compiler is disabled by having the corresponding
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 954141b..ff20e25 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -850,6 +850,7 @@ self-check-cleanup.test \
 self-check-dir.test \
 self-check-env-sanitize.test \
 self-check-exit.test \
+self-check-explicit-skips.test \
 self-check-is_newest.test \
 self-check-me.test \
 self-check-report.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 711f4ca..55647ee 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -1115,6 +1115,7 @@ self-check-cleanup.test \
 self-check-dir.test \
 self-check-env-sanitize.test \
 self-check-exit.test \
+self-check-explicit-skips.test \
 self-check-is_newest.test \
 self-check-me.test \
 self-check-report.test \
diff --git a/tests/defs b/tests/defs
index 873271f..e3c922c 100644
--- a/tests/defs
+++ b/tests/defs
@@ -103,6 +103,10 @@ unset VERBOSE
 ##  Auxiliary shell functions.  ##
 ## ---------------------------- ##
 
+# This is used in `Exit' and in the exit trap.  See comments in the latter
+# for more information,
+am__test_skipped=no
+
 # We use a trap below for cleanup.  This requires us to go through
 # hoops to get the right exit status transported through the signal.
 # So use `Exit STATUS' instead of `exit STATUS' inside of the tests.
@@ -111,8 +115,9 @@ unset VERBOSE
 Exit ()
 {
   set +e
-  (exit $1)
-  exit $1
+  # See comments in the exit trap for the reason we do this.
+  test 77 = $1 && am__test_skipped=yes
+  (exit $1); exit $1
 }
 
 # Print warnings (e.g., about skipped and failed tests) to this file
@@ -604,6 +609,19 @@ if test "$sh_errexit_works" = yes; then
   trap 'exit_status=$?
     set +e
     cd "$testbuilddir"
+    # This is to ensure that a test script does give a SKIP outcome just
+    # because a command in it happens to exit with status 77.  This
+    # behaviour, while from time to time useful to developers, is not
+    # meant to be enabled by default, as it could cause spurious failures
+    # in the wild.  Thus it will be enabled only when the variable
+    # "am_explicit_skips" is set to a "true" value.
+    case $am_explicit_skips in
+      [yY]|[yY]es|1)
+        if test $exit_status -eq 77 && test $am__test_skipped != yes; then
+          exit_status=78
+        fi
+        ;;
+    esac
     case $exit_status,$keep_testdirs in
     0,)
       find $testSubDir -type d ! -perm -700 -exec chmod u+rwx {} ";"
diff --git a/tests/self-check-exit.test b/tests/self-check-exit.test
index 73f3ea5..f8c5334 100755
--- a/tests/self-check-exit.test
+++ b/tests/self-check-exit.test
@@ -21,6 +21,8 @@
 
 . ./defs-static || exit 99
 
+unset am_explicit_skips
+
 for st in 1 2 3 4 5 77 99 126 127 128 129 130 255; do
 
   echo "* Try: Exit $st"
diff --git a/tests/self-check-explicit-skips.test 
b/tests/self-check-explicit-skips.test
new file mode 100755
index 0000000..15bf2d9
--- /dev/null
+++ b/tests/self-check-explicit-skips.test
@@ -0,0 +1,52 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Sanity check for the automake testsuite.
+# Check creation/removal of temporary test working directory by `./defs'.
+
+. ./defs || Exit 1
+
+test x"$sh_errexit_works" = x"yes" || skip_ "no working shell exit trap"
+
+# We still need a little hack to make ./defs work outside automake's
+# tree `tests' subdirectory.  Not a big deal.
+sed "s|^testbuilddir=.*|testbuilddir='`pwd`'|" ../defs-static >defs-static
+diff ../defs-static defs-static && Exit 99
+cp ../defs .
+
+set +e
+
+unset am_explicit_skips stderr_fileno_
+
+$SHELL -c '. ./defs; (exit 77); exit 77' dummy.test
+test $? -eq 77 || Exit 1
+
+am_explicit_skips=no $SHELL -c '. ./defs; sh -c "exit 77"' dummy.test
+test $? -eq 77 || Exit 1
+
+am_explicit_skips=yes $SHELL -c '. ./defs; (exit 77); exit 77' dummy.test
+test $? -eq 78 || Exit 1
+
+am_explicit_skips=y $SHELL -c '. ./defs; sh -c "exit 77"' dummy.test
+test $? -eq 78 || Exit 1
+
+am_explicit_skips=yes $SHELL -c '. ./defs; Exit 77' dummy.test
+test $? -eq 77 || Exit 1
+
+am_explicit_skips=y $SHELL -c '. ./defs; skip_ "foo"' dummy.test
+test $? -eq 77 || Exit 1
+
+:
-- 
1.7.2.3