Re: [PATCH 3/3] {testsuite-work} tests: `lib/' shell scripts transparently tested also with $SHELL

[Peter Rosin]

Den 2011-06-06 21:49 skrev Stefano Lattarini:
> With the previous commit, the user could prefer the use of $SHELL
> over /bin/sh in some tests checking the Automake-provided shell
> scripts by manually exporting `test_prefer_config_shell' to "yes"
> in the environment.  With this commit, we ensure that such tests
> *always* and *transparently* run using both $SHELL and /bin/sh
> to execute the checked scripts.  The `test_prefer_config_shell'
> variable becomes an internal detail, and is no more meant to be
> manually defined or overridden.
> 
> * tests/defs-static.in: Check that `test_prefer_config_shell' is
> not exported in the environment.  Error out if this is the case.
> * tests/config-shell-tests.sh: New file, driver script to run
> checks on the shell scripts in `lib/' using the $SHELL determined
> at configure time instead of the default system shell /bin/sh.
> * Makefile.am (TESTS_EXTENSIONS): Add `.shtst'.
> (SHTST_LOG_COMPILER): Define, it calls `config-shell-tests.sh'.
> (config_shell_tests): Define to a list of tests that wraps other
> `*.test' tests using `config-shell-tests.sh'.
> ($(config_shell_tests)): Dummy dependency declaration required
> in order to have make actually produce expected log files from
> the `.shtst.log' suffix rule.
> (EXTRA_DIST): Distribute `config-shell-tests.sh'.
> (TESTS): Add `$(config_shell_tests)'.
> ---
>  ChangeLog                   |   26 +++++++++++++++++++++
>  tests/Makefile.am           |   29 +++++++++++++++++++++++
>  tests/Makefile.in           |   40 +++++++++++++++++++++++++++++---
>  tests/config-shell-tests.sh |   53 
> +++++++++++++++++++++++++++++++++++++++++++
>  tests/defs-static.in        |    4 ++-
>  5 files changed, 147 insertions(+), 5 deletions(-)
>  create mode 100755 tests/config-shell-tests.sh
> 
> diff --git a/ChangeLog b/ChangeLog
> index 2254d3b..8fa1cb7 100644
> --- a/ChangeLog
> +++ b/ChangeLog
> @@ -1,5 +1,31 @@
>  2011-06-06  Stefano Lattarini  <address@hidden>
>  
> +     tests: `lib/' shell scripts transparently tested also with $SHELL
> +     With the previous commit, the user could prefer the use of $SHELL
> +     over /bin/sh in some tests checking the Automake-provided shell
> +     scripts by manually exporting `test_prefer_config_shell' to "yes"
> +     in the environment.  With this commit, we ensure that such tests
> +     *always* and *transparently* run using both $SHELL and /bin/sh
> +     to execute the checked scripts.  The `test_prefer_config_shell'
> +     variable becomes an internal detail, and is no more meant to be
> +     manually defined or overridden.
> +     * tests/defs-static.in: Check that `test_prefer_config_shell' is
> +     not exported in the environment.  Error out if this is the case.
> +     * tests/config-shell-tests.sh: New file, driver script to run
> +     checks on the shell scripts in `lib/' using the $SHELL determined
> +     at configure time instead of the default system shell /bin/sh.
> +     * Makefile.am (TESTS_EXTENSIONS): Add `.shtst'.
> +     (SHTST_LOG_COMPILER): Define, it calls `config-shell-tests.sh'.
> +     (config_shell_tests): Define to a list of tests that wraps other
> +     `*.test' tests using `config-shell-tests.sh'.
> +     ($(config_shell_tests)): Dummy dependency declaration required
> +     in order to have make actually produce expected log files from
> +     the `.shtst.log' suffix rule.
> +     (EXTRA_DIST): Distribute `config-shell-tests.sh'.
> +     (TESTS): Add `$(config_shell_tests)'.
> +
> +2011-06-06  Stefano Lattarini  <address@hidden>
> +
>       tests: can use also $SHELL to check shell scripts from `lib/'
>       * tests/ar-lib.test: If the variable `$test_prefer_config_shell'
>       is set to "yes", run the script under test with configure-time
> diff --git a/tests/Makefile.am b/tests/Makefile.am
> index d863b28..339f801 100644
> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> @@ -93,6 +93,34 @@ $(depmod_tests:.depmod=.log): depmod-tests.sh 
> depmod-data.log
>  $(depmod_tests):
>  
>  
> +TEST_EXTENSIONS += .shtst
> +SHTST_LOG_COMPILER = $(SHELL) $(srcdir)/config-shell-tests.sh
> +EXTRA_DIST += config-shell-tests.sh
> +
> +# FIXME: this list might probably be made auto-generated, like
> +# $(parallel_tests) is...  But would that worth the hassle?
> +config_shell_tests = \
> +  compile-w.shtst \
> +  compile2-w.shtst \
> +  compile3-w.shtst \
> +  compile4-w.shtst \
> +  compile5-w.shtst \
> +  compile6-w.shtst \
> +  instsh2-w.shtst \
> +  instsh3-w.shtst \
> +  mkinst3-w.shtst \
> +  missing-w.shtst \
> +  missing2-w.shtst \
> +  missing3-w.shtst \
> +  missing5-w.shtst

I expected to see the ar-lib tests here? Or did you not add those
since the AM_PROG_AR macro hasn't been merged? But would it hurt
to test ar-lib with $SHELL prior to that merge?

> +
> +# All `*-w.shtst' tests work by sourcing the script `config-shell-tests.sh'.
> +$(config_shell_tests:.shtst=.log): config-shell-tests.sh
> +# This dummy declaration is required to have make actually produce
> +# expected log files from the `.shtst.log' suffix rule.
> +$(config_shell_tests):
> +
> +
>  # Some testsuite-influential variables should be overridable from the
>  # test scripts, but not from the environment.
>  # We want warning messages and explanations for skipped tests to go to
> @@ -112,6 +140,7 @@ AM_TESTS_ENVIRONMENT = \
>  
>  TESTS = \
>    $(handwritten_tests) \
> +  $(config_shell_tests) \
>    $(parallel_tests) \
>    $(depmod_tests) \
>    $(instspc_tests)
> diff --git a/tests/Makefile.in b/tests/Makefile.in
> index d2975d7..f59a2f2 100644
> --- a/tests/Makefile.in
> +++ b/tests/Makefile.in
> @@ -198,9 +198,12 @@ PTEST_LOG_COMPILE = $(PTEST_LOG_COMPILER) 
> $(AM_PTEST_LOG_FLAGS) \
>  am__test_logs4 = $(am__test_logs3:.instspc.log=.log)
>  INSTSPC_LOG_COMPILE = $(INSTSPC_LOG_COMPILER) $(AM_INSTSPC_LOG_FLAGS) \
>       $(INSTSPC_LOG_FLAGS)
> -TEST_LOGS = $(am__test_logs4:.depmod.log=.log)
> +am__test_logs5 = $(am__test_logs4:.depmod.log=.log)
>  DEPMOD_LOG_COMPILE = $(DEPMOD_LOG_COMPILER) $(AM_DEPMOD_LOG_FLAGS) \
>       $(DEPMOD_LOG_FLAGS)
> +TEST_LOGS = $(am__test_logs5:.shtst.log=.log)
> +SHTST_LOG_COMPILE = $(SHTST_LOG_COMPILER) $(AM_SHTST_LOG_FLAGS) \
> +     $(SHTST_LOG_FLAGS)
>  TEST_LOGS_TMP = $(TEST_LOGS:.log=.log-t)
>  DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)
>  ACLOCAL = @ACLOCAL@
> @@ -303,8 +306,9 @@ top_builddir = @top_builddir@
>  top_srcdir = @top_srcdir@
>  MAINTAINERCLEANFILES = 
>  EXTRA_DIST = ChangeLog-old gen-parallel-tests parallel-tests.sh \
> -     instspc-tests.sh depmod-tests.sh $(handwritten_tests)
> -TEST_EXTENSIONS = .test .ptest .instspc .depmod
> +     instspc-tests.sh depmod-tests.sh config-shell-tests.sh \
> +     $(handwritten_tests)
> +TEST_EXTENSIONS = .test .ptest .instspc .depmod .shtst
>  # Run the tests with the shell detected at configure time.
>  TEST_LOG_COMPILER = $(SHELL)
>  XFAIL_TESTS = all.test auxdir2.test cond17.test gcj6.test \
> @@ -382,6 +386,25 @@ depmod_tests = depcomp-gcc3.depmod depcomp-gcc.depmod \
>       depcomp-cpp.depmod depcomp-msvisualcpp.depmod \
>       depcomp-msvcmsys.depmod
>  DEPMOD_LOG_COMPILER = $(SHELL) $(srcdir)/depmod-tests.sh
> +SHTST_LOG_COMPILER = $(SHELL) $(srcdir)/config-shell-tests.sh
> +
> +# FIXME: this list might probably be made auto-generated, like
> +# $(parallel_tests) is...  But would that worth the hassle?
> +config_shell_tests = \
> +  compile-w.shtst \
> +  compile2-w.shtst \
> +  compile3-w.shtst \
> +  compile4-w.shtst \
> +  compile5-w.shtst \
> +  compile6-w.shtst \
> +  instsh2-w.shtst \
> +  instsh3-w.shtst \
> +  mkinst3-w.shtst \
> +  missing-w.shtst \
> +  missing2-w.shtst \
> +  missing3-w.shtst \
> +  missing5-w.shtst
> +
>  
>  # Some testsuite-influential variables should be overridable from the
>  # test scripts, but not from the environment.
> @@ -402,6 +425,7 @@ AM_TESTS_ENVIRONMENT = \
>  
>  TESTS = \
>    $(handwritten_tests) \
> +  $(config_shell_tests) \
>    $(parallel_tests) \
>    $(depmod_tests) \
>    $(instspc_tests)
> @@ -1367,7 +1391,7 @@ am__tdf = diff-in-tests-lists.tmp
>  all: all-am
>  
>  .SUFFIXES:
> -.SUFFIXES: .depmod .html .instspc .log .ptest .test
> +.SUFFIXES: .depmod .html .instspc .log .ptest .shtst .test
>  $(srcdir)/Makefile.in:  $(srcdir)/Makefile.am $(srcdir)/parallel-tests.am 
> $(srcdir)/instspc-tests.am $(srcdir)/depmod-tests.am 
> $(top_srcdir)/CheckListOfTests.am $(am__configure_deps)
>       @for dep in $?; do \
>         case '$(am__configure_deps)' in \
> @@ -1545,6 +1569,8 @@ recheck recheck-html:
>       @p='$<'; $(am__check_pre) $(INSTSPC_LOG_COMPILE) "$$tst" 
> $(am__check_post)
>  .depmod.log:
>       @p='$<'; $(am__check_pre) $(DEPMOD_LOG_COMPILE) "$$tst" 
> $(am__check_post)
> +.shtst.log:
> +     @p='$<'; $(am__check_pre) $(SHTST_LOG_COMPILE) "$$tst" $(am__check_post)
>  
>  distdir: $(DISTFILES)
>       @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
> @@ -1757,6 +1783,12 @@ $(depmod_tests:.depmod=.log): depmod-tests.sh 
> depmod-data.log
>  # expected log files from the `.depmod.log' suffix rule.
>  $(depmod_tests):
>  
> +# All `*-w.shtst' tests work by sourcing the script `config-shell-tests.sh'.
> +$(config_shell_tests:.shtst=.log): config-shell-tests.sh
> +# This dummy declaration is required to have make actually produce
> +# expected log files from the `.shtst.log' suffix rule.
> +$(config_shell_tests):
> +
>  .PHONY: maintainer-check-list-of-tests
>  maintainer-check-list-of-tests:
>        @if diff -u /dev/null /dev/null >/dev/null 2>&1; then \
> diff --git a/tests/config-shell-tests.sh b/tests/config-shell-tests.sh
> new file mode 100755
> index 0000000..150287e
> --- /dev/null
> +++ b/tests/config-shell-tests.sh
> @@ -0,0 +1,53 @@
> +#! /bin/sh
> +# Copyright (C) 2011 Free Software Foundation, Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2, or (at your option)
> +# any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +
> +# Driver script to run checks on Automake-provided shell scripts using
> +# configure-time $SHELL instead of the system shell /bin/sh.
> +
> +set -e
> +
> +fatal_ () { echo "$0: $*" >&2; exit 99; }
> +
> +# Ensure proper definition of $testsrcdir.
> +. ./defs-static || exit 99
> +test -n "$testsrcdir" || fatal_ "\$testsrcdir is empty or undefined"
> +
> +case $#,$1 in
> +  0,) fatal_ "missing argument";;
> +  1,*-w.shtst) test_name=`expr /"$1" : '.*/\(.*\)-w\.shtst'`;;
> +  1,*) fatal_ "invalid argument \`$1'";;
> +  *) fatal_ "too many arguments";;
> +esac
> +
> +# Run the test using configure-time $SHELL, not /bin/sh, to run the
> +# checked shell scripts.
> +test_prefer_config_shell=yes
> +# This is required to have the wrapped test use a proper temporary
> +# directory to run into.
> +me=$test_name-w
> +# In the spirit of VPATH, we prefer a test in the build tree
> +# over one in the source tree.
> +if test -f "./$test_name.test"; then
> +  . "./$test_name.test"
> +  exit $?
> +elif test -f "$testsrcdir/$test_name.test"; then
> +  . "$testsrcdir/$test_name.test"
> +  exit $?
> +else
> +  fatal_ "cannot find wrapped test \`$test_name.test'"
> +fi
> +
> +exit 255 # Not reached.
> diff --git a/tests/defs-static.in b/tests/defs-static.in
> index 53d9f93..17ed284 100644
> --- a/tests/defs-static.in
> +++ b/tests/defs-static.in
> @@ -70,7 +70,9 @@ fi
>  # Check that the environment is properly sanitized.
>  # Having variables exported to the empty string is OK, since our code
>  # treats such variables as if they were unset.
> -for var in me required parallel_tests original_AUTOMAKE original_ACLOCAL; do
> +for var in me required parallel_tests test_prefer_config_shell \
> +           original_AUTOMAKE original_ACLOCAL
> +do
>    if eval "test x\"\$$var\" != x" && env | grep "^$var=" >/dev/null; then
>      echo "$argv0: variable \`$var' is set in the environment:" \
>           "this is unsafe" >&2


-- 
They are in the crowd with the answer before the question.
> Why do you dislike Jeopardy?