Re: testsuite -C dir

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Paul Eggert on 1/27/2008 11:51 AM:
| Eric Blake <address@hidden> writes:
|
|> +2008-01-27  Eric Blake  <address@hidden>
|> +
|> + Add 'testsuite -C dir'.
|
| Thanks for contributing that.  It looks good to me.

Applied, after one more round of testing with m4 (there were some
additional impacts, like having to use $at_myself rather than $0 to
reinvoke the --list option of the testsuite to name failed tests in the
log).  Also, in the process, I noticed that a standalone test (one where
atconfig is not present), the platform information was not captured in the
log; also the PATH sanitization was taking a long time on cygwin prior to
running the first test.  Fixed as follows:

2008-01-28  Eric Blake  <address@hidden>

        Minor testsuite improvements.
        * lib/autotest/general.m4 (AT_INIT) <PREPARE_TESTS>: Use fewer
        forks when sanitizing PATH.  Always output machine information,
        not just when atconfig was located.

- --
Don't work too hard, make some time for fun as well!

Eric Blake             address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHneGR84KuGfSFAYARAg/0AJsHZONUB4l2NMDxYgI+hXIznQlorwCeO9P/
HZDs6R1z1NYOpbZZ3//L8yU=
=pFb/
-----END PGP SIGNATURE-----

>From b91a600e7f2d9c72ab9865df88ec8e9a9aa02f1e Mon Sep 17 00:00:00 2001
From: Eric Blake <address@hidden>
Date: Mon, 28 Jan 2008 06:57:46 -0700
Subject: [PATCH] Add 'testsuite -C dir'.

* lib/autotest/general.m4 (_AT_ARG_OPTION): Move missing argument
detection...
(AT_INIT) <PARSE_ARGS_END>: ...here, since -k always takes
argument.
<DEFAULTS>: Delay computation of variables based on $at_dir...
<PREPARE_TESTS>: ...to here, since -C can change $at_dir.
<TESTS>: Re-invoke via absolute name, since -C may be in effect.
<PARSE_ARGS>: Parse new option.
<HELP_TUNING>: Document it.
* tests/autotest.at (Choosing where testsuite is run): New test
for this feature.
(Keywords and ranges): Add test for missing -k argument.
* NEWS: Document this.
* doc/autoconf.texi (testsuite Invocation): Likewise.

Signed-off-by: Eric Blake <address@hidden>
---
 ChangeLog               |   18 ++++++
 NEWS                    |    3 +
 doc/autoconf.texi       |    6 ++
 lib/autotest/general.m4 |  136 +++++++++++++++++++++++++++--------------------
 tests/autotest.at       |   28 +++++++++-
 5 files changed, 133 insertions(+), 58 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 27fb894..49cfa45 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,21 @@
+2008-01-28  Eric Blake  <address@hidden>
+
+       Add 'testsuite -C dir'.
+       * lib/autotest/general.m4 (_AT_ARG_OPTION): Move missing argument
+       detection...
+       (AT_INIT) <PARSE_ARGS_END>: ...here, since -k always takes
+       argument.
+       <DEFAULTS>: Delay computation of variables based on $at_dir...
+       <PREPARE_TESTS>: ...to here, since -C can change $at_dir.
+       <TESTS>: Re-invoke via absolute name, since -C may be in effect.
+       <PARSE_ARGS>: Parse new option.
+       <HELP_TUNING>: Document it.
+       * tests/autotest.at (Choosing where testsuite is run): New test
+       for this feature.
+       (Keywords and ranges): Add test for missing -k argument.
+       * NEWS: Document this.
+       * doc/autoconf.texi (testsuite Invocation): Likewise.
+
 2008-01-24  Ralf Wildenhues  <address@hidden>
 
        * build-aux/config.guess, build-aux/config.sub,
diff --git a/NEWS b/NEWS
index ba278ad..855ed52 100644
--- a/NEWS
+++ b/NEWS
@@ -81,6 +81,9 @@ GNU Autoconf NEWS - User visible changes.
 
 ** Autotest now determines $srcdir correctly.
 
+** Testsuites built by autotest now accept a -C/--directory=DIR option
+   to adjust the working directory prior to creating files.
+
 ** Autoconf now requires GNU M4 1.4.5 or later.  Earlier versions of M4 have
    a bug in macro tracing that interferes with the interaction between
    Autoconf and Automake.  GNU M4 1.4.8 or later is recommended.  The
diff --git a/doc/autoconf.texi b/doc/autoconf.texi
index 78628e5..59e6f97 100644
--- a/doc/autoconf.texi
+++ b/doc/autoconf.texi
@@ -20191,6 +20191,12 @@ Display the list of options and exit successfully.
 @itemx -V
 Display the version of the test suite and exit successfully.
 
address@hidden address@hidden
address@hidden -C @var{dir}
+Change the current directory to @var{dir} before creating any files.
+Useful for running the testsuite in a subdirectory from a top-level
+Makefile.
+
 @item --clean
 @itemx -c
 Remove all the files the test suite might have created and exit.  Meant
diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4
index 11ff925..c1101ad 100644
--- a/lib/autotest/general.m4
+++ b/lib/autotest/general.m4
@@ -1,8 +1,8 @@
 # This file is part of Autoconf.                          -*- Autoconf -*-
 # M4 macros used in building test suites.
 
-# Copyright (C) 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007 Free
-# Software Foundation, Inc.
+# Copyright (C) 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008
+# Free Software Foundation, Inc.
 
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -195,8 +195,8 @@ m4_define([AT_help_all], [])
 AS_INIT[]dnl
 m4_divert_push([DEFAULTS])dnl
 AT_COPYRIGHT(
-[Copyright (C) 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007 Free Software
-Foundation, Inc.
+[Copyright (C) 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008
+Free Software Foundation, Inc.
 This test suite is free software; the Free Software Foundation gives
 unlimited permission to copy, distribute and modify it.])
 AS_PREPARE
@@ -361,31 +361,6 @@ fi
 ## ---------------------- ##
 m4_divert_pop([PREPARE_TESTS])dnl back to DEFAULTS
 
-# Load the config file.
-for at_file in atconfig atlocal
-do
-  test -r $at_file || continue
-  . ./$at_file || AS_ERROR([invalid content: $at_file])
-done
-
-# Autoconf <=2.59b set at_top_builddir instead of at_top_build_prefix:
-: ${at_top_build_prefix=$at_top_builddir}
-
-# atconfig delivers names relative to the directory the test suite is
-# in, but the groups themselves are run in testsuite-dir/group-dir.
-if test -n "$at_top_srcdir"; then
-  builddir=../..
-  for at_dir in srcdir top_srcdir top_build_prefix
-  do
-    at_val=AS_VAR_GET([at_$at_dir])
-    case $at_val in
-      [[\\/$]]* | ?:[[\\/]]* ) at_prefix= ;;
-      *) at_prefix=../../ ;;
-    esac
-    AS_VAR_SET([$at_dir], [$at_prefix$at_val])
-  done
-fi
-
 # Not all shells have the 'times' builtin; the subshell is needed to make
 # sure we discard the 'times: not found' message from the shell.
 at_times_p=false
@@ -413,7 +388,7 @@ at_groups=
 # Whether a write failure occurred
 at_write_fail=0
 
-# The directory we are in.
+# The directory we run the suite in.  Default to . if no -C option.
 at_dir=`pwd`
 # An absolute reference to this testsuite script.
 dnl m4-double quote, to preserve []
@@ -421,23 +396,8 @@ dnl m4-double quote, to preserve []
   [\\/]* | ?:[\\/]* ) at_myself=$as_myself ;;
   * ) at_myself=$at_dir/$as_myself ;;
 esac]
-# The directory the whole suite works in.
-# Should be absolute to let the user `cd' at will.
-at_suite_dir=$at_dir/$as_me.dir
-# The file containing the suite.
-at_suite_log=$at_dir/$as_me.log
-# The file containing the location of the last AT_CHECK.
-at_check_line_file=$at_suite_dir/at-check-line
-# The file containing the exit status of the last command.
-at_status_file=$at_suite_dir/at-status
-# The files containing the output of the tested commands.
-at_stdout=$at_suite_dir/at-stdout
-at_stder1=$at_suite_dir/at-stder1
-at_stderr=$at_suite_dir/at-stderr
-# The file containing the function to run a test group.
-at_test_source=$at_suite_dir/at-test-source
-# The file containing dates.
-at_times_file=$at_suite_dir/at-times
+# Whether -C is in effect.
+at_change_dir=false
 m4_divert_pop([DEFAULTS])dnl
 m4_wrap([m4_divert_text([DEFAULTS],
 [
@@ -563,6 +523,15 @@ do
        at_groups="$at_groups$at_range "
        ;;
 
+    # Directory selection.
+    --directory | -C )
+        at_prev=--directory
+       ;;
+    --directory=* )
+        at_change_dir=:
+        at_dir=$at_optarg
+       ;;
+
     # Keywords.
     --keywords | -k )
        at_prev=--keywords
@@ -616,6 +585,9 @@ m4_divert_push([PARSE_ARGS_END])dnl
   esac
 done
 
+# Verify our last option didn't require an argument
+AS_IF([test -n "$at_prev"], [AS_ERROR([`$at_prev' requires an argument.])])
+
 # Selected test groups.
 if test -z "$at_groups"; then
   at_groups=$at_groups_all
@@ -661,6 +633,8 @@ cat <<_ATEOF || at_write_fail=1
 
 dnl extra quoting prevents emacs whitespace mode from putting tabs in output
 Execution tuning:
+  -C, --directory=DIR
+[                 change to directory DIR before starting]
   -k, --keywords=KEYWORDS
 [                 select the tests matching all the comma-separated KEYWORDS]
 [                 multiple \`-k' accumulate; prefixed \`!' negates a KEYWORD]
@@ -724,9 +698,63 @@ fi
 m4_divert_pop([VERSION_END])dnl
 m4_divert_push([PREPARE_TESTS])dnl
 
+# Take any -C into account.
+if $at_change_dir ; then
+  if test x- = x$"at_dir" ; then
+    at_dir=./-
+  fi
+  test x != x"$at_dir" && cd "$at_dir" \
+    || AS_ERROR([unable to change directory])
+  at_dir=`pwd`
+fi
+
+# Load the config file.
+for at_file in atconfig atlocal
+do
+  test -r $at_file || continue
+  . $at_file || AS_ERROR([invalid content: $at_file])
+done
+
+# Autoconf <=2.59b set at_top_builddir instead of at_top_build_prefix:
+: ${at_top_build_prefix=$at_top_builddir}
+
+# atconfig delivers names relative to the directory the test suite is
+# in, but the groups themselves are run in testsuite-dir/group-dir.
+if test -n "$at_top_srcdir"; then
+  builddir=../..
+  for at_dir_var in srcdir top_srcdir top_build_prefix
+  do
+    at_val=AS_VAR_GET([at_$at_dir_var])
+    case $at_val in
+      [[\\/$]]* | ?:[[\\/]]* ) at_prefix= ;;
+      *) at_prefix=../../ ;;
+    esac
+    AS_VAR_SET([$at_dir_var], [$at_prefix$at_val])
+  done
+fi
+
+# The directory the whole suite works in.
+# Should be absolute to let the user `cd' at will.
+at_suite_dir=$at_dir/$as_me.dir
+# The file containing the suite.
+at_suite_log=$at_dir/$as_me.log
+# The file containing the location of the last AT_CHECK.
+at_check_line_file=$at_suite_dir/at-check-line
+# The file containing the exit status of the last command.
+at_status_file=$at_suite_dir/at-status
+# The files containing the output of the tested commands.
+at_stdout=$at_suite_dir/at-stdout
+at_stder1=$at_suite_dir/at-stder1
+at_stderr=$at_suite_dir/at-stderr
+# The file containing the function to run a test group.
+at_test_source=$at_suite_dir/at-test-source
+# The file containing dates.
+at_times_file=$at_suite_dir/at-times
+
 # Don't take risks: use only absolute directories in PATH.
 #
-# For stand-alone test suites, AUTOTEST_PATH is relative to `.'.
+# For stand-alone test suites (ie. atconfig was not found),
+# AUTOTEST_PATH is relative to `.'.
 #
 # For embedded test suites, AUTOTEST_PATH is relative to the top level
 # of the package.  Then expand it into build/src parts, since users
@@ -1145,17 +1173,17 @@ else
     # Summary of failed and skipped tests.
     if test $at_fail_count != 0; then
       echo "Failed tests:"
-      $SHELL "$[0]" $at_fail_list --list
+      $SHELL "$at_myself" $at_fail_list --list
       echo
     fi
     if test $at_skip_count != 0; then
       echo "Skipped tests:"
-      $SHELL "$[0]" $at_skip_list --list
+      $SHELL "$at_myself" $at_skip_list --list
       echo
     fi
     if test $at_xpass_count != 0; then
       echo "Unexpected passes:"
-      $SHELL "$[0]" $at_xpass_list --list
+      $SHELL "$at_myself" $at_xpass_list --list
       echo
     fi
     if test $at_fail_count != 0; then
@@ -1249,12 +1277,6 @@ m4_divert_text([PARSE_ARGS_BEGIN],
 m4_ifvaln([$3],,[at_arg_[]m4_bpatsubst([AT_first_option], -, _)=false])dnl
 at_arg_given_[]m4_bpatsubst([AT_first_option], -, _)=false
 ])dnl m4_divert_text DEFAULTS
-m4_ifval([$3],[m4_divert_once([PARSE_ARGS_END],
-[
-##
-## Verify our last option didn't require an argument
-##
-AS_IF([test -n "$at_prev"],[AS_ERROR([`$at_prev' requires an argument.])])])])
 m4_divert_text([PARSE_ARGS],
 [dnl Parse the options and args when necessary.
 m4_ifvaln([$3],
diff --git a/tests/autotest.at b/tests/autotest.at
index e043d7a..4fbdba2 100644
--- a/tests/autotest.at
+++ b/tests/autotest.at
@@ -2,7 +2,8 @@
 
 AT_BANNER([Autotest.])
 
-# Copyright (C) 2004, 2005, 2006, 2007 Free Software Foundation, Inc.
+# Copyright (C) 2004, 2005, 2006, 2007, 2008 Free Software Foundation,
+# Inc.
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -395,6 +396,29 @@ AT_CHECK([(cd micro-suite.dir/1 && ./run MY_VAR='one 
space')],
 ])
 
 
+# Controlling where the testsuite is run.
+AT_CHECK_AT_TEST([Choosing where testsuite is run],
+  [AT_CHECK([:])], [], [], [], [], [
+dnl AT_CHECK_AT_TEST tests the default of running in `.'.
+AT_CHECK([rm micro-suite.log && mkdir sub1 sub2])
+dnl check specifying a different relative path to run in.
+AT_CHECK([./micro-suite -C sub1], [0], [ignore], [])
+AT_CHECK([test -f micro-suite.log], [1])
+AT_CHECK([test -f sub1/micro-suite.log], [0])
+dnl check specifying an absolute path to run in.
+AT_CHECK([./micro-suite --directory="`pwd`/sub2"], [0], [ignore], [])
+AT_CHECK([test -f micro-suite.log], [1])
+AT_CHECK([test -f sub2/micro-suite.log], [0])
+dnl check for failure detection with bad, missing, or empty directory.
+AT_CHECK([./micro-suite -C nonesuch], [1], [ignore], [ignore])
+AT_CHECK([./micro-suite -C ''], [1], [ignore], [ignore])
+AT_CHECK([./micro-suite -C -], [1], [ignore], [ignore])
+AT_CHECK([./micro-suite -C], [1], [ignore], [ignore])
+dnl check that --help overrides bad directory selection.
+AT_CHECK([./micro-suite -C nonesuch --help], [0], [ignore], [])
+])
+
+
 ## -------- ##
 ## Banners. ##
 ## -------- ##
@@ -491,6 +515,8 @@ AT_CLEANUP
 AT_CHECK_AUTOM4TE([--language=autotest -o k k.at])
 dnl check that AT_KEYWORDS does not duplicate words
 AT_CHECK([grep 'key1.*key1' k], [1])
+dnl check that -k requires an argument
+AT_CHECK([./k -k], [1], [], [ignore])
 
 # AT_CHECK_KEYS(TESTSUITE-OPTIONS, PATTERN1, COUNT1, PATTERN2, COUNT2)
 m4_define([AT_CHECK_KEYS],
-- 
1.5.3.8


>From c6c22af0e79a7f08c150f6dc59841be914dfa66d Mon Sep 17 00:00:00 2001
From: Eric Blake <address@hidden>
Date: Mon, 28 Jan 2008 07:02:43 -0700
Subject: [PATCH] Minor testsuite improvements.

* lib/autotest/general.m4 (AT_INIT) <PREPARE_TESTS>: Use fewer
forks when sanitizing PATH.  Always output machine information,
not just when atconfig was located.

Signed-off-by: Eric Blake <address@hidden>
---
 ChangeLog               |    5 +++++
 lib/autotest/general.m4 |   12 ++++++++----
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 49cfa45..b8e7fb0 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,10 @@
 2008-01-28  Eric Blake  <address@hidden>
 
+       Minor testsuite improvements.
+       * lib/autotest/general.m4 (AT_INIT) <PREPARE_TESTS>: Use fewer
+       forks when sanitizing PATH.  Always output machine information,
+       not just when atconfig was located.
+
        Add 'testsuite -C dir'.
        * lib/autotest/general.m4 (_AT_ARG_OPTION): Move missing argument
        detection...
diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4
index c1101ad..662f2cb 100644
--- a/lib/autotest/general.m4
+++ b/lib/autotest/general.m4
@@ -785,8 +785,11 @@ esac])
 # builtins' (eg., cd) stderr directly: Ultrix's sh hates that.
 at_new_path=
 _AS_PATH_WALK([$at_path],
-[as_dir=`(cd "$as_dir" && pwd) 2>/dev/null`
-test -d "$as_dir" || continue
+[test -d "$as_dir" || continue
+case $as_dir in
+  [[\\/]]* | ?:[[\\/]]* ) ;;
+  * ) as_dir=`(cd "$as_dir" && pwd) 2>/dev/null` ;;
+esac
 case $PATH_SEPARATOR$at_new_path$PATH_SEPARATOR in
   *$PATH_SEPARATOR$as_dir$PATH_SEPARATOR*) ;;
   $PATH_SEPARATOR$PATH_SEPARATOR) at_new_path=$as_dir ;;
@@ -832,10 +835,11 @@ AS_BOX(m4_defn([AT_TESTSUITE_NAME])[.])
       echo
     done
 
-    AS_UNAME
-    echo
   fi
 
+  AS_UNAME
+  echo
+
   # Contents of the config files.
   for at_file in atconfig atlocal
   do
-- 
1.5.3.8