DEJATOOL and multiple tests [PATCH]

[Richard Dawe]

Hello.

After reading the DejaGnu manual a little more and playing with it
a bit more, I think my original suggestion of removing 'set tool'
from site.exp is correct.

In the DejaGnu manual is in the section "Config File Values" it says
that --tool is equivalent to setting tool ('set tool ...'). Note that
--tool does not set the tool executable to be tested. You have to execute
the program manually in the test script. runtest supports passing in
variables on the command-line to help, e.g.:

    runtest MYPROGRAM=/path/to/myprogram

--tool is used to select which test suite to run. In the section
"Runtest" it says:

"For example, including "--tool gcc" on the runtest command line runs
tests from all test subdirectories whose names match gcc.*, and uses
one of the initialization modules named config/*-gcc.exp."

I looked closer at the examples from DejaGnu 1.4.3 and they seem
to confirm all this. None of the examples have 'set tool ...'
in any scripts.

The patch below removes the 'set tool ...' line from site.exp. It also
adds a number of new test cases for DejaGnu:

* multiple tools;
* check that the default test name is the same as the package;
* check that a failing test fails "make check";
* check that TCL errors fail "make check", when --status is passed
  to runtest.

One potential problem is that the TCL error test depends on the form
of the missing close-brace error message. I doubt that text will
change much, though.

There is a problem with the test (dejagnu3.test) I committed before:
it doesn't actually test anything. I'm sure I tested that it did fail,
when I changed the expected output, before committing. Sorry about that.
Anyway, the patch below fixes the test, so it actually will fail,
if the expected output does not match the actual output.

The patch is against CVS HEAD.

OK to commit (to HEAD and branch-1.7)?

Thanks, bye, Rich =]

2003-04-17  Richard Dawe  <address@hidden>

        * lib/am/dejagnu.am (site.exp): Don't define tool in site.exp,
        because we set that by invoking runtest with --tool.
        * tests/Makefile.am (TESTS): Add dejagnu4.test, dejagnu5.test,
        dejagnu6.test, dejagnu7.test.
        * tests/dejagnu4.test: New test.
        * tests/dejagnu5.test: New test.
        * tests/dejagnu6.test: New test.
        * tests/dejagnu7.test: New test.

        * tests/dejagnu3.test: Make sure the DejaGnu test is actually run.

Index: NEWS
===================================================================
RCS file: /cvs/automake/automake/NEWS,v
retrieving revision 1.209
diff -p -u -3 -r1.209 NEWS
--- NEWS        14 Apr 2003 19:59:29 -0000      1.209
+++ NEWS        17 Apr 2003 22:12:07 -0000
@@ -83,6 +83,9 @@ New in 1.7a:
   overridden.  Fortunately, it appears that few packages use a
   non-PHONY `html' rule.
 
+Bugs fixed in 1.7a:
+* Multiple tests are now correctly supported in DEJATOOL.
+
 
 New in 1.7:
 * Autoconf 2.54 is required.
Index: lib/am/dejagnu.am
===================================================================
RCS file: /cvs/automake/automake/lib/am/dejagnu.am,v
retrieving revision 1.19
diff -p -u -3 -r1.19 dejagnu.am
--- lib/am/dejagnu.am   11 Oct 2002 19:05:31 -0000      1.19
+++ lib/am/dejagnu.am   17 Apr 2003 22:12:08 -0000
@@ -79,7 +79,6 @@ site.exp: Makefile
        @echo '## these variables are automatically generated by make ##' 
>site.tmp
        @echo '# Do not edit here.  If you wish to override these values' 
>>site.tmp
        @echo '# edit the last section' >>site.tmp
-       @echo 'set tool $(DEJATOOL)' >>site.tmp
        @echo 'set srcdir $(srcdir)' >>site.tmp
        @echo "set objdir `pwd`" >>site.tmp
 ## Quote the *_alias variables because they might be empty.
Index: tests/Makefile.am
===================================================================
RCS file: /cvs/automake/automake/tests/Makefile.am,v
retrieving revision 1.483
diff -p -u -3 -r1.483 Makefile.am
--- tests/Makefile.am   17 Apr 2003 19:06:45 -0000      1.483
+++ tests/Makefile.am   17 Apr 2003 22:12:08 -0000
@@ -138,6 +138,10 @@ defun2.test \
 dejagnu.test \
 dejagnu2.test \
 dejagnu3.test \
+dejagnu4.test \
+dejagnu5.test \
+dejagnu6.test \
+dejagnu7.test \
 depacl2.test \
 depcomp.test \
 depcomp2.test \
Index: tests/dejagnu3.test
===================================================================
RCS file: /cvs/automake/automake/tests/dejagnu3.test,v
retrieving revision 1.1
diff -p -u -3 -r1.1 dejagnu3.test
--- tests/dejagnu3.test 17 Apr 2003 19:06:45 -0000      1.1
+++ tests/dejagnu3.test 17 Apr 2003 22:12:08 -0000
@@ -38,15 +38,18 @@ cat > Makefile.am << 'END'
 AUTOMAKE_OPTIONS = dejagnu
 
 DEJATOOL = hammer
+
+AM_RUNTESTFLAGS = HAMMER=$(srcdir)/hammer
 END
 
-mkdir testsuite
-mkdir testsuite/hammer.test
+mkdir hammer.test
 
-cat > testsuite/hammer.test/hammer.exp << 'END'
+cat > hammer.test/hammer.exp << 'END'
 set test test
+spawn $HAMMER
 expect {
-Everything looks like a nail to me! { pass "$test" }
+    "Everything looks like a nail to me!" { pass "$test" }
+    default { fail "$test" }
 }
 END
 
--- /dev/null   Tue May  5 21:32:27 1998
+++ tests/dejagnu4.test Thu Apr 17 22:56:20 2003
@@ -0,0 +1,85 @@
+#! /bin/sh
+# Copyright (C) 2003  Free Software Foundation, Inc.
+#
+# This file is part of GNU Automake.
+#
+# GNU Automake is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# GNU Automake is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with autoconf; see the file COPYING.  If not, write to
+# the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+# Check that the DejaGnu rules work for a simple program and test case.
+
+required=runtest
+. ./defs || exit 1
+
+set -e
+
+cat > hammer << 'END'
+#! /bin/sh
+echo "Everything looks like a nail to me!"
+END
+
+chmod +x hammer
+
+cat > spanner << 'END'
+#! /bin/sh
+echo "I'm a right spanner!"
+END
+
+chmod +x spanner
+
+echo AC_OUTPUT >> configure.in
+
+cat > Makefile.am << 'END'
+AUTOMAKE_OPTIONS = dejagnu
+
+DEJATOOL = hammer spanner
+
+AM_RUNTESTFLAGS = HAMMER=$(srcdir)/hammer SPANNER=$(srcdir)/spanner
+END
+
+mkdir hammer.test
+mkdir spanner.test
+
+cat > hammer.test/hammer.exp << 'END'
+set test test
+spawn $HAMMER
+expect {
+    "Everything looks like a nail to me!" { pass "$test" }
+    default { fail "$test" }
+}
+END
+
+cat > spanner.test/spanner.exp << 'END'
+set test test
+spawn $SPANNER
+expect {
+    "I'm a right spanner!" { pass "$test" }
+    default { fail "$test" }
+}
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE --add-missing
+
+./configure
+
+$MAKE check
+test -f hammer.log
+test -f hammer.sum
+test -f spanner.log
+test -f spanner.sum
+
+$MAKE distcheck
--- /dev/null   Tue May  5 21:32:27 1998
+++ tests/dejagnu5.test Thu Apr 17 22:59:56 2003
@@ -0,0 +1,69 @@
+#! /bin/sh
+# Copyright (C) 2003  Free Software Foundation, Inc.
+#
+# This file is part of GNU Automake.
+#
+# GNU Automake is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# GNU Automake is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with autoconf; see the file COPYING.  If not, write to
+# the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+# Check that the DejaGnu rules do look for a testsuite named after
+# the package by default.
+
+required=runtest
+. ./defs || exit 1
+
+set -e
+
+# Get the package name from configure.in.
+package=`$FGREP AC_INIT configure.in | sed -e 's:^[^[]*\[\([^]]*\)\].*$:\1:'`
+
+cat > $package << 'END'
+#! /bin/sh
+echo "Ah, we have been expecting you, Mr. Blond."
+END
+
+chmod +x $package
+
+echo AC_OUTPUT >> configure.in
+
+cat > Makefile.am << 'END'
+AUTOMAKE_OPTIONS = dejagnu
+
+END
+
+echo "AM_RUNTESTFLAGS = PACKAGE=\$(srcdir)/$package" >> Makefile.am
+
+mkdir $package.test
+
+cat > $package.test/$package.exp << 'END'
+set test "test"
+spawn $PACKAGE
+expect {
+    "Ah, we have been expecting you, Mr. Blond." { pass "$test" }
+    default { fail "$test" }
+}
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE --add-missing
+
+./configure
+
+$MAKE check
+test -f $package.log
+test -f $package.sum
+
+$MAKE distcheck
--- /dev/null   Tue May  5 21:32:27 1998
+++ tests/dejagnu6.test Thu Apr 17 23:04:33 2003
@@ -0,0 +1,64 @@
+#! /bin/sh
+# Copyright (C) 2003  Free Software Foundation, Inc.
+#
+# This file is part of GNU Automake.
+#
+# GNU Automake is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# GNU Automake is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with autoconf; see the file COPYING.  If not, write to
+# the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+# Check that "make check" fails, when a DejaGnu test fails.
+
+required=runtest
+. ./defs || exit 1
+
+set -e
+
+cat > faildeja << 'END'
+#! /bin/sh
+echo whatever
+END
+
+chmod +x faildeja
+
+echo AC_OUTPUT >> configure.in
+
+cat > Makefile.am << 'END'
+AUTOMAKE_OPTIONS = dejagnu
+
+DEJATOOL = faildeja
+
+AM_RUNTESTFLAGS = FAILDEJA=$(srcdir)/faildeja
+END
+
+mkdir faildeja.test
+
+cat > faildeja.test/faildeja.exp << 'END'
+set test test
+spawn $FAILDEJA
+expect {
+    default { fail "$test" }
+}
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE --add-missing
+
+./configure
+
+$MAKE check || true
+test -f faildeja.log
+test -f faildeja.sum
+$FGREP 'FAIL: test' faildeja.sum
--- /dev/null   Tue May  5 21:32:27 1998
+++ tests/dejagnu7.test Thu Apr 17 23:07:08 2003
@@ -0,0 +1,63 @@
+#! /bin/sh
+# Copyright (C) 2003  Free Software Foundation, Inc.
+#
+# This file is part of GNU Automake.
+#
+# GNU Automake is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# GNU Automake is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with autoconf; see the file COPYING.  If not, write to
+# the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+# Check that "make check" fails, when we invoke DejaGnu tests with --status
+# (to detect TCL errors) on a file with TCL errors.
+
+required=runtest
+. ./defs || exit 1
+
+cat > failtcl << 'END'
+#! /bin/sh
+echo whatever
+END
+
+chmod +x failtcl
+
+echo AC_OUTPUT >> configure.in
+
+cat > Makefile.am << 'END'
+AUTOMAKE_OPTIONS = dejagnu
+
+DEJATOOL = failtcl
+
+AM_RUNTESTFLAGS = --status FAILTCL=$(srcdir)/failtcl
+END
+
+mkdir failtcl.test
+
+cat > failtcl.test/failtcl.exp << 'END'
+set test test
+spawn $FAILTCL
+expect {
+    default { pass "$test" }
+# Oops, no closing brace.
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE --add-missing
+
+./configure
+
+$MAKE check || true
+test -f failtcl.log
+test -f failtcl.sum
+$FGREP 'missing close-brace' failtcl.sum