Re: automake/488: "make check" may seem to succeed when there are failed tests in multiple tools

[Ralf Wildenhues]

Hi Paul,

* Paul Lunau wrote on Wed, Apr 12, 2006 at 09:54:16PM CEST:
> 
> If you use dejagnu for testing and have multiple tools (i.e. multiple
> entries in the DEJATOOL variable) then the success/failure of "make
> check" is only dependent on the success/failure of the tests on the
> final tool. Thus there may be failures in the tests on previous tools,
> but at first glance "make check" has passed.
> 
> I believe the correct behaviour should be for make check to fail if
> any tests on any of the tools fail.

I agree.  Thanks for the bug report!

OK to apply this patch to fix this in the spirit of Paul's suggestion,
and amend the dejagnu4.test to expose the failure?

(One could surely write a new test, in fact that's what I did at first;
it would be very similar to dejagnu4.test, however, and this change
costs little additional testing time :)

Cheers,
Ralf

2006-04-19  Paul Lunau  <address@hidden>  (tiny change)
            Ralf Wildenhues  <address@hidden>

        * lib/am/dejagnu.am (check-DEJAGNU): Fail when at least one test
        failed, instead of when the last one failed (PR automake/488).
        Report from Paul Lunau.
        * tests/dejagnu4.test: Update to expose this.
        * THANKS: Update.

Index: THANKS
===================================================================
RCS file: /cvs/automake/automake/THANKS,v
retrieving revision 1.279
diff -u -r1.279 THANKS
--- THANKS      9 Apr 2006 07:18:33 -0000       1.279
+++ THANKS      19 Apr 2006 16:01:08 -0000
@@ -205,6 +206,7 @@
 Paul D. Smith          address@hidden
 Paul Eggert            address@hidden
 Paul Jarc              address@hidden
+Paul Lunau             address@hidden
 Paul Martinolich       address@hidden
 Paul Thomas            address@hidden
 Pavel Roskin           address@hidden
Index: lib/am/dejagnu.am
===================================================================
RCS file: /cvs/automake/automake/lib/am/dejagnu.am,v
retrieving revision 1.24
diff -u -r1.24 dejagnu.am
--- lib/am/dejagnu.am   14 May 2005 20:28:52 -0000      1.24
+++ lib/am/dejagnu.am   19 Apr 2006 16:01:14 -0000
@@ -1,5 +1,5 @@
 ## automake - create Makefile.in from Makefile.am
-## Copyright (C) 1994, 1995, 1996, 1997, 2000, 2001, 2003
+## Copyright (C) 1994, 1995, 1996, 1997, 2000, 2001, 2003, 2006
 ## Free Software Foundation, Inc.
 
 ## This program is free software; you can redistribute it and/or modify
@@ -58,11 +58,13 @@
 ## If runtest can't be found, print a warning but don't die.  It is
 ## pointless to cause a failure if the tests cannot be run at all.
        if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
-         l='$(DEJATOOL)'; for tool in $$l; do \
-           $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) 
$(RUNTESTFLAGS); \
+         exit_status=0; l='$(DEJATOOL)'; for tool in $$l; do \
+           if $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS) 
$(RUNTESTFLAGS); \
+           then :; else exit_status=1; fi; \
          done; \
        else echo "WARNING: could not find \`runtest'" 1>&2; :;\
-       fi
+       fi; \
+       exit $$exit_status
 
 
 ## ------------------- ##
Index: tests/dejagnu4.test
===================================================================
RCS file: /cvs/automake/automake/tests/dejagnu4.test,v
retrieving revision 1.4
diff -u -r1.4 dejagnu4.test
--- tests/dejagnu4.test 14 May 2005 20:28:54 -0000      1.4
+++ tests/dejagnu4.test 19 Apr 2006 16:01:00 -0000
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2003  Free Software Foundation, Inc.
+# Copyright (C) 2003, 2006  Free Software Foundation, Inc.
 #
 # This file is part of GNU Automake.
 #
@@ -19,6 +19,7 @@
 # Boston, MA 02110-1301, USA.
 
 # Check that the DejaGnu rules work for a simple program and test case.
+# Also check PR 488: Failure of the first of several tools tested.
 
 required=runtest
 . ./defs || exit 1
@@ -86,3 +87,11 @@
 test -f spanner.sum
 
 $MAKE distcheck
+
+# Test for PR 488.
+sed 's/E\(verything\)/Not e\1/' hammer > thammer
+mv thammer hammer
+chmod +x hammer
+
+$MAKE check && exit 1
+: