[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: automake/488: "make check" may seem to succeed when there are failed
From: |
Ralf Wildenhues |
Subject: |
Re: automake/488: "make check" may seem to succeed when there are failed tests in multiple tools |
Date: |
Wed, 19 Apr 2006 18:04:36 +0200 |
User-agent: |
Mutt/1.5.11 |
Hi Paul,
* Paul Lunau wrote on Wed, Apr 12, 2006 at 09:54:16PM CEST:
>
> If you use dejagnu for testing and have multiple tools (i.e. multiple
> entries in the DEJATOOL variable) then the success/failure of "make
> check" is only dependent on the success/failure of the tests on the
> final tool. Thus there may be failures in the tests on previous tools,
> but at first glance "make check" has passed.
>
> I believe the correct behaviour should be for make check to fail if
> any tests on any of the tools fail.
I agree. Thanks for the bug report!
OK to apply this patch to fix this in the spirit of Paul's suggestion,
and amend the dejagnu4.test to expose the failure?
(One could surely write a new test, in fact that's what I did at first;
it would be very similar to dejagnu4.test, however, and this change
costs little additional testing time :)
Cheers,
Ralf
2006-04-19 Paul Lunau <address@hidden> (tiny change)
Ralf Wildenhues <address@hidden>
* lib/am/dejagnu.am (check-DEJAGNU): Fail when at least one test
failed, instead of when the last one failed (PR automake/488).
Report from Paul Lunau.
* tests/dejagnu4.test: Update to expose this.
* THANKS: Update.
Index: THANKS
===================================================================
RCS file: /cvs/automake/automake/THANKS,v
retrieving revision 1.279
diff -u -r1.279 THANKS
--- THANKS 9 Apr 2006 07:18:33 -0000 1.279
+++ THANKS 19 Apr 2006 16:01:08 -0000
@@ -205,6 +206,7 @@
Paul D. Smith address@hidden
Paul Eggert address@hidden
Paul Jarc address@hidden
+Paul Lunau address@hidden
Paul Martinolich address@hidden
Paul Thomas address@hidden
Pavel Roskin address@hidden
Index: lib/am/dejagnu.am
===================================================================
RCS file: /cvs/automake/automake/lib/am/dejagnu.am,v
retrieving revision 1.24
diff -u -r1.24 dejagnu.am
--- lib/am/dejagnu.am 14 May 2005 20:28:52 -0000 1.24
+++ lib/am/dejagnu.am 19 Apr 2006 16:01:14 -0000
@@ -1,5 +1,5 @@
## automake - create Makefile.in from Makefile.am
-## Copyright (C) 1994, 1995, 1996, 1997, 2000, 2001, 2003
+## Copyright (C) 1994, 1995, 1996, 1997, 2000, 2001, 2003, 2006
## Free Software Foundation, Inc.
## This program is free software; you can redistribute it and/or modify
@@ -58,11 +58,13 @@
## If runtest can't be found, print a warning but don't die. It is
## pointless to cause a failure if the tests cannot be run at all.
if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \
- l='$(DEJATOOL)'; for tool in $$l; do \
- $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS)
$(RUNTESTFLAGS); \
+ exit_status=0; l='$(DEJATOOL)'; for tool in $$l; do \
+ if $$runtest $(AM_RUNTESTFLAGS) $(RUNTESTDEFAULTFLAGS)
$(RUNTESTFLAGS); \
+ then :; else exit_status=1; fi; \
done; \
else echo "WARNING: could not find \`runtest'" 1>&2; :;\
- fi
+ fi; \
+ exit $$exit_status
## ------------------- ##
Index: tests/dejagnu4.test
===================================================================
RCS file: /cvs/automake/automake/tests/dejagnu4.test,v
retrieving revision 1.4
diff -u -r1.4 dejagnu4.test
--- tests/dejagnu4.test 14 May 2005 20:28:54 -0000 1.4
+++ tests/dejagnu4.test 19 Apr 2006 16:01:00 -0000
@@ -1,5 +1,5 @@
#! /bin/sh
-# Copyright (C) 2003 Free Software Foundation, Inc.
+# Copyright (C) 2003, 2006 Free Software Foundation, Inc.
#
# This file is part of GNU Automake.
#
@@ -19,6 +19,7 @@
# Boston, MA 02110-1301, USA.
# Check that the DejaGnu rules work for a simple program and test case.
+# Also check PR 488: Failure of the first of several tools tested.
required=runtest
. ./defs || exit 1
@@ -86,3 +87,11 @@
test -f spanner.sum
$MAKE distcheck
+
+# Test for PR 488.
+sed 's/E\(verything\)/Not e\1/' hammer > thammer
+mv thammer hammer
+chmod +x hammer
+
+$MAKE check && exit 1
+:
- Re: automake/488: "make check" may seem to succeed when there are failed tests in multiple tools,
Ralf Wildenhues <=