[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: quiet mode for make check (silence passing tests, list failing tests
From: |
Chris Pickett |
Subject: |
Re: quiet mode for make check (silence passing tests, list failing tests) |
Date: |
Mon, 03 Mar 2008 12:13:06 -0500 |
User-agent: |
Thunderbird 1.5.0.14 (Macintosh/20071210) |
Hi Ralf and Akim and co.,
(Please include me in any reply, I am not subscribed.)
Ralf Wildenhues wrote:
* Chris Pickett wrote on Wed, Jan 23, 2008 at 10:56:29AM CET:
I have one test program for each module in my library. Each writes to
stdout and stderr. Since I have many tests, this generates a lot of
output when I run `make check'. I would like to change `make check' so
that it:
1) Causes only those tests that fail to print anything
2) Additionally lists the tests that fail at the end
But it's also possible inside Makefile.am. In fact, there have been
patches floating around for parallel testsuites, which can hopefully be
integrated before 1.11, which deal smartly with verbose output (keeping
verbose log files of failed tests).
I found the check.mk patch (set of redefinitions) you were referring to:
http://www.mail-archive.com/address@hidden/msg12373.html
The original is available here:
https://trac.lrde.org/vaucanson/browser/trunk/build-aux/check.mk
I adapted it for my own project here:
https://svn.sable.mcgill.ca/sable/spmt/libspmt/build-aux/check.mk
These are the differences in order if you diff the two files (comments
are painful with trailing backslashes):
1) Remove a blank line for each individual test log:
FAIL: <test>
============
<--- here
Output from test.
2) Don't print PASS: <test> or FAIL: <test> for each test as it
executes. This is just too verbose for me; if everything passes, all I
want is a green bar (box) at the bottom. If something fails, I want the
full log, not just a message that it failed.
3) Don't show the exit status for a test. Too verbose for me.
4) Remove the boilerplate echoing the PACKAGE_STRING and the name of the
file into $(TEST_SUITE_LOG) since this is obvious from position. Also
remove some idiosyncratic formatting stuff (".. contents:: :depth: 2").
5) Remove the "See $(subdir)/$(TEST_SUITE_LOG)" message (see change #7).
I want to keep the red bar (box) message concise.
6) s/Please report it to/Please report to/ in line with current Automake.
7) Always send $(TEST_SUITE_LOG) to stdout. If a test fails, I'm going
to check what happened anyways. Originally, a VERBOSE variable guarded
that, but I think VERBOSE should be default.
8) Show $(TEST_SUITE_LOG) before the am__text_box announcing the total
number of tests that pass/fail. I think you want the red/green bar to
appear at the bottom of the output and not have to scroll back for it.
9) Always remove $(TEST_SUITE_LOG) so that the pass/fail output is
always regenerated. This does not significantly affect running time.
Additionally, in order to use this check.mk, I added the following to my
tests/Makefile.am:
# this rule seems necessary to quiet errors
# about check.mk not knowing how to create .log
.log:
LAZY_TEST_SUITE = 1
TEST_LOGS = $(TESTS:=.log)
# parallel/quiet make check from vaucanson-1.1.1
include $(top_srcdir)/build-aux/check.mk
I don't really know about the .log rule but meh, it works. Also, I
tried using the EXTRA_PROGRAMS variable, and while it apparently let the
tests run concurrently with the compilations, I prefer the output I get
with make check (`<test> is up to date.') and also since change #2
removes the immediate feedback on pass/fail I still have to wait for
everything to run anyway. Trying to look for a red message and hit
Ctrl-c and then cat the appropriate tests/<test>.log is not what I want.
Anyway, thanks Akim for the hard work on check.mk. It's much much
better for me. By the way, my output looks like this:
gmake check-TESTS
gmake[2]: Entering directory `$HOME/libspmt/tests'
rm -f test-suite.log
gmake[3]: Entering directory `$HOME/libspmt/tests'
FAIL: predictor
===============
Running suite(s): predictor
Assertion failed: predictor != NULL, file predictor.c, line 127
66%: Checks: 3, Failures: 1, Errors: 0
predictor.c:46:F:core:check_half_width_return_value:0: Assertion
'(spmt_predictor_predict (predictor) == 3)' failed
==================================================
1 of 48 tests failed
Please report to chris.pickett AT mail.mcgill.ca
==================================================
gmake[3]: *** [test-suite.log] Error 1
gmake[3]: Leaving directory `$HOME/libspmt/tests'
gmake[2]: *** [check-TESTS] Error 2
gmake[2]: Leaving directory `$HOME/libspmt/tests'
gmake[1]: *** [check-am] Error 2
gmake[1]: Leaving directory `$HOME/libspmt/tests'
gmake: *** [check-recursive] Error 1
where the box is in red. In case you're wondering, I'm using the Check
unit testing framework for C. It works quite well for me, but I'm also
a maintainer. When things do work, the output looks like this now:
make check-TESTS
rm -f test-suite.log
=====================
All 48 tests passed
=====================
where the box is in green. That's on a different machine (OS X, not
AIX), and for some reason I don't get the entering/leaving messages.
Not sure which I prefer. Of course, all the .log files are available
for individual tests, regardless of pass/fail status.
Finally, I noticed that check.mk is GPL, where as the Autoconf stuff in
build-aux comes with this notice:
# As a special exception to the GNU General Public License, if you
# distribute this file as part of a program that contains a
# configuration script generated by Autoconf, you may include it under
# the same distribution terms that you use for the rest of that program.
I don't know if it matters. I'm assuming that using check.mk as GPL
doesn't mean that my library is forced to be GPL:
## This file comes from vaucanson-1.1.1. Since it only affects the
## build process, it is assumed that libspmt is not forced to be GPL.
but I guess I would feel better if that exception notice was there?
(That `?' has a mysterious meaning.) I suppose if check.mk gets into
Automake (which I hope) then everything will straighten out anyway.
Cheers,
Chris
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- Re: quiet mode for make check (silence passing tests, list failing tests),
Chris Pickett <=