[lmi] Adding a useful "make check" target

[Vadim Zeitlin]

 Hello,

 Sorry for returning to this topic again, I know we've discussed this in
the past, but unfortunately I don't think we've ever arrived at any
conclusion, so I'd like to make another attempt to do it.

 As a reminder, my problem is that I'd like to have a simple way of running
the lmi test suite and getting a clear output about the success and,
especially, failures in it and I don't think the official lmi makefiles
provide any way to do this. Just doing "make unit_tests" floods the
terminal with tons of output that this is not useful at all and, AFAIK, the
recommended way of running the test suite is to use the following command,
appearing in nychthemeral_test.sh:

make "$coefficiency" --output-sync=recurse unit_tests 2>&1 \
  | tee >(grep '\*\*\*') >(grep \?\?\?\?) >(grep '!!!!' --count | xargs printf 
'%d tests succeeded\n') >"$log_dir"/unit_tests

 However this is still not ideal for several reasons. First, this is
obviously much less simple than the standard "make check" and it's
definitely not something you can just type into the terminal, which doesn't
exactly motivate you to run the test suite more often.

 Second, the output from this command is still not very useful. E.g. here
is what I get in one of my builds right now:
---------------------------------- >8 --------------------------------------
???? uncaught exception: std::runtime_error: Instantiation failed.
???? returning with error code 200
????
???? errors detected; see stdout for details
???? uncaught exception: std::runtime_error: Instantiation failed.
???? returning with error code 200
????
???? errors detected; see stdout for details
???? test failed:
???? test failed: 0
???? test failed:
???? test failed: 0
???? uncaught exception: std::runtime_error: Unable to parse xml file 
'Z:/opt/lmi/data': Permission denied
???? returning with error code 200
????
???? errors detected; see stdout for details
???? test failed:
???? test failed: 0
???? test failed:
???? test failed: 0
???? 4 test errors detected; 60 tests succeeded
???? returning with error code 201
????
???? errors detected; see stdout for details
67 tests succeeded
---------------------------------- >8 --------------------------------------
Leaving aside the issue of why do these errors happen, the obvious problem
here is that the output doesn't say which tests failed nor, in fact, even
clearly mentions the number of failures: you can see "4 test errors" but
also "60 tests succeeded" and "67 tests succeeded", and neither is
compatible with there being 71 tests in all (which is something not shown
in this output neither). You can painstakingly extract this information
from the log file, but why should it be necessary to do it?

 Let's compare the above with "make check" output, using the makefile
created by configure (this ASCII-only email doesn't show the nice green and
red colours used for PASS and FAIL labels respectively):
---------------------------------- >8 --------------------------------------
PASS: test_account_value
FAIL: test_authenticity
FAIL: test_cache_file_reads
PASS: test_callback
[...snip many more similar lines...]
PASS: test_bourn_cast
PASS: test_expression_template_0
PASS: test_vector
============================================================================
Testsuite summary for lmi 1.0.0
============================================================================
# TOTAL: 71
# PASS:  62
# SKIP:  0
# XFAIL: 0
# FAIL:  9
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to gchicares@sbcglobal.net
============================================================================
make[2]: *** [Makefile:20242: test-suite.log] Error 1
make[2]: Leaving directory '/home/zeitlin/build/lmi'
make[1]: *** [Makefile:20350: check-TESTS] Error 2
make[1]: Leaving directory '/home/zeitlin/build/lmi'
make: *** [Makefile:21053: check-am] Error 2
---------------------------------- >8 --------------------------------------

 The output here is longer, but incomparably more useful. It immediately
shows the number of failed tests and clearly shows which tests fail right
in front of you (again, "FAIL" labels stand out thanks to their colour).
Moreover, the make command actually exits with an error code, as expected,
rather than always returning 0, whether all tests succeeded or not, as the
lmi-specific incantation above does. The details of the failures can still
be extracted from the log file, but you can also just run one of the
failing tests directly to see them immediately.


 Would you please consider improving lmi makefiles to do something more
similar to what automake test harness does? Having pretty colours is not
mandatory, but I'd really like to have at least the following:

1. Simple make target exiting with 0 only if the test suite passed.
2. Clear indication of the failed tests (i.e. their names).
3. The detailed error messages available in a file somewhere, without
   having to write any complicated shell pipelines or copying them from
   somewhere every time.

 I would, of course, be ready to implement the above in lmi makefiles if
you agree that this would be useful, but as this is inevitably going to
take some time (as anything non-trivial done in make language), I'd like to
check that you actually do agree with this.

 Of course, if you don't, I'd also like to know why, i.e. what possible
advantages of the current approach, that I fail to notice, do you see?

 Thanks in advance,
VZ

pgpNXPddgjhSC.pgp
Description: PGP signature