lmi
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [lmi] Improving usability of automated tests


From: Greg Chicares
Subject: Re: [lmi] Improving usability of automated tests
Date: Fri, 10 Feb 2017 01:34:00 +0000
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.6.0

On 2017-02-08 20:56, Greg Chicares wrote:
> On 2017-02-08 16:28, Vadim Zeitlin wrote:
>> On Wed, 8 Feb 2017 01:59:02 +0000 Greg Chicares <address@hidden> wrote:
>> 
>> GC> On 2017-01-13 23:30, Greg Chicares wrote:
>> GC> [...]
>> GC> > /opt/lmi/src/lmi[0]$make $coefficiency unit_tests >../log 2>&1
>> GC> > /opt/lmi/src/lmi[0]$sed -f diagnostics.sed ../log 2>&1 |less -S
>> GC> > [and searching for '\*\*' therein, excluding the test-framework test]
>> GC> 
>> GC> s/, excluding.*[]]/d
>> GC> 
>> GC> commit 453051202f5bd5d4683b825f0a7cf429445823e2
>> GC> 
>> GC> It may seem like only slight incremental progress, but it removes
>> GC> a step that required manual activity and thought, and I run the
>> GC> unit tests many times every day.
>> 
>>  Do I understand correctly that I can now check for the unit test failures
>> just by searching for "***"? Or do I need to still filter out make errors
>> using diagnostics.sed? If so, perhaps we could use another marker for lmi
>> test failures, e.g. "!!!", to make it even easier?
> 
> Good point: 'make' uses a series of asterisks, so I'll cause these
> tests to use something else instead.

commit a306d67

> I don't yet feel confident about depending on any simple filtering rule.
> A test may...
>  - succeed entirely
>  - run to completion, but report errors
>  - start running, but then abend
>  - fail to compile
> and I need to make sure I'm either
>  - detecting absolutely all the abnormalities, or
>  - count the successes and compare against an expected total.
> But I'm trying not to spend any time on this yet, at least not until
> we're happy enough with input sequences. (Otherwise, I would add that
> maybe marking failures '????' and successes '!!!!' would be best;
> but there I go again.)

Of course I couldn't put a productivity improvement like that aside,
because inspecting test results is so tedious, and not having to do so
will pay a nice dividend many times every day.

Now I'm confident that we can count make failures, unit-test failures,
and unit-test successes, and compare them against the expectation
  {0, 0, 63 currently}
with one magical command. But first, here's a throwaway patch to
induce some failures:

---------8<--------8<--------8<--------8<--------8<--------8<--------8<-------
diff --git a/account_value_test.cpp b/account_value_test.cpp
index cbac144..159d27f 100644
--- a/account_value_test.cpp
+++ b/account_value_test.cpp
@@ -37,6 +37,6 @@ void AccountValueTest::Test()
 int test_main(int, char*[])
 {
     AccountValueTest::Test();
-    return EXIT_SUCCESS;
+    return gibberish;
 }
 
diff --git a/sandbox_test.cpp b/sandbox_test.cpp
index 401b86d..8ba2030 100644
--- a/sandbox_test.cpp
+++ b/sandbox_test.cpp
@@ -25,6 +25,8 @@
 
 int test_main(int, char*[])
 {
+    BOOST_TEST(false);
+
     return 0;
 }
 
--------->8-------->8-------->8-------->8-------->8-------->8-------->8-------

Now I run my magical command, and--annuit cœptis!--I get one of
those spurious errors that would be prevented by that github
pull request that I'm really, really not going to delve into
now because I need to get back to input sequences:

/opt/lmi/src/lmi[0]$make $coefficiency unit_tests 2>&1 | tee >(grep '\*\*\*') 
>(grep '????') >(grep '!!!!' --count | sed -e's/$/ tests succeeded/') >../log
make[2]: *** [account_value_test.o] Error 1
???? test failed: 0
???? 1 test errors detected; 0 tests succeeded
???? returning with error code 201
???? 
???? errors detected; see stdout for details
???? uncaught exception: std::runtime_error: Unable to parse xml file 
'sample.strata': Document is empty
???? returning with error code 200
???? 
???? errors detected; see stdout for details
61 tests succeeded

It took me forever (almost) to write that command in a way that
wouldn't spew the whole voluminous test log into my terminal.
Even adding ">/dev/null 2>&1" at the end wouldn't stop it.
Tellingly, a search engine indicated that nobody in the world
ever had such an intractable problem with process substitution
and redirection...because it turned out that I hadn't escaped
the triple asterisk expected from 'make'.

I'd welcome suggestions for further improvements, though I have
little hope that any will be as helpful as those three backslashes.

Please let me know if you're using 'fancy.make' (no, I didn't
think so) because otherwise I'll remove it (although it did have
the virtue of escaping asterisks).




reply via email to

[Prev in Thread] Current Thread [Next in Thread]