[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [lmi] Improving usability of automated tests
From: |
Greg Chicares |
Subject: |
Re: [lmi] Improving usability of automated tests |
Date: |
Fri, 10 Feb 2017 01:34:00 +0000 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.6.0 |
On 2017-02-08 20:56, Greg Chicares wrote:
> On 2017-02-08 16:28, Vadim Zeitlin wrote:
>> On Wed, 8 Feb 2017 01:59:02 +0000 Greg Chicares <address@hidden> wrote:
>>
>> GC> On 2017-01-13 23:30, Greg Chicares wrote:
>> GC> [...]
>> GC> > /opt/lmi/src/lmi[0]$make $coefficiency unit_tests >../log 2>&1
>> GC> > /opt/lmi/src/lmi[0]$sed -f diagnostics.sed ../log 2>&1 |less -S
>> GC> > [and searching for '\*\*' therein, excluding the test-framework test]
>> GC>
>> GC> s/, excluding.*[]]/d
>> GC>
>> GC> commit 453051202f5bd5d4683b825f0a7cf429445823e2
>> GC>
>> GC> It may seem like only slight incremental progress, but it removes
>> GC> a step that required manual activity and thought, and I run the
>> GC> unit tests many times every day.
>>
>> Do I understand correctly that I can now check for the unit test failures
>> just by searching for "***"? Or do I need to still filter out make errors
>> using diagnostics.sed? If so, perhaps we could use another marker for lmi
>> test failures, e.g. "!!!", to make it even easier?
>
> Good point: 'make' uses a series of asterisks, so I'll cause these
> tests to use something else instead.
commit a306d67
> I don't yet feel confident about depending on any simple filtering rule.
> A test may...
> - succeed entirely
> - run to completion, but report errors
> - start running, but then abend
> - fail to compile
> and I need to make sure I'm either
> - detecting absolutely all the abnormalities, or
> - count the successes and compare against an expected total.
> But I'm trying not to spend any time on this yet, at least not until
> we're happy enough with input sequences. (Otherwise, I would add that
> maybe marking failures '????' and successes '!!!!' would be best;
> but there I go again.)
Of course I couldn't put a productivity improvement like that aside,
because inspecting test results is so tedious, and not having to do so
will pay a nice dividend many times every day.
Now I'm confident that we can count make failures, unit-test failures,
and unit-test successes, and compare them against the expectation
{0, 0, 63 currently}
with one magical command. But first, here's a throwaway patch to
induce some failures:
---------8<--------8<--------8<--------8<--------8<--------8<--------8<-------
diff --git a/account_value_test.cpp b/account_value_test.cpp
index cbac144..159d27f 100644
--- a/account_value_test.cpp
+++ b/account_value_test.cpp
@@ -37,6 +37,6 @@ void AccountValueTest::Test()
int test_main(int, char*[])
{
AccountValueTest::Test();
- return EXIT_SUCCESS;
+ return gibberish;
}
diff --git a/sandbox_test.cpp b/sandbox_test.cpp
index 401b86d..8ba2030 100644
--- a/sandbox_test.cpp
+++ b/sandbox_test.cpp
@@ -25,6 +25,8 @@
int test_main(int, char*[])
{
+ BOOST_TEST(false);
+
return 0;
}
--------->8-------->8-------->8-------->8-------->8-------->8-------->8-------
Now I run my magical command, and--annuit cœptis!--I get one of
those spurious errors that would be prevented by that github
pull request that I'm really, really not going to delve into
now because I need to get back to input sequences:
/opt/lmi/src/lmi[0]$make $coefficiency unit_tests 2>&1 | tee >(grep '\*\*\*')
>(grep '????') >(grep '!!!!' --count | sed -e's/$/ tests succeeded/') >../log
make[2]: *** [account_value_test.o] Error 1
???? test failed: 0
???? 1 test errors detected; 0 tests succeeded
???? returning with error code 201
????
???? errors detected; see stdout for details
???? uncaught exception: std::runtime_error: Unable to parse xml file
'sample.strata': Document is empty
???? returning with error code 200
????
???? errors detected; see stdout for details
61 tests succeeded
It took me forever (almost) to write that command in a way that
wouldn't spew the whole voluminous test log into my terminal.
Even adding ">/dev/null 2>&1" at the end wouldn't stop it.
Tellingly, a search engine indicated that nobody in the world
ever had such an intractable problem with process substitution
and redirection...because it turned out that I hadn't escaped
the triple asterisk expected from 'make'.
I'd welcome suggestions for further improvements, though I have
little hope that any will be as helpful as those three backslashes.
Please let me know if you're using 'fancy.make' (no, I didn't
think so) because otherwise I'll remove it (although it did have
the virtue of escaping asterisks).