RE: [lmi] Problem of the week: testing a testing tool

[Boutin, Wendy]

Greg Chicares wrote:
> make unit_tests build_type=mpatrol unit_test_targets=input_test
> [add '.exe' at the end for msw]
> 
> Running input_test:
>   Speed tests...
>   Overhead: [3.931e+000] 1 iteration took 3930 milliseconds
>   Vector  : [9.064e+000] 1 iteration took 9063 milliseconds
>   Read    : [1.#IOe+000] 0 iterations took 15031 milliseconds
>   Write   : [7.772e+000] 1 iteration took 7771 milliseconds
>   'cns' io: [1.#IOe+000] 0 iterations took 54502 milliseconds
>   'ill' io: [1.#IOe+000] 0 iterations took 14409 milliseconds
> 
> .... 27 tests succeeded
> no errors detected
>   mpatrol: total warnings:    0
>   mpatrol: total errors:      0
> 
> 0. What's obviously wrong here on the face of it?

These three lines:

  Read    : [1.#IOe+000] 0 iterations took 15031 milliseconds
  'cns' io: [1.#IOe+000] 0 iterations took 54502 milliseconds
  'ill' io: [1.#IOe+000] 0 iterations took 14409 milliseconds

are wrong in several ways:
- [1.#IOe+000] identifies a limit problem when converting a certain
    type (I think)
    http://mail.python.org/pipermail/tutor/2006-November/051290.html
- zero iterations should take zero milliseconds, but the output attributes
    many more milliseconds than zero

This line:
  Write   : [7.772e+000] 1 iteration took 7771 milliseconds
seems to demonstrate a potential rounding discrepancy between
  7.772e+000
and
  7771 milliseconds
I suspect something's gone awry with the timer.

Given what I can see on the face of it, these results don't correspond
to with the results I observe above:
> .... 27 tests succeeded

I don't actually count twenty seven tests.

> no errors detected

This *is* an error: [1.#IOe+000]

>   mpatrol: total warnings:    0
>   mpatrol: total errors:      0

And because of my last two comments, I'm not convinced these
last two lines are actually correct either, but that's pure
speculation because of the faulty data surrounding it. And,
as Rick mentioned, I'm having a little trouble with mpatrol,
so I'm only guessing you added that output locally.

> 1. Which revisions introduced defects observable above?

This one looks like a candidate:
  
http://cvs.savannah.nongnu.org/viewcvs/lmi/timer.hpp?root=lmi&sortby=date&r2=1.11&r1=1.10
but I can reproduce the problem in earlier revisions if I
make this change:

 -  std::string TimeAnAliquot(F f, double max_seconds = 1.0)
 +  std::string TimeAnAliquot(F f, double max_seconds = 0.001)

> 2. How could those defects have been detected automatically?

Setting up a unit test for each condition that causes the problem
and running the tests regularly. I think 'make cvs_ready' could've
assisted with this one because it actually uses mpatrol, which
illuminated symptoms of the problems. Although I don't believe
mpatrol actually contributes to the cause. I think mpatrol helped
track it down. That tool makes programs run so much slower that
it may have caused it to hit a maximum limit; this area starts
getting a little shady for me.
 
> 3. How could those defects have been prevented?

If unit tests existed that covered these particular problems,
then regression testing the unit tests and analyzing the results
before committing the changes should help prevention. Today, I'm
not certain (because I couldn't test it) if 'cvs_ready' would've
helped because I aborted my attempt to fix that problem in light
of getting through this one.

Although the rounding discrepancies I saw in the timer output
may have been catchable if the unit tests were regression tested.
Running 'nychthemeral_test' and analyzing its results in the
duration it represents more regularly could've also helped
prevent this.

> 4. How should those defects be removed?

I'm not exactly sure that I truly identified the problems yet, so
I haven't entered the solution domain. But more generally speaking,
unit tests should be developed that cover the "problem" conditions
and none should fail.

---------------------------------------------------------
This e-mail transmission may contain information that is proprietary, 
privileged and/or confidential and is intended exclusively for the person(s) to 
whom it is addressed. Any use, copying, retention or disclosure by any person 
other than the intended recipient or the intended recipient's designees is 
strictly prohibited. If you are not the intended recipient or their designee, 
please notify the sender immediately by return e-mail and delete all copies. 

---------------------------------------------------------