Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer

[Stefano Lattarini]

Hi Ralf, thanks for the reviews.

On Monday 18 July 2011, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Fri, Jul 15, 2011 at 12:36:17AM CEST:
> > The new code for parsing the testsuite-generated `.log' files,
> > as introduced in commit `v1.11-872-gc96b881', considers each
> > `:test-result:' field anywhere in a `.log' file as a declaration
> > of a test result, and accounts for it as such in the testsuite
> > summary.  Unfortunately this could easily cause spurious test
> > failures being reported in the testsuite summary.  This happened
> > in practice with the Automake's own testsuite; for example:
> > 
> >   $ make check TESTS='check12-p.test'; echo exit: $?
> >   ...
> >   PASS: check12-p.test
> >   =====================================
> >   4 of 5 tests failed
> >   See tests/test-suite.log
> >   Please report to address@hidden
> >   =====================================
> >   make[2]: *** [test-suite.log] Error 1
> >   make: *** [check-am] Error 2
> >   exit: 2
> > 
> > This change introduces a new special `:test-result:' "END", that,
> > when seen, prevents the rest of the log file from being parsed.
> > 
> > For more information, refer to the thread:
> > <http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html>
> > 
> > * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log
> > file as soon as the special ":test-result:END" directive is seen.
> > Related changes and enhancements.
> > * lib/test-driver: Protect the rest of the log after the result
> > lined with a ":test-result:END" directive.
> > * tests/parallel-tests-no-spurious-summary.test: New test.
> > * tests/test-driver-end-test-results.test: Likewise.
> > * tests/Makefile.am (TESTS): Update.
> 
> I'm still not sold on this.
>
I'm surely open to discuss changes to this new "log format" (which I
admittedly concocted quite in a hurry and without much thought), and to
later amend it as appropriate.  But I also think that, for the moment,
this patch definitely improves the previous situation, as it removes
various spurious failures in our own testsuite without requiring the use
of any hack, nor any extensive testsuite tweaking; so I'd like to check
it in anyway.  Then we can (and will!) think how to improve the situation
in follow-up patches (that is OK as long as we do so before merging the
'test-protocols' branch into 'master').

> It is not as robust as a test protocol
> could be; also I haven't seen this approach being used in any other test
> suite environments.  Whether some line is considered having a result or
> not depends on (possibly far-away) context, on whether aggregation with
> other results has happened.  When passed through email, misquoting can
> change not only the interpretation of the misquoted text (which could be
> expected) but possibly also of later, correctly quoted (or not quoted)
> text.
>
Good points.  Hmm... the more I think about this issue, the more I'm
convinced that we are (ok, I am ;-) starting to unduly use the `.log'
files for two ortoghonal purposes:
 1. keeping a verbose and detailed log of a test script run; and
 2. keeping a sort of "dumb database" of test results, which is used
    to determine the final testsuite summary (and also by "make recheck"
    to decide which tests to re-run).
Maybe it's time we begin to use two different set of files for this two
different roles?  E.g., for each `foo.test', we generate a `foo.log' file
and a, say, `foo.tdat' (Test DATa file) containing the test results and
possibly other metadata used by our harness?  This might also allow for
easier extensibility of such metadata in the future.

> Please, this is really important: we need to research the other test
> protocols, what they do to be robust here.  Don't NIH here, because the
> experience we have is not enough to not mess up this.  Consider this
> research as part of the work needed for the SoC assignment, it is a very
> important part (as it is not easily corrected once released).
>
Honestly, I really don't have a clue about where to start looking for such
"prior art", and my rather naive Google searches haven't returned anything
useful.  Do you have any suggestion?  Even a loose one might be an helpful
starting point.

Thanks,
  Stefano