Re: [GSoC] Early design discussion for TAP/SubUnit support in automake.

[Robert Collins]

Very sorry for the slow response, been EBUSY with real-life.

On Sun, May 22, 2011 at 11:42 PM, Stefano Lattarini
<address@hidden> wrote:
> On Sunday 22 May 2011, Ralf Wildenhues wrote:
>> Hi Stefano, and sorry for the long delay,
>>
> No problem, you had warned me in due time about such possible delays this
> month; so there's really no need to apologize.
>
>> * Stefano Lattarini wrote on Fri, Apr 29, 2011 at 11:21:06AM CEST:
>> > Now that my GSoC application "automake - Interfacing with a test protocol
>> > like TAP or subunit" has been officially accepted, I'd like to start
>> > discussing with the community some early, high-level design and interface
>> > decisions.
>>
>>
>> >  1. Reuse parallel-tests "framework"
>> >  -----------------------------------
>> >
>> >   The new TAP/SubUnit support should reuse as much of the current
>> >   parallel-tests implementation and semantics as possible.  In particular,
>> >   it should be able to run different test scripts in parallel, generate a
>> >   `.log' file for each test script and a "summarizing" `test-suite.log'
>> >   file, honour the make variables AM_TESTS_ENVIRONMENT, TESTS_ENVIRONMENT
>> >   and AM_COLOR_TESTS and the environment variable VERBOSE, and support
>> >   different extensions for the test scripts, with extension-specific "log
>> >   compilers" and flags (the stuff enabled by TESTS_EXTENSIONS,
>> >   <ext>_LOG_COMPILER, etc.).
>>
>> Sounds all sane.
>>
>> >   The XFAIL_TESTS variable might be still supported for the sake of
>> >   backward-compatibility (see below for the details), but it should be
>> >   deprecated, since TAP and SubUnit offer better and more granular ways
>> >   to express expected failures.
>>
>> OK.
>>
>> In another mail:
>> > Thinking again about this, it might be worth trying to be even more 
>> > consistent
>> > with the existing parallel-tests functionality, and use an 
>> > `ext_TEST_PROTOCOL'
>> > variable (or similar) instead of a global `tests-protocol' option.  With 
>> > some
>> > tweaking to the post-processing of `.log' files done in `lib/am/check.am' 
>> > (to
>> > generate `$(TEST_SUITE_LOG)'), this might allow greater code reuse and a 
>> > more
>> > consistent API.
>> >
>> > I've started experimenting with this idea, and I'm not seeing any obvious
>> > shortcoming right now.  I'm hoping I'll be able to post some experimental
>> > patches soon enough.
>>
>> Allowing to specify that per-test is a good idea for transitioning test
>> suites.
>>
> About this, in my first two "tentative" patches:
>  <http://lists.gnu.org/archive/html/automake-patches/2011-05/msg00093.html>
> I've taken an even more general approach, allowing the developer to define
> and use his own program(s) to:
>  1. launch his test scripts,
>  2. interpreter their results,
>  3. display these results on screen, and
>  4. format and generate the log files.
> All of this is attainable simply by assigning a variable `LOG_WRAPPER'
> (and extensione-specific counterparts of it), and, well, obviously
> providing a real "driver" script that obeys to minimal command-line
> interface (so that it can grasp the options the Automake-generated
> Makefiles passes to it).  Then we will hopefully be able to implement
> our TAP/SubUnit parsers on the top of this feature (thus making it
> indirectly more tested, which is always good for a new feature).

If the subunit parser just gets all the output from the test script,
you might want to use the subunit parser itself: folk in the
bootstrap-set couldn't (obviously), but most other environments can
use high level languages freely. (I don't object to more parser
implementations existing, I'm just thinking about reuse where
possible.

>> I hope to look into the posted patches later today.
>>
> About this, please note that I might be AFK until this evening.  So have
> no haste.
>
>> >  2. New automake option `tests-protocol'
>> >  ---------------------------------------
>> >
>> >  The Tap/SubUnit support in the Automake-generated testsuite drivers
>> >  should be enabled by a new (argument-requiring) option `tests-protocol',
>> >  that will be used to specify the level of support for, detection of, and
>> >  enforcing of SubUnit/TAP streams.
>> >
>> >  The possible values for `tests-protocol' will be:
>> >   - tests-protocol=tap
>> >     All test scripts are expected to use the TAP protocol.
>> >   - tests-protocol=subunit
>> >     All test scripts are expected to use the SubUnit protocol.
>> >   - tests-protocol=adaptive
>>
>> The way you describe "adaptive", it sounds like it should rather be
>> named something like "detect" or "detected" or so.
>>
> I'd like to withdraw this proposal now that we can define per-extension test
> protocols.  Having our hypotetical "client developer" rename his test scripts
> as they get converted to use TAP/SubUnit is IMO better than we having to
> implement in Automake a probably non-trivial "metaparser" that could end up
> being scarcerly used anyway.  WDYT?
>
>> >     Each test script is expected to print on its first line of output
>> >     which protocol it uses (the exact format of this special line is
>> >     still to be determined); if this line is unrecognized, the driver
>> >     should assume that the test script uses no protocol.  Also, in
>> >     this case, we should continue to honour XFAIL_TESTS.  All of this
>> >     should help to maximize backward-compatibility.
>>
>>
>>
>> >  3. Console output from the test driver
>> >  ---------------------------------------
>> >
>> >   This output should remain as close as possible to the one already
>> >   provided by the current parallel-tests driver.  The following example
>> >   should help to clarify what I mean.
>> >
>> >   Assume we are using `tests-protocol=adaptive', and let TESTS be defined
>> >   to "pass.test skip.test subunit.test tap.test".  Here, `pass.test' and
>> >   `skip.test' are test scripts that use no protocol (and that exit with
>> >   status `0' and `77' respectively), `subunit.test' is a test script using
>> >   the SubUnit protocol and containing four testcases (one passing, one
>> >   failing, one skipped and one which incurs in an expected failure), and
>> >   `tap.test' is a test script using the TAP protocol which runs two
>> >   successful testcases, then encounters an internal error and bails out
>> >   (using the "Bail out!" directive).
>> >
>> >   With such a setup, this is the output I'd expect from "make check":
>> >     PASS: pass.test
>> >     SKIP: skip.test
>> >     PASS: subunit.test [testcase name/description]
>> >     FAIL: subunit.test [testcase name/ description]
>> >     SKIP: subunit.test [testcase name/description] [reason for skipping]
>> >     XFAIL: subunit.test [testcase name/description] [failure reason]
>> >     PASS: tap.test [testcase name/description]
>> >     PASS: tap.test [testcase name/description]
>> >     ERROR: tap.test [reason for the bailout]
>> >
>> >   Of course, the `color-tests' option should make the above output properly
>> >   colorized; the attached html file shows what colors I'd expect.
>>
>> Wrt. to the details of the output, I wouldn't fix things too early.
>> Two things two consider here:
>> - if tests are run in parallel, you want to avoid intermixing output
>>   from different tests as much as possible,
>>
> I disagree.  Once the test script name is printed in each line which report
> the result of one of its test cases, I see no issue with intermixed lines.
> I.e., I don't believe that this:
>
>  PASS: pass.test
>  PASS: subunit.test [foo]
>  PASS: tap.test [t1]
>  SKIP: skip.test
>  PASS: tap.test [t2]
>  FAIL: subunit.test [bar]
>  SKIP: subunit.test [baz] [reason for skipping]
>  ERROR: tap.test [reason for the bailout]
>  XFAIL: subunit.test [quux] [failure reason]
>
> is any less clear than this:
>
>  PASS: pass.test
>  SKIP: skip.test
>  PASS: subunit.test [testcase name/description]
>  FAIL: subunit.test [testcase name/ description]
>  SKIP: subunit.test [testcase name/description] [reason for skipping]
>  XFAIL: subunit.test [testcase name/description] [failure reason]
>  PASS: tap.test [testcase name/description]
>  PASS: tap.test [testcase name/description]
>  ERROR: tap.test [reason for the bailout] XXX
>
>>   but even more than that, you want to avoid intermixing within lines.
>>
> OTOH, I do believe this is a real concern, to be carefully addressed and
> tested for.  Thanks for bringing this up.

For Both TAP and subunit the test script running needs to feed into a
single parser: the issue with intermingling then is all about how the
parsers structure their output.

Ideally you would use a threadsafe structure.

For subunit, *if* you were working in python you could use the
existing multithreaded logic testrepository uses, which can mingle
multiple streams correctly, preserving datestamps - a single ui parser
then handles all the combined output.

>>   Gathering output and
>>   printing it all at once can help here, but the downside is possibly
>>   losing output if tests are interrupted, or the driver doesn't finish
>>   correctly.
>> - some sort of bypass of the (currently hidden) output is helpful,
>>   some gnulib-using packages do that already by redirecting stderr
>>   before running the test, and printing a reason for skipping from
>>   within the test.
>>
> [thinking about this put on the TODO list]
>
>> I think the basic premise to take into account is that the test driver
>> author (you) may not know all requirements a future test author may
>> have.
>>
> That's why I like my current idea of allowing the client developers
> to provide their own testsuite driver.
>
>> >  4. RST support and HTML generation: should be dropped?
>> >  ------------------------------------------------------
>>
>> Good question.  You could do a poll on the automake list (in a separate
>> mail & thread, to gain visbility).  Or just defer the implementation
>> for later.
>>
>> In another mail:
>> > BTW, I also hope that new interface I'm planning will make it easier to
>> > implement HTML report generation also for TAP and SubUnit tests, with a
>> > consistent reuse of the existing code.  In which case my considerations
>> > above will become moot.
>>
>> Cool.
>>
> The downside of this is that it might place additional burdens on the
> writer of test drivers. Hmmm... maybe a new option "no-html-test" or
> similar is warranted?

At least for subunit, and I believe TAP afficiondos will agree for
TAP, you'd generally run subunit2html [made upname] to generate a html
report. An example of such a formatter is subunit2junithtml, and I
believe samba have a perl based html formatter for subunit already
(Jelmer will know more about that).

-Rob