Re: [GSoC] Early design discussion for TAP/SubUnit support in automake.

[Stefano Lattarini]

Reference:
 <http://lists.gnu.org/archive/html/automake/2011-04/msg00069.html>

On Friday 29 April 2011, Stefano Lattarini wrote:
>
> First Tentative Decisions and Questions on the User Interface
> =============================================================
> 
> [CUT]
> 
>  2. New automake option `tests-protocol'
>  ---------------------------------------
> 
>  The Tap/SubUnit support in the Automake-generated testsuite drivers
>  should be enabled by a new (argument-requiring) option `tests-protocol',
>  that will be used to specify the level of support for, detection of, and
>  enforcing of SubUnit/TAP streams.
> 
>  The possible values for `tests-protocol' will be:
>   - tests-protocol=tap
>     All test scripts are expected to use the TAP protocol.
>   - tests-protocol=subunit
>     All test scripts are expected to use the SubUnit protocol.
>   - tests-protocol=adaptive
>     Each test script is expected to print on its first line of output
>     which protocol it uses (the exact format of this special line is
>     still to be determined); if this line is unrecognized, the driver
>     should assume that the test script uses no protocol.  Also, in
>     this case, we should continue to honour XFAIL_TESTS.  All of this
>     should help to maximize backward-compatibility.
>
Thinking again about this, it might be worth trying to be even more consistent
with the existing parallel-tests functionality, and use an `ext_TEST_PROTOCOL'
variable (or similar) instead of a global `tests-protocol' option.  With some
tweaking to the post-processing of `.log' files done in `lib/am/check.am' (to
generate `$(TEST_SUITE_LOG)'), this might allow greater code reuse and a more
consistent API.

I've started experimenting with this idea, and I'm not seeing any obvious
shortcoming right now.  I'm hoping I'll be able to post some experimental
patches soon enough.

As usual, if you have suggestions, ideas or objection, I'd ble glad to hear
them.

>
> [CUT]
>
>  4. RST support and HTML generation: should be dropped?
>  ------------------------------------------------------
> 
>   I haven't really looked into this yet, so I don't know how much additional
>   work (if any!) would be to continue to support generation of HTML reports
>   from the testsuite-generated `.log' files in the new TAP/SubUnit testsuite
>   driver.  Still, one question that bugs me is: how much is this HTML report
>   generation used in Real Life?  Is it really worth to support it out of the
>   box, or could we leave it to the interested used to write their own rules
>   (maybe in a reusable `*.am' fragment)?
>
BTW, I also hope that new interface I'm planning will make it easier to
implement HTML report generation also for TAP and SubUnit tests, with a
consistent reuse of the existing code.  In which case my considerations
above will become moot.

Regards,
  Stefano