[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [GSoC] Early design discussion for TAP/SubUnit support in automake.
From: |
Robert Collins |
Subject: |
Re: [GSoC] Early design discussion for TAP/SubUnit support in automake. |
Date: |
Wed, 6 Jul 2011 17:52:05 +1200 |
Very sorry for the slow response, been EBUSY with real-life.
On Sun, May 22, 2011 at 11:42 PM, Stefano Lattarini
<address@hidden> wrote:
> On Sunday 22 May 2011, Ralf Wildenhues wrote:
>> Hi Stefano, and sorry for the long delay,
>>
> No problem, you had warned me in due time about such possible delays this
> month; so there's really no need to apologize.
>
>> * Stefano Lattarini wrote on Fri, Apr 29, 2011 at 11:21:06AM CEST:
>> > Now that my GSoC application "automake - Interfacing with a test protocol
>> > like TAP or subunit" has been officially accepted, I'd like to start
>> > discussing with the community some early, high-level design and interface
>> > decisions.
>>
>>
>> > 1. Reuse parallel-tests "framework"
>> > -----------------------------------
>> >
>> > The new TAP/SubUnit support should reuse as much of the current
>> > parallel-tests implementation and semantics as possible. In particular,
>> > it should be able to run different test scripts in parallel, generate a
>> > `.log' file for each test script and a "summarizing" `test-suite.log'
>> > file, honour the make variables AM_TESTS_ENVIRONMENT, TESTS_ENVIRONMENT
>> > and AM_COLOR_TESTS and the environment variable VERBOSE, and support
>> > different extensions for the test scripts, with extension-specific "log
>> > compilers" and flags (the stuff enabled by TESTS_EXTENSIONS,
>> > <ext>_LOG_COMPILER, etc.).
>>
>> Sounds all sane.
>>
>> > The XFAIL_TESTS variable might be still supported for the sake of
>> > backward-compatibility (see below for the details), but it should be
>> > deprecated, since TAP and SubUnit offer better and more granular ways
>> > to express expected failures.
>>
>> OK.
>>
>> In another mail:
>> > Thinking again about this, it might be worth trying to be even more
>> > consistent
>> > with the existing parallel-tests functionality, and use an
>> > `ext_TEST_PROTOCOL'
>> > variable (or similar) instead of a global `tests-protocol' option. With
>> > some
>> > tweaking to the post-processing of `.log' files done in `lib/am/check.am'
>> > (to
>> > generate `$(TEST_SUITE_LOG)'), this might allow greater code reuse and a
>> > more
>> > consistent API.
>> >
>> > I've started experimenting with this idea, and I'm not seeing any obvious
>> > shortcoming right now. I'm hoping I'll be able to post some experimental
>> > patches soon enough.
>>
>> Allowing to specify that per-test is a good idea for transitioning test
>> suites.
>>
> About this, in my first two "tentative" patches:
> <http://lists.gnu.org/archive/html/automake-patches/2011-05/msg00093.html>
> I've taken an even more general approach, allowing the developer to define
> and use his own program(s) to:
> 1. launch his test scripts,
> 2. interpreter their results,
> 3. display these results on screen, and
> 4. format and generate the log files.
> All of this is attainable simply by assigning a variable `LOG_WRAPPER'
> (and extensione-specific counterparts of it), and, well, obviously
> providing a real "driver" script that obeys to minimal command-line
> interface (so that it can grasp the options the Automake-generated
> Makefiles passes to it). Then we will hopefully be able to implement
> our TAP/SubUnit parsers on the top of this feature (thus making it
> indirectly more tested, which is always good for a new feature).
If the subunit parser just gets all the output from the test script,
you might want to use the subunit parser itself: folk in the
bootstrap-set couldn't (obviously), but most other environments can
use high level languages freely. (I don't object to more parser
implementations existing, I'm just thinking about reuse where
possible.
>> I hope to look into the posted patches later today.
>>
> About this, please note that I might be AFK until this evening. So have
> no haste.
>
>> > 2. New automake option `tests-protocol'
>> > ---------------------------------------
>> >
>> > The Tap/SubUnit support in the Automake-generated testsuite drivers
>> > should be enabled by a new (argument-requiring) option `tests-protocol',
>> > that will be used to specify the level of support for, detection of, and
>> > enforcing of SubUnit/TAP streams.
>> >
>> > The possible values for `tests-protocol' will be:
>> > - tests-protocol=tap
>> > All test scripts are expected to use the TAP protocol.
>> > - tests-protocol=subunit
>> > All test scripts are expected to use the SubUnit protocol.
>> > - tests-protocol=adaptive
>>
>> The way you describe "adaptive", it sounds like it should rather be
>> named something like "detect" or "detected" or so.
>>
> I'd like to withdraw this proposal now that we can define per-extension test
> protocols. Having our hypotetical "client developer" rename his test scripts
> as they get converted to use TAP/SubUnit is IMO better than we having to
> implement in Automake a probably non-trivial "metaparser" that could end up
> being scarcerly used anyway. WDYT?
>
>> > Each test script is expected to print on its first line of output
>> > which protocol it uses (the exact format of this special line is
>> > still to be determined); if this line is unrecognized, the driver
>> > should assume that the test script uses no protocol. Also, in
>> > this case, we should continue to honour XFAIL_TESTS. All of this
>> > should help to maximize backward-compatibility.
>>
>>
>>
>> > 3. Console output from the test driver
>> > ---------------------------------------
>> >
>> > This output should remain as close as possible to the one already
>> > provided by the current parallel-tests driver. The following example
>> > should help to clarify what I mean.
>> >
>> > Assume we are using `tests-protocol=adaptive', and let TESTS be defined
>> > to "pass.test skip.test subunit.test tap.test". Here, `pass.test' and
>> > `skip.test' are test scripts that use no protocol (and that exit with
>> > status `0' and `77' respectively), `subunit.test' is a test script using
>> > the SubUnit protocol and containing four testcases (one passing, one
>> > failing, one skipped and one which incurs in an expected failure), and
>> > `tap.test' is a test script using the TAP protocol which runs two
>> > successful testcases, then encounters an internal error and bails out
>> > (using the "Bail out!" directive).
>> >
>> > With such a setup, this is the output I'd expect from "make check":
>> > PASS: pass.test
>> > SKIP: skip.test
>> > PASS: subunit.test [testcase name/description]
>> > FAIL: subunit.test [testcase name/ description]
>> > SKIP: subunit.test [testcase name/description] [reason for skipping]
>> > XFAIL: subunit.test [testcase name/description] [failure reason]
>> > PASS: tap.test [testcase name/description]
>> > PASS: tap.test [testcase name/description]
>> > ERROR: tap.test [reason for the bailout]
>> >
>> > Of course, the `color-tests' option should make the above output properly
>> > colorized; the attached html file shows what colors I'd expect.
>>
>> Wrt. to the details of the output, I wouldn't fix things too early.
>> Two things two consider here:
>> - if tests are run in parallel, you want to avoid intermixing output
>> from different tests as much as possible,
>>
> I disagree. Once the test script name is printed in each line which report
> the result of one of its test cases, I see no issue with intermixed lines.
> I.e., I don't believe that this:
>
> PASS: pass.test
> PASS: subunit.test [foo]
> PASS: tap.test [t1]
> SKIP: skip.test
> PASS: tap.test [t2]
> FAIL: subunit.test [bar]
> SKIP: subunit.test [baz] [reason for skipping]
> ERROR: tap.test [reason for the bailout]
> XFAIL: subunit.test [quux] [failure reason]
>
> is any less clear than this:
>
> PASS: pass.test
> SKIP: skip.test
> PASS: subunit.test [testcase name/description]
> FAIL: subunit.test [testcase name/ description]
> SKIP: subunit.test [testcase name/description] [reason for skipping]
> XFAIL: subunit.test [testcase name/description] [failure reason]
> PASS: tap.test [testcase name/description]
> PASS: tap.test [testcase name/description]
> ERROR: tap.test [reason for the bailout] XXX
>
>> but even more than that, you want to avoid intermixing within lines.
>>
> OTOH, I do believe this is a real concern, to be carefully addressed and
> tested for. Thanks for bringing this up.
For Both TAP and subunit the test script running needs to feed into a
single parser: the issue with intermingling then is all about how the
parsers structure their output.
Ideally you would use a threadsafe structure.
For subunit, *if* you were working in python you could use the
existing multithreaded logic testrepository uses, which can mingle
multiple streams correctly, preserving datestamps - a single ui parser
then handles all the combined output.
>> Gathering output and
>> printing it all at once can help here, but the downside is possibly
>> losing output if tests are interrupted, or the driver doesn't finish
>> correctly.
>> - some sort of bypass of the (currently hidden) output is helpful,
>> some gnulib-using packages do that already by redirecting stderr
>> before running the test, and printing a reason for skipping from
>> within the test.
>>
> [thinking about this put on the TODO list]
>
>> I think the basic premise to take into account is that the test driver
>> author (you) may not know all requirements a future test author may
>> have.
>>
> That's why I like my current idea of allowing the client developers
> to provide their own testsuite driver.
>
>> > 4. RST support and HTML generation: should be dropped?
>> > ------------------------------------------------------
>>
>> Good question. You could do a poll on the automake list (in a separate
>> mail & thread, to gain visbility). Or just defer the implementation
>> for later.
>>
>> In another mail:
>> > BTW, I also hope that new interface I'm planning will make it easier to
>> > implement HTML report generation also for TAP and SubUnit tests, with a
>> > consistent reuse of the existing code. In which case my considerations
>> > above will become moot.
>>
>> Cool.
>>
> The downside of this is that it might place additional burdens on the
> writer of test drivers. Hmmm... maybe a new option "no-html-test" or
> similar is warranted?
At least for subunit, and I believe TAP afficiondos will agree for
TAP, you'd generally run subunit2html [made upname] to generate a html
report. An example of such a formatter is subunit2junithtml, and I
believe samba have a perl based html formatter for subunit already
(Jelmer will know more about that).
-Rob
- Re: [GSoC] Early design discussion for TAP/SubUnit support in automake.,
Robert Collins <=