Re: [GSoC Proposal] automake - Interfacing with a test protocol like TAP or subunit

[Stefano Lattarini]

On Sunday 20 March 2011, Robert Collins wrote:
> On Sun, Mar 20, 2011 at 8:53 PM, Ralf Wildenhues <address@hidden> wrote:
> > Hi Robert,
> >
> > thanks for the feedback.  I have a couple of questions:
> >
> > * Robert Collins wrote on Sun, Mar 20, 2011 at 05:10:16AM CET:
> >> TAP is an extremely simple protocol, and the extensions to it to
> >> support things like not needing to maintain the count of tests,
> >> additional debug data and so on are pretty rudimentary. subunit, which
> >> I've mentioned before was written after TAP, to solve similar problems
> >> and address the issues in TAP itself.
> >
> > Are TAP and subunit compatible on their common subset?  If not, why not?
> 
> You can convert TAP to subunit, and you can convert the things TAP can
> represent from subunit to TAP. subunit's core is more structured than
> TAP, so the two protocols don't pun as each other at all.
>
If I'm not misreading the class TAP2SubUnit in python/subunit/__init__.py,
converting from TAP into subunit shouldn't be much difficult, even with
just the POSIX tools available.  This means that we could add support for
TAP "today", and if "tomorrow" we decide (or our users tell us) to start
supporting also SubUnit, we could add a SubUnit parser, and then continue
to support also TAP by writing a TAP->SubUnit converter and filtering TAP
output into it -- which would allow us to drop the TAP parser (less code
to maintain) without losing backward-compatibility.

> >> Unlike TAP subunit supports attachments (binary and text) to tests,
> >
> > Handling of binary data may end up being quite tricky within a
> > restricted Posix environment with only a few tools available.
> > What if a consumer cannot handle them?  Is there possibility
> > for graceful fallback?
> 
> I didn't think posix had 7 bit limits? Anyhow, there is a baseline
> profile which assumes just a single description of the error in a test
> - it uses \r] to delimit a traceback. Alternatively, a C parser - on
> my 'sometime' TODO list - will probably clock in small enough to
> bundle for projects very low in the dependency stack.
>
But unfortunately, for projects that do not ship C code, we cannot
make the assumption that a C compiler exists on the taget system :-(

You might say "Well, if the user lacks even a C 89 compiler, he
won't able to run the tests, period"; and I'd agree with you, but
unfortunately AC_PROG_CC and relatives cause the configure script
to abort if no working C compiler can be found :-(
And trying to force the definiton of $CC to some dummy value (like
":" or "no") won't work either.

> I think a reasonable approach would be:
>  - have a tiny minimal 'list test and report' consumer which can be
>    bundled for the bootstrap closure of projects.
>  - detect (via autoconf) the presence of existing subunit consumers
>    and preferentially use those.
>
Not completely sure about this.  My hope is that our TAP/SubUnit consumer
will integrate well with:
  1. the parallel-tests feature, and
  2. test that are shell scripts run with the `xtrace' flag on;
which means that output (on either stdout/stderr) that is not recognized
by the TAP/SubUnit protocol must go to the log files, not to the user
terminal.

> However, even this may not be needed- most bring-up situations involve
> cross compiling these days, so just depending on the presence of an
> executable subunit-stats (for instance) may be sufficient : cross
> compile a set of C based subunit consumers, done.
> 
> >> test naming, tagging, timestamping (permitting robust timing data even
> >> in parallelised or distributed testing).
> >>
> >> Integrating with TAP is basically uninteresting to anyone working in a
> >> high level language: Python, Ruby,Java, C++ etc.
> >
> > Well, Perl is a high-level language too, I'd say.  ;-)
> 
> It is and it isn't - it has a very-close-to-the-metal feel to it. perl
> 6 may change this more dramatically :).
> 
> -Rob
> 

Regards,
   Stefano