TAP support in automake (was: Re: automake 1.11.3 check-TESTS and command line length)

[Stefano Lattarini]

Hi Bob.

On 03/31/2012 07:15 PM, Bob Friesenhahn wrote:
> On Wed, 22 Feb 2012, Stefano Lattarini wrote:
>>>
>> Are you referring to the GraphicsMagick testsuite?  If yes, it seems to me
>> that it could benefit greatly from the use of TAP once automake 1.12 is out;
>> for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one
>> single TAP test (and similarly for many other auto-generated tests), thus
>> removing the need for a huge number of files without reducing granularity
>> of the results.
> 
> I took a look at what "TAP" is and see that I have already been using it
> already (not knowing what it was called) for Perl-related tests. The
> fundamental problem with TAP is that documented implementations
> (http://testanything.org/wiki/index.php/TAP_Consumers) require additional
> packages on the system such as Perl, Ruby, Javascript, Java, etc.
> I can not expect users who want to verify their build with 'make check'
> to install an additional language interpreter.
> 
> It seems that (while I was sleeping) Automake 1.12 will be providing support
> for TAP tests 
> (http://lists.gnu.org/archive/html/automake/2011-04/msg00069.html).
> Such support will not be worthwhile if it depends on Perl or some other
> interpreter since this might not be installed on the end-user's machine.
>
Well, automake being automake, it only depends on portable awk[1] ;-)
OK, portable "nawk" actually (as found by the autoconf builtin macro
AC_PROG_AWK), but I don't know of any non-museum machine lacking that.

 [1] Automake also offers an implementation of its TAP consumer that is
     written in perl and based on the TAP::Parser module; but you are
     absolutely not required to use or know it -- at the moment, it is
     not even documented IIRC.

> If it is based on portable C
> (e.g. http://www.eyrie.org/~eagle/software/c-tap-harness/)
> or portable shell script then it will be sufficiently usable.
>
> So what is the story about TAP in Automake 1.12?  Is it time to describe
> the new feature to us mere mortals (i.e. people on address@hidden)
> rather than Automake developers?
>
The new feature is already documented in the automake manual on the
'master' branch; and this documentation should be release-quality
already.  But maybe I should make this new feature more visible before
the 1.12 release?  If yes, how do you suggest to do so?

As for the implementation quality, there's still an annoying glitch in
that you need to *manually* copy the automake-provided 'tap-driver.sh'
script yourself into your package[2] -- "automake --add-missing" won't
work -- and define some variables by hand; all of this is documented
in the manual, and should hopefully be fixed for automake 1.12.1.

 [2] The new automake option option '--print-libdir' should help
     mitigating this annoyance.

Regards,
  Stefano