Re: on naming test files for parallel test harness and not removing extensions

[Stefano Lattarini]

On 02/07/2013 11:01 AM, Marco Maggi wrote:
> Ciao,
>
Ciao Marco, sorry for the delay.

>   this is somewhat a silly request... :-) I am a new user of
> the parallel  test harness; I have  a project with a  set of
> tests that I want to  run under different implementations of
> the same language; I put all the tests in a library and then
> load the  library from test  programs, one program  for each
> language implementation[1].
> 
>   Due to the way Automake names  the .log and .trs files (by
> removing the  selected file extensions from  program files),
> for the language implementations Guile  and Vicare I have to
> name the test programs:
> 
>    test-sofa-guile.guile
>    test-sofa-vicare.vicare
> 
> rather than just:
> 
>    test-sofa.guile
>    test-sofa.vicare
> 
> else I get .log and .trs  file names conflict; this is a bit
> ugly.  Unless I missed it, there  is no way to overcome this
> uglyness with Automake up to version 1.13.
>
A possible workaround would be to place the guile and vicare
tests in two separate subdirectories, so that there is no conflict
for the .log files:

   vicare-tests/test-sofa.vicare
   guile-tests/test-sofa.guile

The parallel tests harness will generate the following log files
from the tests above:

    vicare-tests/test-sofa.log
    guile-tests/test-sofa.log

so no conflict.

And note that you don't need to recurse in the two subdirectories
to obtain that effect; you can just have:

   TESTS = \
     vicare-tests/test-sofa.vicare \
     guile-tests/test-sofa.guile

in you top-level Makefile (in fact, I'd advise against extra make
recursion whenever possible).

Of course, this makes sense only if you have several tests for
each of guile and vicare; if you only have a couple of test each,
this approach would just bring to a pointless proliferation of
basically dummy directories ...

>  Other  users may  be in  the same  situation (for  example
> Pythonistas  also  have  to   deal  with  multiple  language
> implementations).   So  I  wonder   if  it  is  possible  to
> introduce an option that disables building the names of .log
> and .trs  files by first stripping  the selected extensions;
> so program files like:
> 
>    test-sofa.guile
>    test-sofa.vicare
> 
> would generate:
> 
>    test-sofa.guile.log
>    test-sofa.guile.trs
>    test-sofa.vicare.log
>    test-sofa.vicare.trs
> 
> with no conflict.
>
You can already do this by writing a custom test runner that take a script
and runs it with the correct interpreter (guile or vicare, in your case),
depending on its extensions; then you can have this in your Makefile.am:

  TEST_EXTENSIONS = # empty
  LOG_COMPILER = my-test-runner
  TESTS = test-sofa.guile test-sofa.vicare

and the generated .log and .trs files should keep the extension of the
original test file.  The downside of this is that tit will require you
to write more non-trivial code.

All in all, I'm not truly convinced your use case is general/relevant
enough to warrant a new automake option; my opinion is that:

  - if you have only few tests per interpreter, the workaround you are
    currently using (of having the interpreter name also as part of the
    test basename) is not too cumbersome, and works pretty well;

  - if you have several tests per interpreter, you can go with my
    proposed workaround of "one subdirectory per interpreter".

> TIA
> 
> [1] From line 224 onwards:
>     <http://github.com/marcomaggi/r6rs-sofa/blob/master/tests/Makefile.am>

HTH,
  Stefano