Re: Documentation for the parallel-tests driver. [4/4]

[Akim Demaille]

> > >  think that you should detail how to write a .test->.log rule,
> > showing the default value for instance (I use GNU Make syntax).
> >
> >  # From a test file to a log file.
> >  # Do not use a regular `.test.log:' rule here, since in that case  
> > the
> >  # following rule (without incoming extension) will mask this one.
> >  %.log: %.test $(check_programs)
> >         @$(am__check_pre) $${dir}$< $(am__check_post)
> >
> > because that's where the user can explain how the test is run.
>
> Erm, the framework in Automake is _not_ supposed to allow the user to
> write such rules herself.  That would likely be problematic, the least
> of which being that it's difficult to make Automake version-agnostic.
>
> Consequently, there is little point in showing users how to write such
> a rule.

I strongly disagree here.  This was one feature that prompted the  
writing of check.mk: very very often for batch-programs you write a  
test driver which takes a test file which contains things like an  
input, expected stdout, stderr, and exit status.  Then testing the  
program is running the test-driver on each test case.  The Automake  
model is a nuisance in that context, it imposes a painful overhead by  
requiring the tests to be executables.

The fact that I was to be able to declare the extension for my test is  
not just for the pleasure not to use .test, but because I want to be  
able to write rules based on the extension.

I can understand that you don't like exposing the way to write new  
rules, but then you really remove something from the model.

Maybe you'd rather have something like (sorry, I don't remember the  
names you chose, but you get the idea)

        TESTS_EXTENSIONS = chk test foo
        chk_TEST_DRIVER = ./check.sh
        foo_TEST_DRIVER = $(top_builddir)/bin/foo

So you don't write the rule, just the name of the "compiler".  It  
looks nicer with "EXTENSIONS" instead of "SUFFIXES", but we can find  
something similar.

They is really, really precious, do not throw that away.


> > Something I dislike very much in the current framework for tests in
> > Automake is that each test must be an executable, so it is very
> > inconvenient to test a given program foo with many different inputs.
> > In that case I use
> >
> >   TESTS = 1.foo 2.foo 3.foo ...
> >
> > and
> >
> >  %.log: %.test $(check_programs)
> >  	@$(am__check_pre) $(top_builddir)/src/foo $${dir}$< $ 
> > (am__check_post)
> >
> > i.e., the testee is foo, and TESTS are plain files, not a bazillion  
> > of
> > 1.sh that simple "exec $(top_builddir)/src/foo 1.foo".
>
> Try
>  TESTS_ENVIRONMENT = $(top_builddir)/src/foo
>
> and adding $(top_builddir)/src/foo to check_SCRIPTS.
>
> If you have only some files that are to be run with foo, and some not,

Exactly.  Besides, it really more looks like a hack than something  
clean.  I want to be able to modify TESTS_ENVIRONMENT on the make  
command line to pass envvars and not having to worry about what may be  
there.

> consider looking at coreutils generalization of this:
> <http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=tests/check.mk;hb=HEAD 
> >.

This is exactly what I don't want to do.  This is a hack to work  
around a limitation in Automake.  It's completely centralizing the  
dispatch instead of simply relying on the normal model in Make:  
suffixes.

I wrote check.mk with a clear and simple model in mind: we are  
compiling tests, and they can be written in several "languages".  This  
fits nicely the Make model and gives all the rest (parallelism etc.)  
for free.  *And* it provides an opportunity to present a uniform  
interface a la Automake, using

        TESTS_SUITES = foo-suite.log bar-suite.log
        foo_suite_SOURCES = 1.foo 2.foo
        bar_suite_SOURCES = a.bar b.bar

        foo_COMPILER = ./foo
        bar_COMPILER = ../bin/bar --verbose --warnings=all

etc.