Re: Parallel tests execution [0/4]

[Ralf Wildenhues]

* Akim Demaille wrote on Fri, Oct 17, 2008 at 11:12:45AM CEST:
> Le 16 oct. 08 à 22:02, Ralf Wildenhues a écrit :

>>> Actually I never meant to have hard error stop the whole test suite.
>>> The point of hard-errors as they were defined in check.mk was to make
>>> them *non* ignorable.  For instance our test suite raises a hard- 
>>> error if the program make a segmentation fault, which we never want
>>> to tolerate.
>>
>> I don't understand.  What is the difference to a normail FAIL then,
>> i.e., to the process exiting with 1?
>
> By non-ignorable, I meant cannot be XFAILed.  It was meant for error  
> that should never be accepted.

> At some point we were developping more-or-less test driven: many tests  
> were written way before it was possible to pass them, so we knew that  
> the tests would fail and they were XFAIL'ed.  Yet, it was not acceptable 
> for the executable being tested to die with a SEGV.  So simple failures 
> are just exit 1, and hard-errors (or 'unacceptable errors') had an exit 
> status that could not be saved by XFAIL.

Hmm.  Ah, ok.

So the desired semantics are: we have a test that is marked as XFAIL
currently, but if it does this and that, then that should be a FAIL
because it is worse than the expected failure.

Yes, that sounds interesting.  And it is different from my proposed
semantics:

We have a failure that should prevent further test from even being
tried.


Now, the implementation of my proposed semantics /almost/ allows to have
your proposed semantics, too: you can run
  make -k check

and the driver will run all tests, even after hard failures.  In the
presence of hard failures, however, it will fail to produce a summary
and the test-suite.log file.  Could be a bit problematic when output
is very long.

I'm not sure whether I want to introduce both semantics.  I see yours as
reasonable, I'm not sure how applicable mine would be in practice.
Opinions appreciated.

[ Thanks section ]
> Well, it was developped by me for Vaucanson, and we usually hide our  
> names behind "The Vaucanson Group".  That was part of my work at EPITA.  
> So I would completely drop the Vaucanson group from there.  How about 
> this?
>
>>  ## This code is adapted from check.mk which was originally
>>  ## written at EPITA/LRDE, further developer at Gostai, then made
>>  ## its way from GNU coreutils to end up, largely rewritten,
>>  ## in Automake.

Thanks, that's good.

Cheers,
Ralf