Re: Parallel tests execution [0/4]

[Ralf Wildenhues]

Hi Akim,

slowly making progress.

* Akim Demaille wrote on Mon, Oct 20, 2008 at 12:02:21PM CEST:
> Le 18 oct. 08 à 07:37, Ralf Wildenhues a écrit :
>> * Akim Demaille wrote on Fri, Oct 17, 2008 at 11:12:45AM CEST:
>>> At some point we were developping more-or-less test driven: many  
>>> tests were written way before it was possible to pass them, so we
>>> knew that the tests would fail and they were XFAIL'ed.  Yet, it was
>>> not  acceptable for the executable being tested to die with a SEGV.
>>> So simple  failures are just exit 1, and hard-errors (or
>>> 'unacceptable errors') had an  exit status that could not be saved
>>> by XFAIL.

>> So the desired semantics are: we have a test that is marked as XFAIL
>> currently, but if it does this and that, then that should be a FAIL
>> because it is worse than the expected failure.
>
> Correct.

>> Yes, that sounds interesting.  And it is different from my proposed
>> semantics:
>>
>> We have a failure that should prevent further test from even being
>> tried.
>
> Yes, I have understood that.  And it sounds like an interesting feature.

>> Now, the implementation of my proposed semantics /almost/ allows to  
>> have your proposed semantics, too: you can run
>>  make -k check
>
> That's really different: you now require the user, who runs the test  
> suite, to change the way he should run it.

Good point.  Shouldn't do that.

>> I'm not sure whether I want to introduce both semantics.  I see yours
>> as reasonable, I'm not sure how applicable mine would be in practice.
>> Opinions appreciated.
>
> Your semantics can be addressed via dependencies, don't it?

No, but dependencies make the feature more useful. The way my proposed
patches described dependencies between tests, they describe only order
relations.  So if
  bar.log: foo.log

then bar.test will be run even if foo.test FAILed.  However, if foo.test
failed HARD-in-my-sense-of-hard, then bar.test won't be run with such a
dependency (but only because no further test will be run).

Makes sense?

I'm wondering whether I should drop my idea of hard errors for now.
It can be emulated by checking some conditions in the test driver,
I guess, whereas yours can't easily.

> In my test suites I do have the case of tests that cannot succeed under 
> some conditions, but n that case they all individually exit SKIP.  That 
> way, even if you run just some of the tests (via explicit make check 
> TESTS=...), they will be ignored, even if you skipped some of the HARD 
> error (in your sense) tests.

Cheers,
Ralf