Re: Mauve test question

[Michael Koch]

Am Dienstag, 28. Dezember 2004 20:43 schrieb Archie Cobbs:
> Thomas Zander wrote:
> >>>Huh? Why is adding broken tests the right thing to do? And
> >>> besides, if a broken test is added, this way there will be
> >>> motivation to resolve the discrepancy. With a whitelist, a
> >>> broken test can get added but no one will notice and then it
> >>> just sits there getting stale.
> >>
> >>Its common practise to add new code to one implementation, e.g
> >> GNU classpath or libgcj, and test it for a while and later merge
> >> it to kaffe. According to you the mauve tests don't need to be
> >> added before it's included in all implementations because
> >> nothing may be broken.
> >
> > Ehm; just being a bystander;  a broken test in Archies email is a
> > test that does not work properly (harness.check(1 ==2)).
> > A broken test we are talking about, and what Michael seems to
> > imply; is a test that is fully correct, and will (probably) run
> > correctly on Suns JVM, but fails on another.
> >
> > Lets call the former a broken, and the latter a failing test,
> > please :)
> >
> > Mauve is not suppost to hold _any_ broken tests, right?
>
> Thanks, I was misreading the original comment. We all agree Mauve
> may contain "failing" tests but should never contain "broken"
> tests.
>
> Now, back to the original point... I've made a proposal for
> cleaning up the Mauve mess. For those who don't like it, please
> explain your proposal and most importantly exactly how this
> "whitelist" (or whatver) will be maintained. Who is going to do the
> maintenance work? When are they going to do it? Etc.
>
> I don't really care how we do it, but these seem to be reasonable
> requirements. Maybe we should try to agree on these first:
>
> - It should be possible to test any JVM using some "official" set
> of tests which a Classpath JVM should pass and see clearly and
> obviously any failures; a "perfect" JVM would have no failures.
>
> - When a new test is added to Mauve, by default it is automatically
>    added to the set of Classpath tests. I.e., it's not possible for
>    newly added tests to not actually run against Classpath without
>    explicitly configuring things that way.
>
> The point is that every test has a known state, one of:
>
> (a) don't even bother trying it (doesn't compile or we know it
> fails) (b) it is a known failure (i.e., a Classpath bug)
> (c) it should pass (i.e., if it fails, it must be a JVM bug)
> (d) it should pass but there can be false negatives depending on
> the JVM
>
> and most importantly the maintenance of these states is simle
> enough that we don't lose track and get back into the mess we're in
> now.
>
> By the way, saying "fix ./batch_run to respect xfails" or "fix the
> normal mauve run to not ignore tests that don't compile" is fair
> game (as long as you are willing to do the work :-)

Perhaps both should be done. Then all would be happy.
Note that Thomas Fitzsimmons wanted to work on this too. I don't know 
if he has done already something thin this direction.


Michael
-- 
Homepage: http://www.worldforge.org/