xtest vs test

[LachlanA]

Greetings all,

In the context of bug 45507 [http://savannah.gnu.org/bugs/?45507], JWE said
" I don't think anyone likes to see tests that fail, but removing the tests
is definitely not what I'd like to see...

I'm not even sure I like to see xtests because then people just get used to
the idea that the tests fail, that they are expected to fail, and then don't
even look to see why or whether they are problems that should really be
fixed."

I think this issue applies to many more bugs, so I'm replying here.

I think there are four classes of people.
1. "Typical users" who just use the release, and trust that it works.
2. Those who compile a release from source, and use "make check" to see
whether or not to trust the release.
3. Those who compile from dev sources, and report bugs.
4. Developers.

Group 1 doesn't care about xtest vs test.

To my understanding, xtest is to reassure group 2 that we are aware of the
bug, and know that they can still trust Octave to work.

For group 3, we'll reinstate a proper (not "x") test as soon as 4.2.0 is
forked, and missing feedback for a few weeks doesn't matter.

For group 4, we should treat xtest failures as candidates for fixing anyway.

All of this means that I think that all* tests that we know will fail on
relatively common systems, but do not mean that Octave becomes unreliable,
should be xtests for the release (and reverted as soon as the 4.2.0 branch
is forked from default).

* That is an extreme position for argument's sake.  Obviously there will be
many exceptions.

What do others think?

Cheers,
Lachlan



--
View this message in context: 
http://octave.1599824.n4.nabble.com/xtest-vs-test-tp4678131.html
Sent from the Octave - Maintainers mailing list archive at Nabble.com.