Re: Test suite regressions vs expected failures

[Mike Miller]

On Tue, Aug 22, 2017 at 11:15:34 -0400, John W. Eaton wrote:
> If we close the report as "won't fix", then should we have tests that fail
> forever?  Or should we have another way to flag them in the tests so that
> they are reported as "issues we know about but don't plan to fix"?  That was
> the question I was originally trying to ask.

Yeah, or in this case, a bug that fixed some things and allowed other
things to remain unfixed.

I think we should be consistent with the markings, so in theory the
source tree can be scraped for bug numbers, the bug tracker can be
scraped for the corresponding reports, and they will all be either open
(no '*' marking) or closed as fixed (with '*' marking).

We could add another marking like '!' to indicate a bug that shows some
behavior that we are not going to fix, but are intentionally adding a
test anyway to show that we are not compatible.

But since the test will always fail, what does that show? Should the
test suite report a problem if a test marked "won't fix" actually passes
instead?

-- 
mike

signature.asc
Description: PGP signature