Re: Excluding FAILs from UNSUPPORTED test cases

[Arsen Arsenović]

Hi,

On Saturday, 1 October 2022 02:05:35 CEST Jacob Bachmeyer wrote:
> This suggests that early pruning would actually hide bugs in GCC:  if
> the compiler is not supposed to produce the message in question, then
> pruning it causes the test to spuriously pass and the XFAIL result is
> correct.  (I have not yet developed the level of skill to do much more
> on GCC than building and using it, so I am afraid that those tests
> are not particularly meaningful to me.  :-/  )
That sounds about right, and is what these tests do.  I'm not sure what 
a good way to prevent overzealous pruning of messages would even be..  
The only solution I came up with was to use regexes and short circuiting 
(i.e. s/(msgs|to|keep)|prune regexp/\1/), but this seems rather 
inelegant.

> Or did early pruning expose a bug in the GCC testsuite:  the pruning
> function is overzealous and removes messages that GCC should not have
> produced and therefore should have been left for DejaGnu to detect?
DejaGnu would have detected these messages if they happened, which they 
didn't, which is why this test is an XFAIL.  Ultimately, I think that in 
this instance, I just got bit by the compare script treating XPASSes 
same as PASSes, when they should've been considered failures here (i.e. 
the { xfail } tag is added on them because them passing would be a 
problem, since these tests concern themselves with bogus diagnostics).

Thinking about it more, with dg-prune being an "Optional callback to 
delete output from the tool that can occur even in successful ("pass") 
situations and interfere with output pattern matching," it does make 
sense for it to operate on just excess errors.  ${tool}-dg-test can 
already do any pruning that would mess with pattern matching earlier, if 
so desired.  Maybe the best solution to this dilemma is to clarify the 
docs ("interfere with output pattern matching" -> "interfere with excess 
error checks" or something in that spirit) and collecting errors for a 
late dispatch.

Thanks,
-- 
Arsen Arsenović

signature.asc
Description: This is a digitally signed message part.