|
From: | Jacob Bachmeyer |
Subject: | Re: Excluding FAILs from UNSUPPORTED test cases |
Date: | Sat, 24 Sep 2022 23:24:06 -0500 |
User-agent: | Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 SeaMonkey/1.1.17 Mnenhy/0.7.6.0 |
Arsen Arsenović wrote:
Hi, On Saturday, 24 September 2022 06:24:56 CEST you wrote:This is how I understood the output analysis routine: remove expected lines first (i.e. those matched by dg-messages), then lines that shouldn't be a FAIL but are noise or such instead (i.e. prune), which allows ${tool}-dg-prune and dg-prune-output to be less careful about what they remove (as they don't have to worry about removing diagnostics that should be matched by dg-{error,warning}). Changing ${tool}-dg-prune to execute earlier might be a bit too drastic, because it could expose some overzealous pruning routines.But when it's unsupported because ${tool}-dg-prune returns "::unsupported::" the individual checks within the file still appear in the results. I realise that in the target selector case, the test is just skipped before doing anything, and in the prune case we don't know that it's unsupported until *after* testing it. But it seems to me that ideally the individual checks would get "retroactively skipped" if tool-dg-prune returns one of ::untested::, ::unresolved::, or ::unsupported::. Maybe that's not easy to do though.This actually looks like a bug in dg.exp to me. The output from ${tool}-dg-test is first collected, all at once, (line 696) and then analyzed (lines 701-749). I would expect ${tool}-dg-prune to have an opportunity to modify the text to be analyzed /before/ the analysis is done, but the current code checks for messages first, /then/ calls ${tool}-dg-prune (and the framework's prune_warnings procedure) to remove irrelevant output, then declares excess errors if the final text is nonempty.
Running ${tool}-dg-prune earlier works both ways, allowing the callback prune routine to also remove or modify any output that might confuse the message-matching mechanism. Really, given the documented (such as it is) feature for ${tool}-dg-prune to be able to declare the entire test unsupported, I would expect it to filter the results before messages are checked.
Overzealous pruning routines are bugs in the testsuites that have them, since they would be very likely to hide /actual/ excess errors, thereby causing the test for excess errors to PASS when it should FAIL.
On the other hand, giving ${tool}-dg-test a way to terminate analysis early, before all the output processing, could maybe constrained enough to not change what existing code does (which is why I suggested it earlier).What do you think?
Tcl provides a way to do that (I think) if you really want: unsupported "freestanding libstdc++" return -code return ...but this would be a bit of a hack. -- Jacob
[Prev in Thread] | Current Thread | [Next in Thread] |