bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case

[Rob Savoye]

On 6/25/20 10:01 AM, Pedro Alves wrote:

>> It could be changed, but those are much larger changes to the code and 
>> overall behavior than simply adding an UNRESOLVED result to report the 
>> failure, and if I understand correctly, the presence of UNRESOLVED results 
>> marks the test results as invalid or partially-valid at best, which is 
>> correct if a test script failed with a Tcl error.

  Adding an additional call makes it harder to diff the .sum files. Not
a huge big deal, but still... UNRESOLVED says a human has to look at
this test case output and decide, the actual error should be in the .log
file. UNTESTED means something went wrong with the test, and the test is
invalid, which is probably what you'd want as the test state for a Tcl
error.

> Currently with DejaGnu only the case of calling an unknown function stops
> the test run AFAIK.  Any other tcl error like calling a proc with the wrong
> number of arguments, or treating a non-array variable as an array, etc. is
> caught by the catch in the runtest proc in runtest.exp, and the testrun
> continues.  That to me clearly indicates that the original intention was
> to catch errors and continue.

  The original intent was to catch some errors, nd ignore others. :-)
Catch is used for errors that can be ignored or worked around. Unknown
was basically to catch programming errors, since as noted, there are
many paths though the code as you change options or environments. They'd
get lost in the output unless it aborted.

> It is just that the "unknown" case wasn't thought of.  So I argue that not
> making unknown proc calls abort the run is a bug fix, and making DejaGnu
> abort the run for other errors by default is a behavior change that no real
> testsuite built around DejaGnu is asking for.

  I think the default should be to abort on a unknown Tcl error, and
have an option to ignore them and continue. As a person that has run
many long test runs, I agree I don't want some errors to abort the test
run. But to be honest, a dependable test run should mean no bugs in the
test framework. A buggy test framework means you can trust the test results.

  Aborting was for use during debugging of test framework code itself,
so supposedly you should should never have an Tcl error during a test
run. There are times when even during debugging you'd want it to
continue, and then collect all the bugs later to fix in batch mode.

        - rob -