bug#67392: dejagnu too noisy by default

[Jacob Bachmeyer]

Tom Tromey wrote:
> I think runtest prints too much information to stdout by default.
>
> Consider the appended output.  This comes from running a single gdb
> test.
>
> A little bit of this is gdb-specific (the "HIP" override line).  But
> most of it is not.
>   

What is "HIP"?

I have to warn you, again, that monkey-patching the framework like that 
is not supported and is likely to break in a future release, probably 
around the time that native parallel testing is introduced but possibly 
before then.  (You will, of course, be able to define a procedure named 
"default_target_compile" in the tool-init context but it may not be the 
"default_target_compile" that the framework calls anymore.)  Also, 
default_target_compile is slated to be entirely rewritten before the 
next release, so whatever matching you are using may also break.  
Further, default_target_compile is not intended as a customization 
point, but rather as a default for targets that do not have their own 
${target}_compile procedures defined in the site configuration.

The documented API call is target_compile, which only calls 
default_target_compile if ${target}_compile does not exist.  
Monkeypatching default_target_compile will break at any site that 
actually has target-specific compile procedures.

There are plans to provide for extensibility; current Git master has 
most of a spec strings module in lib/specs.exp that is planned to be 
used as part of a table-driven new default_target_compile; an API for 
testsuites to insert additional values will be provided.  The tentative 
plan is to add categories to the layers mechanism:  site, local, 
builtin, and fallback.  The "site" category is reserved for 
site-specific configuration and, since the site configuration files can 
determine what testsuite is about to run, is the highest priority.  The 
"local" category allows testsuites to override the standard behavior.  
The "builtin" category will contain the information currently hardwired 
into default_target_compile.  Lastly, the "fallback" category exists 
explicitly to allow testsuites to extend DejaGnu as GDB did with Go and 
Rust, but without causing upstream changes to mysteriously vanish when 
testing GDB.

> Most output is not really useful.  In fact, pretty much the only lines I
> might normally be interested in are the "Running [...]" line and the
> result summary, so that if I "make -j8 check" I can easily see which
> tests are being run.  (FWIW for -j8 even the summary isn't interesting
> because gdb's Makefile has to collate the results at the end anyway.)
>   

The future native parallel testing support will take care of that.  As 
long as the GDB Makefiles do not pass the relevant option (probably also 
-j) to runtest, the parallel testing infrastructure will not be used.

> So, I think runtest should print a lot less text by default.  Emitting
> it to the .log file is sufficient (although some could even be trimmed
> from there).
>   

The largish block makes finding a previous run easier when rolling 
through terminal scrollback.  It visually stands out from the "Running 
..." lines very eye-catchingly.

> If you don't like that, though, how about a command line option to make
> it be more quiet?
>   

The question here is what should such an option actually do?  Which 
messages should be suppressed?

> prentzel. runtest gdb.cp/local-static.exp
> WARNING: Couldn't find the global config file.
> Using 
> /home/tromey/gdb/build/gdb/testsuite/../../../binutils-gdb/gdb/testsuite/lib/gdb.exp
>  as tool init file.
> NOTE: Dejagnu's default_target_compile is missing support for HIP, using local 
> override
> Test run by tromey on Wed Nov 22 16:53:00 2023
> Native configuration is x86_64-pc-linux-gnu
>
>                 === gdb tests ===
>
> Schedule of variations:
>     unix
>
> Running target unix
> Using /usr/share/dejagnu/baseboards/unix.exp as board description file for 
> target.
> Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
> Using 
> /home/tromey/gdb/build/gdb/testsuite/../../../binutils-gdb/gdb/testsuite/config/unix.exp
>  as tool-and-target-specific interface file.
> Running 
> /home/tromey/gdb/build/gdb/testsuite/../../../binutils-gdb/gdb/testsuite/gdb.cp/local-static.exp
>  ...
>
>                 === gdb Summary ===
>
> # of expected passes            407
> /home/tromey/gdb/build/gdb/gdb version  15.0.50.20231121-git -nw -nx -q -iex "set height 0" -iex "set width 0" -data-directory /home/tromey/gdb/build/gdb/data-directory 
>   

DejaGnu is specifically intended to support complex testing 
environments, so please consider a user with multiple test targets, of 
different architectures, possibly connected using different interfaces, 
all managed from one workstation.  That said, perhaps some different 
defaults may be appropriate in the narrow case of running one specific 
test script, locally.

While writing this, I had an idea for a possible new feature:  a 
"oneshot" testing mode, perhaps available with a --oneshot option, that 
will set up the testsuite and run exactly one script, with minimal 
output, perhaps as little as the summary lines and ${tool}_version 
output.  Implementing this right now would require extensive changes to 
runtest, so it will sit on my TODO list until I find an elegant way to 
do it, but the goal would be something like:

8<------

prentzel. runtest --oneshot gdb.cp/local-static.exp
# of expected passes            407
/home/tromey/gdb/build/gdb/gdb version  15.0.50.20231121-git -nw -nx -q -iex "set height 
0" -iex "set width 0" -data-directory /home/tromey/gdb/build/gdb/data-directory

8<------

The intended use of --oneshot is for rapid testing while developing a 
feature or chasing a bug, so logs would probably also be different in 
oneshot mode but this requires more thought.  I presume that your 
site.exp is setting tool to "gdb"?


-- Jacob