Re: [PATCH DejaGNU/GCC 0/1] Support per-test execution timeout factor

[Maciej W. Rozycki]

On Wed, 3 Jan 2024, Hans-Peter Nilsson wrote:

> > > Hmm.  I think it would be more correct to emphasize that the 
> > > existing dg-timeout-factor affects both the tool execution *and* 
> > > the test execution, whereas your new dg-test-timeout-factor only 
> > > affects the test execution.  (And still measured on the host.)
> > 
> >  Not really, `dg-timeout-factor' is only applied to tool execution and it 
> > doesn't affect test execution.
> 
> Let's stop here: that statement is just incorrect.
> 
> There might be a use for separating timeouts, but the premise of 
> dg-timeout-factor not affecting the execution of an (executable) 
> test is plain wrong.  Something is off here.  Are we using the 
> same terminology?

 So I found some time finally and did a little bit of investigation and I 
think I can see what's been going on here.

 The thing is the timeouts are treated differently depending on whether
tests are run locally or on a remote target board.  Then I've been working 
with remote boards most of the time and obviously wasn't vigilant enough 
on this occasion to notice the different semantics.

 This boils down to how the handling of `dg-timeout-factor' has been 
done in gcc/testsuite/lib/timeout.exp, by overriding `standard_wait', 
which however is only used by DejaGNU for commands that are run locally.

 So taking gcc/testsuite/gcc.c-torture/execute/20000112-1.c as an example 
and a boring remote unix target board with the following settings (among 
others; leaving out compilation options as irrelevant):

load_generic_config "unix"
set_board_info rsh_prog ssh
set_board_info rcp_prog scp
set_board_info username macro
set_board_info hostname www.xxx.yyy.zzz
set_board_info timeout 500
set_board_info gcc,timeout 700

(say this is foo.exp) I run:

$ make -C obj/gcc RUNTESTFLAGS="--target_board foo execute.exp=20000112-1.c" -k 
-i check-gcc-c

and get these timeouts (among others) reported in gcc.log:

Executing on host: .../obj/gcc/gcc/xgcc -B.../obj/gcc/gcc/ 
.../src/gcc/gcc/testsuite/gcc.c-torture/execute/20000112-1.c    
-fdiagnostics-plain-output -O0  -w -lm  -o ./20000112-1.exe    (timeout = 700)
Executing on foo: /tmp/runtest.35135/20000112-1.exe    (timeout = 300)

-- as you can see the general board timeout (at 500) is ignored, the GCC 
execution timeout is correctly set (to 700), and the test execution 
timeout is taken from the default (at 300) buried and hardcoded in 
`remote_exec' in the absence of an override supplied by `unix_load' 
calling it.

 Now with 20000112-1.c modified to include:

/* { dg-timeout-factor 5 } */

and the same testsuite invocation I get these timeouts reported instead:

Executing on host: .../obj/gcc/gcc/xgcc -B.../obj/gcc/gcc/ 
.../src/gcc/gcc/testsuite/gcc.c-torture/execute/20000112-1.c    
-fdiagnostics-plain-output -O0  -w -lm  -o ./20000112-1.exe    (timeout = 3500)
Executing on foo: /tmp/runtest.35836/20000112-1.exe    (timeout = 300)

-- so the GCC execution timeout is correctly multiplied by 5 (to 3500), 
however the test execution timeout is unchanged (at 300).

 Then with a simulator board, such as one using these settings:

set_board_info slow_simulator 0
load_generic_config "sim"
set_board_info sim "qemu-riscv64 -cpu rv64"
set_board_info timeout 500
set_board_info gcc,timeout 700
set_board_info sim_time_limit 900

I can see by patching `standard_wait' that the test execution timeout does 
get multiplied by `dg-timeout-factor':

spawn qemu-riscv64 -cpu rv64 ./20000112-1.exe
Running standard_wait on bar (timeout = 3500)

(I've double-checked with the same patch that `standard_wait' indeed does 
not get called in the remote case).  So it seems like we're both right in 
the relevant areas, and the mess is perhaps even worse.

 I think Jacob has had a valid concern about encapsulation and we can 
start from there.  Indeed the same remote execution timeout applies to 
maintenance commands used to prepare a remote executable beforehand and 
then delete it afterwards (which I don't normally make use of in my test 
environments though, so I had to realise that first and then make explicit 
changes to my setup to obtain these entries), more specifically:

PASS: gcc.c-torture/execute/20000112-1.c   -O0  (test for excess errors)
Executing on foo: mkdir -p /tmp/runtest.36823   (timeout = 300)
spawn [open ...]
XYZ0ZYX
Executing on foo: chmod +x /tmp/runtest.36823/20000112-1.exe    (timeout = 300)
spawn [open ...]
XYZ0ZYX
Executing on foo: /tmp/runtest.36823/20000112-1.exe    (timeout = 300)
spawn [open ...]
XYZ0ZYX
Executing on foo: rm -f  /tmp/runtest.36823/20000112-1.exe.o 
/tmp/runtest.36823/20000112-1.exe    (timeout = 300)
spawn [open ...]
XYZ0ZYX
PASS: gcc.c-torture/execute/20000112-1.c   -O0  execution test

so indeed more changes are required to get this mess sorted properly, as 
all these commands except from the one to run the test case itself can be 
assumed to take a nominal amount of time to complete.  I'll see if I can 
read through the proposals posted and come up with any conclusions.

 In any case thank you for bringing my attention to the flaw in my 
observations.

  Maciej