Re: [PATCH v2 7/8] iotests: register each I/O test separately with meson

[Thomas Huth]

On 29/03/2023 13.18, Daniel P. Berrangé wrote:
> On Wed, Mar 29, 2023 at 12:47:51PM +0200, Thomas Huth wrote:
> > On 03/03/2023 17.07, Daniel P. Berrangé wrote:
> > > Currently meson registers a single test that invokes an entire group of
> > > I/O tests, hiding the test granularity from meson. There are various
> > > downsides of doing this
> > >
> > >    * You cannot ask 'meson test' to invoke a single I/O test
> > >    * The meson test timeout can't be applied to the individual
> > >      tests
> > >    * Meson only gets a pass/fail for the overall I/O test group
> > >      not individual tests
> > >    * If a CI job gets killed by the GitLab timeout, we don't
> > >      get visibility into how far through the I/O tests
> > >      execution got.
> > >
> > > This switches meson to perform test discovery by invoking 'check' in
> > > dry-run mode. It then registers one meson test case for each I/O
> > > test. Parallel execution remains disabled since the I/O tests do not
> > > use self contained execution environments and thus conflict with
> > > each other.
> > >
> > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > > ---
> > >    tests/qemu-iotests/meson.build | 35 ++++++++++++++++++++++++++++------
> > >    1 file changed, 29 insertions(+), 6 deletions(-)
> > >
> > > diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
> > > index 323a4acb6a..a162f683ef 100644
> > > --- a/tests/qemu-iotests/meson.build
> > > +++ b/tests/qemu-iotests/meson.build
> > > @@ -32,16 +32,39 @@ foreach k, v : emulators
> > >      endif
> > >    endforeach
> > > +qemu_iotests_check_cmd = files('check')
> > > +
> > >    foreach format, speed: qemu_iotests_formats
> > >      if speed == 'quick'
> > >        suites = 'block'
> > >      else
> > >        suites = ['block-' + speed, speed]
> > >      endif
> > > -  test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), 
> > > format],
> > > -       depends: qemu_iotests_binaries, env: qemu_iotests_env,
> > > -       protocol: 'tap',
> > > -       suite: suites,
> > > -       timeout: 0,
> > > -       is_parallel: false)
> > > +
> > > +  args = ['-tap', '-' + format]
> > > +  if speed == 'quick'
> > > +      args += ['-g', 'auto']
> > > +  endif
> > > +
> > > +  rc = run_command(
> > > +      [qemu_iotests_check_cmd] + args + ['-n'],
> > > +      check: true,
> > > +  )
> > > +
> > > +  foreach item: rc.stdout().strip().split()
> > > +      args = ['-tap', '-' + format, item,
> > > +              '--source-dir', meson.current_source_dir(),
> > > +              '--build-dir', meson.current_build_dir()]
> > > +      # Some individual tests take as long as 45 seconds
> > > +      # Bump the timeout to 3 minutes for some headroom
> > > +      # on slow machines to minimize spurious failures
> > > +      test('io-' + format + '-' + item,
> > > +           qemu_iotests_check_cmd,
> > > +           args: args,
> > > +           depends: qemu_iotests_binaries,
> > > +           env: qemu_iotests_env,
> > > +           protocol: 'tap',
> > > +           timeout: 180,
> > > +           suite: suites)
> > > +  endforeach
> > >    endforeach
> >
> > Seems like this somehow broke compilation on NetBSD:
> >
> > https://gitlab.com/thuth/qemu/-/jobs/4021584713#L2980
>
> I ran it locally and got the meson-log.txt file which reports
>
>    env: python3: No such file or directory
>
> and indeed there is no python3 binary present in our netbsd
> VM.
>
> our tests/vm/netbsd script works around this by passing an
> explicit --python=python3.7 arg to configure, but the way
> we invoke the 'check' script means it is just using the
> "#!/usr/bin/env python3"  logic instead.

Ah, that rings a bell - python scripts must not have the executable flags, 
otherwise meson tries to run them directly instead of using the interpreter 
that has been specified with the --python option.

 Thomas