Re: [PATCH] tests: Disable migration-test

[Peter Maydell]

On Tue, 21 Feb 2023 at 15:36, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
>
> * Peter Maydell (peter.maydell@linaro.org) wrote:
> > On Tue, 21 Feb 2023 at 15:21, Dr. David Alan Gilbert
> > <dgilbert@redhat.com> wrote:
> > > Damn this is really going to impact the stability of migration if we
> > > don't regularly test.
> > > But fundamentally, I've never been able to debug much of the reports
> > > that come from flakyness in gitlab ci; we're not getting the most basic
> > > information like which subtest or where we're upto in the test which
> > > makes it very very hard to debug.
> >
> > Right, but if you want more information you need to change the
> > tests and/or test harness to provide it.
>
> I don't think the migration test is doing anything odd in that respect;
> We've just got a bunch of qtest tests; having a test framework which
> doesn't tell you which test failed is very difficult.

In the specific case here of the FreeBSD job it does, though
you have to search backwards for 'migration' to find it:
https://gitlab.com/qemu-project/qemu/-/jobs/3806090216

It prints
ok 23 /aarch64/migration/multifd/tcp/plain/none
# starting QEMU: exec ./qemu-system-aarch64 -qtest
unix:/tmp/qtest-32469.sock -qtest-log /dev/null -chardev
socket,path=/tmp/qtest-32469.qmp,id=char0 -mon
chardev=char0,mode=control -display none -accel kvm -accel tcg
-machine virt,gic-version=max -name source,debug-threads=on -m 150M
-serial file:/tmp/migration-test-2A1201/src_serial -cpu max -kernel
/tmp/migration-test-2A1201/bootsect 2>/dev/null -accel qtest
# starting QEMU: exec ./qemu-system-aarch64 -qtest
unix:/tmp/qtest-32469.sock -qtest-log /dev/null -chardev
socket,path=/tmp/qtest-32469.qmp,id=char0 -mon
chardev=char0,mode=control -display none -accel kvm -accel tcg
-machine virt,gic-version=max -name target,debug-threads=on -m 150M
-serial file:/tmp/migration-test-2A1201/dest_serial -incoming defer
-cpu max -kernel /tmp/migration-test-2A1201/bootsect 2>/dev/null
-accel qtest
# starting QEMU: exec ./qemu-system-aarch64 -qtest
unix:/tmp/qtest-32469.sock -qtest-log /dev/null -chardev
socket,path=/tmp/qtest-32469.qmp,id=char0 -mon
chardev=char0,mode=control -display none -accel kvm -accel tcg
-machine virt,gic-version=max -name target,debug-threads=on -m 150M
-serial file:/tmp/migration-test-2A1201/dest_serial -incoming defer
-cpu max -kernel /tmp/migration-test-2A1201/bootsect -accel qtest
Bail out! 
ERROR:../tests/qtest/migration-helpers.c:205:wait_for_migration_status:
assertion failed: (g_test_timer_elapsed() <
MIGRATION_STATUS_WAIT_TIMEOUT)

so it has successfully run tests up to 23, and then
crashed on test 24 (which will be
migration/multifd/tcp/plain/cancel).

The test framework also collects and logs any information
the test case wants to print to stderr -- the reason
there is no information there other than the assertion
message is because that's the only thing the test emits.

The local macos test log I have here also shows it failing
on test 24.

The macos tests are easy to provoke failures here during
'make check', it happens at least 1 time in 2. So if you
want to write a patch to print out more information to
the test log I'm happy to run it and collect the output.

thanks
-- PMM