Re: [PATCH] tests/qtest/migration-test: Disable migration/multifd/tcp/plain/cancel

[Peter Maydell]

On Wed, 22 Mar 2023 at 20:15, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Tue, 7 Mar 2023 at 09:53, Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > On Sat, 4 Mar 2023 at 15:39, Peter Maydell <peter.maydell@linaro.org> wrote:
> > >
> > > On Thu, 2 Mar 2023 at 17:22, Peter Maydell <peter.maydell@linaro.org> 
> > > wrote:
> > > >
> > > > migration-test has been flaky for a long time, both in CI and
> > > > otherwise:
> > > >
> > >
> > >
> > > > In the cases where I've looked at the underlying log, this seems to
> > > > be in the migration/multifd/tcp/plain/cancel subtest.  Disable that
> > > > specific subtest by default until somebody can track down the
> > > > underlying cause. Enthusiasts can opt back in by setting
> > > > QEMU_TEST_FLAKY_TESTS=1 in their environment.
> > >
> > > So I'm going to apply this, because hopefully it will improve
> > > the reliability a bit, but it's clearly not all of the
> > > issues with migration-test, because in the course of the
> > > run I was doing to test it before applying it I got this
> > > error from the OpenBSD VM:
> > >
> > >  32/646 qemu:qtest+qtest-aarch64 / qtest-aarch64/migration-test
> > >            ERROR          134.73s   killed by signal 6 SIGABRT
> > > ――――――――――――――――――――――――――――――――――――― ✀  
> > > ―――――――――――――――――――――――――――――――――――――
> > > stderr:
> > > qemu-system-aarch64: multifd_send_sync_main: channel 15 has already quit
> > > qemu-system-aarch64: failed to save SaveStateEntry with id(name): 2(ram): 
> > > -1
> > > qemu-system-aarch64: Failed to connect to '127.0.0.1:19581': Address
> > > already in use
> > > query-migrate shows failed migration: Failed to connect to
> > > '127.0.0.1:19581': Address already in use
> > > **
> > > ERROR:../src/tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > > assertion failed: (!g_str_equal(status, "failed"))
> > >
> > > (test program exited with status code -6)
> >
> > Got another repeat of this one today; again, on the OpenBSD VM:
> >
> >  32/646 qemu:qtest+qtest-aarch64 / qtest-aarch64/migration-test
> >            ERROR
> >   131.28s   killed by signal 6 SIGABRT
> > ――――――――――――――――――――――――――――――――――――― ✀  
> > ―――――――――――――――――――――――――――――――――――――
> > stderr:
> > qemu-system-aarch64: multifd_send_sync_main: channel 15 has already quit
> > qemu-system-aarch64: failed to save SaveStateEntry with id(name): 2(ram): -1
> > qemu-system-aarch64: Failed to connect to '127.0.0.1:30312': Address
> > already in use
> > query-migrate shows failed migration: Failed to connect to
> > '127.0.0.1:30312': Address already i
> > n use
> > **
> > ERROR:../src/tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > assertion failed: (!
> > g_str_equal(status, "failed"))
> >
> > (test program exited with status code -6)
> > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
>
> This one's still here (openbsd VM again):
>
>  37/774 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test
>            ERROR          565.42s   killed by signal 6 SIGABRT
> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> qemu-system-x86_64: multifd_send_sync_main: channel 15 has already quit
> qemu-system-x86_64: Failed to connect to '127.0.0.1:7856': Address
> already in use
> query-migrate shows failed migration: Failed to connect to
> '127.0.0.1:7856': Address already in use
> **
> ERROR:../src/tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> assertion failed: (!g_str_equal(status, "failed"))
>
> (test program exited with status code -6)

And again, here on x86 macos:

 32/626 qemu:qtest+qtest-aarch64 / qtest-aarch64/migration-test
           ERROR          413.00s   killed by signal 6 SIGABRT
――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stderr:
qemu-system-aarch64: multifd_send_sync_main: channel 14 has already quit
qemu-system-aarch64: Failed to connect to '127.0.0.1:52689':
Connection reset by peer
query-migrate shows failed migration: Failed to connect to
'127.0.0.1:52689': Connection reset by peer
**
ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
assertion failed: (!g_str_equal(status, "failed"))

(test program exited with status code -6)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――


If this isn't sufficient output to be able to figure out the
problem, can we add more diagnostics to the test and/or the
migration code so that the test does produce helpful output?

thanks
-- PMM