Re: Migration tests are very slow in the CI

[Thomas Huth]

On 08/08/2022 14.14, Daniel P. Berrangé wrote:
> On Mon, Aug 08, 2022 at 01:57:17PM +0200, Thomas Huth wrote:
> >   Hi!
> >
> > Seems like we're getting more timeouts in the CI pipelines since commit
> > 2649a72555e ("Allow test to run without uffd") enabled the migration tests
> > in more scenarios.
> >
> > For example:
> >
> >   https://gitlab.com/qemu-project/qemu/-/jobs/2821578332#L49
> >
> > You can see that the migration-test ran for more than 20 minutes for each
> > target (x86 and aarch64)! I think that's way too much by default.
>
> Definitely too much.
>
> > I had a check whether there is one subtest taking a lot of time, but it
> > rather seems like each of the migration test is taking 40 to 50 seconds in
> > the CI:
> >
> >   https://gitlab.com/thuth/qemu/-/jobs/2825365836#L44
>
> Normally with CI we expect a constant slowdown factor, eg x2.
>
> I expect with migration though, we're triggering behaviour whereby
> the guest workload is generating dirty pages quicker than we can
> migrate them over localhost. The balance in this can quickly tip
> to create an exponential slowdown.

If I run the aarch64 migration-test on my otherwise idle x86 laptop, it also 
takes already ca. 460 seconds to finish, which is IMHO also already too much 
for a normal "make check" run (without SPEED=slow).

> I'm not sure if  'g_test_slow' gives us enough granularity though, as
> if we enable that, it'll impact the whole test suite, not just
> migration tests.

We could also check for the GITLAB_CI environment variable, just like we 
already do it in some of the avocado-based tests ... but given the fact that 
the migration test is already very slow on my normal x86 laptop, I think I'd 
prefer if we added some checks with g_test_slow() in there ...

Are there any tests in migration-test.c that are rather redundant and could 
be easily skipped in quick mode?

 Thomas