Re: s390 private runner CI job timing out

[Christian Borntraeger]

Am 06.04.23 um 14:39 schrieb Peter Maydell:
> On Thu, 6 Apr 2023 at 13:30, Thomas Huth <thuth@redhat.com> wrote:
> > The thing is: it shouldn't take that long to build QEMU and run the tests
> > here, theoretically. Some days ago, the job was finishing in 39 minutes:
> >
> >    https://gitlab.com/qemu-project/qemu/-/jobs/3973481571
> >
> > The recent run took 74 minutes:
> >
> >    https://gitlab.com/qemu-project/qemu/-/jobs/4066136770
> >
> > That's almost a factor of two! So there is definitely something strange
> > going on.
>
> So that 39 minute run was about 22 minutes compile, 17 minutes test.
> The 74 minute run was 45 minutes compile, 30 minutes test.
> The number of compile steps in meson was pretty much the same
> (10379 vs 10384) in each case. So the most plausible conclusion
> seems like "the VM mysteriously got slower by nearly a factor of 2",
> given that the slowdown seems to affect the compile and test
> stages about equally.
>
> The VM has been up for 44 days, so we can rule out "rebooted into
> a new kernel with some kind of perf bug".

Looks like the team is looking into some storage delays at the moment.
So it would be good to get some numbers for io wait and steal on the
next run to see if this is related to storage or CPU consumption.
Maybe collect /proc/stat before and after a run. Or login and use top.