[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] Slow qemu guest performance after host is resumed from
From: |
Ruslan Habalov |
Subject: |
Re: [Qemu-devel] Slow qemu guest performance after host is resumed from suspended state |
Date: |
Wed, 7 Dec 2016 21:53:15 +0100 |
Following Stefan's suggestion I've investigated the possibility of
potential problems with interrupts.
Unfortunately, I can see no differences between both scenarios regarding
the interrupt counter growth (c.f.
https://evonide.com/images/qemu_proc_interrupts.png).
After a short conversation with him I think we can exclude this as a
problem source.
Please let me know if you have any further advice how I should proceed
further in order to triage this bug.
On Wed, Dec 7, 2016 at 7:49 PM, Stefan Hajnoczi <address@hidden> wrote:
> On Sun, Dec 4, 2016 at 10:52 PM, Ruslan Habalov <address@hidden> wrote:
> > Hi everyone,
> >
> > I'm having the following qemu issue since about February on Xubuntu.
> >
> > Steps to reproduce:
> > 1) Boot host and boot qemu guest - everything works perfect
> > 2) Shutdown qemu guest
> > 3) Suspend host and resume host
> > 4) Boot qemu guest again - the guest is now extremely slow
> >
> > Current system setup:
> > host: kernel 4.8.0-27-generic (also reproducible with older kernel
> versions)
> > qemu guest: Linux or Windows affected (tested both)
> >
> > I've tried to debug the issue on my own with the following results:
> > ------------------------------------------------------------
> ------------------------------------------------------------------
> > https://www.evonide.com/images/qemu_problem.png
> > - This shows "perf top $QEMU_PID" from guest boot to shutdown for three
> > different scenarios:
> > Left: normal guest behavior after fresh boot of host system
> > Middle: very slow guest after host was suspended and resumed
> > Right: test run with disabled HPET (since I initially assumed some
> > problems with HPET)
> >
> > https://www.evonide.com/images/qemu_hpet_investigation.png
> > - Results of "perf record -g" and "perf report"
> > According to stefanha (qemu IRC) this doesn't seem to indicate any
> error
> > sources.
> >
> > https://www.evonide.com/images/qemu_vm_exit_healthy_vs_unhealthy.png
> > - Results of running: "perf record -a -e kvm:kvm_exit" and "perf
> > report" from guest boot to shutdown
> > - Left: normal situation (after fresh host boot)
> > - Right: bad situation (after suspend and resume on host and boot of
> > qemu guest)
> > There were ~300 out of order events reported by perf for the "bad"
> > scenario (vs. only 2 for the "good" scenario).
> > ------------------------------------------------------------
> ------------------------------------------------------------------
> > I was also recommended to check the host timers (e.g. /proc/timer_list).
> > However, this file has more than 800 lines and I am unsure what to look
> for.
> >
> > Please let me know if you need further information on this issue.
>
> Unfortunately I don't see an obvious pattern in the kvm_exit reasons.
> Maybe someone on address@hidden can help, I have CCed the list.
>
> One idea that might explain the read_hpet trace you collected is an
> interrupt storm. That means an interrupt is firing all the time
> without doing useful work. This could explain why the vcpu keeps
> halting (leading to many ktime_get() calls on the host).
>
> You can check the interrupt storm theory by collecting
> /proc/interrupts on the host and inside the guest. If you compare the
> healthy system against the slowed system there might be an obvious
> difference in the interrupt counters. That would allow you to track
> down which device keeps firing an interrupt.
>
> Stefan
>