[Qemu-discuss] QEMU/KVM Bug: Sporadic freezing in Windows 2012R2

[Matthew Anderson]

Hi All,

I've run into a bug that I can't seem to solve and need some advice on
what to do next. Environment is -
Ubuntu 14.04 (Kernel 3.13-35)
Qemu 2.1.1
Dual Ivy Bridge-EP E5-2697 v2
256GB

The problem I'm having is that a reasonably large guest (48gb, 12cpu)
running Server 2012R2 standard will hang for 2-5 seconds under load
(~60% CPU and even under 10%) every 30 to 90 minutes. Any monitoring
on the guest shows that CPU usage jumps to 100% during the freeze and
the Windows perfmon tool shows a gap in CPU usage reporting indicating
that it's not doing anything at all during the freeze and skipping
interupts. During the freeze the guest replys to pings very slowly at
>1000ms.

Current command line is -

qemu-system-x86_64 -enable-kvm -name ServerWithProblem -S -machine
pc-q35-2.1,accel=kvm,usb=off -cpu qemu64,-svm -m 49152 -realtime
mlock=off -smp 12,sockets=12,cores=1,threads=1 -uuid
6b5cecbf-385e-43a1-b894-99b72efcae8b -no-user-config -nodefaults
-chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/ServerWithProblem.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -no-hpet -no-shutdown -boot
order=c,menu=on,strict=on -device
i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device
pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -drive
file=rbd:ssd/ServerWithProblem:auth_supported=none,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=threads
-device 
virtio-blk-pci,ioeventfd=off,event_idx=off,scsi=off,bus=pci.2,addr=0x2,drive=drive-virtio-disk0,id=virtio-disk0
-drive 
file=/var/lib/libvirt/images/vio.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=25,id=hostnet0 -device
e1000,netdev=hostnet0,id=net0,mac=52:54:00:21:54:7f,bus=pci.2,addr=0x1
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:4 -device
cirrus-vga,id=video0,bus=pcie.0,addr=0x1 -device
virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x3

What I've tried so far is -
Different QEMU version (2.0-ubuntu, 2.0 source, 2.1, 2.1.1)
Different kernels (3.13-ubuntu, 3.15.8, 3.16.2)
Different machine type (Q35 vs i440)
Enable and disable Hyper-v enlightenments
Change guest CPU type (QEMU64 vs Sandy Bridge vs Host)
Disable APICV usage in the kvm_intel module
Enable/Disable KSM and numa balancing

All of that and so far it's still occurring no matter what. Disabling
APICV in the kvm module appears to have made the freezes a little
shorter (~2 seconds) but I can't say that with absolute certainty.
I've tried to replicate the problem on another host with Nehalem CPU's
but haven't seen the same issue. I wrote a small script to ping the
guest and trigger a perf record for 5 seconds during the freeze and
got the below details -

Kernel 3.13-25, Qemu 2.1.1, guest has all HV enlightenments enabled,
kvm_intel with apicv disabled

# Overhead          Command       Shared Object
                  Symbol
# ........  ...............  ..................
..............................................
#
    14.93%  qemu-system-x86  [kernel.kallsyms]   [k]
native_write_msr_safe
    13.44%  qemu-system-x86  [kernel.kallsyms]   [k] vmx_vcpu_run
    13.13%  qemu-system-x86  [kernel.kallsyms]   [k] fget_light
    11.73%  qemu-system-x86  [kernel.kallsyms]   [k] x86_decode_insn
     9.80%  qemu-system-x86  [kernel.kallsyms]   [k] vmx_vcpu_load
     9.71%  qemu-system-x86  [kernel.kallsyms]   [k] mmu_set_spte
     6.19%  qemu-system-x86  [kernel.kallsyms]   [k] update_cfs_shares
     5.19%  qemu-system-x86  [kernel.kallsyms]   [k]
x86_emulate_instruction
     4.68%  qemu-system-x86  [kernel.kallsyms]   [k]
_raw_spin_lock_irqsave
     3.26%  qemu-system-x86  [kernel.kallsyms]   [k] enqueue_entity
     3.00%  qemu-system-x86  [kernel.kallsyms]   [k] pte_list_add
     2.63%  qemu-system-x86  qemu-system-x86_64  [.]
0x00000000000de670
     0.71%  qemu-system-x86  [kernel.kallsyms]   [k] rcu_irq_exit
     0.66%  qemu-system-x86  [kernel.kallsyms]   [k]
generic_smp_call_function_single_interrupt
     0.64%  qemu-system-x86  [kernel.kallsyms]   [k] remote_function
     0.28%  qemu-system-x86  [kernel.kallsyms]   [k] kvm_arch_vcpu_load

Can anyone suggest where I go from here to track down the issue?

Thanks