[Qemu-devel] [Bug 1829696] [NEW] qemu-kvm takes 100% CPU when running redhat/centos 7.6 guest VM OS

[Launchpad Bug Tracker]

You have been subscribed to a public bug:

Description
===========
When running redhat or centos 7.6 guest os on vm,
the cpu usage is very low on vm(100% idle), but on host,
qemu-kvm reports 100% cpu busy usage.

After searching some related bugs report,
I suspect that it is due to the clock settings in vm's domain xml.
My Openstack cluster uses the default clock settings as follow:
    <clock offset='utc'>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='hpet' present='no'/>
    </clock>
And in this report, https://bugs.launchpad.net/qemu/+bug/1174654
it claims that <timer name='rtc' track='guest'/> can solve the 100% cpu usage 
problem when using Windows Image Guest OS,
but I makes some tests, the solusion dose not work for me.


Steps to reproduce
==================
* create a vm using centos or redhat 7.6 image
* using sar tool inside vm and host to check the cpu usage, and compare them


Expected result
===============
host's cpu usage report should be same with vm's cpu usage


Actual result
=============
vm's cpu usage is 100% idle, host's cpu usage is 100% busy


Environment
===========
1. Exact version of OpenStack you are running.
# rpm -qa | grep nova
openstack-nova-compute-13.1.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-common-13.1.2-1.el7.noarch

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?
# libvirtd -V
libvirtd (libvirt) 3.9.0

# /usr/libexec/qemu-kvm --version
QEMU emulator version 2.6.0 (qemu-kvm-ev-2.6.0-28.el7_3.6.1), Copyright (c) 
2003-2008 Fabrice Bellard


Logs & Configs
==============
The VM xml:
<domain type='kvm' id='29'>
  <name>instance-00005022</name>
  <uuid>7f5a66a5-****-****-****-75dec****bbb</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
      <nova:package version="13.1.2-1.el7"/>
      <nova:name>*******</nova:name>
      <nova:creationTime>2019-05-20 03:08:46</nova:creationTime>
      <nova:flavor name="2d2dab36-****-****-****-246e9****110">
        <nova:memory>2048</nova:memory>
        <nova:disk>12</nova:disk>
        <nova:swap>2048</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="********************">****</nova:user>
        <nova:project uuid="********************">****</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="4496a420-****-****-****-b50f****ada3"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
    <vcpupin vcpu='0' cpuset='27'/>
    <emulatorpin cpuset='27'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='1'/>
    <memnode cellid='0' mode='strict' nodeset='1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Fedora Project</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>13.1.2-1.el7</entry>
      <entry name='serial'>64ab0e89-****-****-****-05312ef66983</entry>
      <entry name='uuid'>7f5a66a5-****-****-****-75decaf82bbb</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.3.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>IvyBridge</model>
    <topology sockets='1' cores='1' threads='1'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='xsaveopt'/>
    <numa>
      <cell id='0' cpus='0' memory='2097152' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' 
function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source 
file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk.swap'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' 
function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <source 
file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk.config'/>
      <backingStore/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-1'/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:a6:ea:4f'/>
      <source bridge='brq52c66dc3-64'/>
      <bandwidth>
        <inbound average='102400'/>
        <outbound average='102400'/>
      </bandwidth>
      <target dev='tapa29e94e5-42'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x0'/>
    </interface>
    <serial type='file'>
      <source 
path='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/console.log'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/10'/>
      <target type='isa-serial' port='1'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source 
path='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5910' autoport='yes' listen='0.0.0.0' 
keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' 
function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

CPU Usage Report inside VM:
# sar -u -P 0 1 5
Linux 3.10.0-957.el7.x86_64 (******)    05/20/2019      _x86_64_        (1 CPU)

11:34:40 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
11:34:41 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
11:34:42 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
11:34:43 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
11:34:44 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
11:34:45 AM       0      0.00      0.00      0.00      0.00      0.00    100.00
Average:          0      0.00      0.00      0.00      0.00      0.00    100.00

CPU Usage Report ON HOST(the vm's cpu is pinned on host's no.27 physic cpu):
# sar -u -P 27 1 5
Linux 3.10.0-862.el7.x86_64 (******)    05/20/2019      _x86_64_        (48 CPU)

11:34:40 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
11:34:41 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
11:34:42 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
11:34:43 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
11:34:44 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
11:34:45 AM      27    100.00      0.00      0.00      0.00      0.00      0.00
Average:         27    100.00      0.00      0.00      0.00      0.00      0.00

clocksource inside VM:
# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
kvm_clock

** Affects: qemu
     Importance: Undecided
         Status: New


** Tags: centos clock qemu-kvm redhat
-- 
qemu-kvm takes 100% CPU when running redhat/centos 7.6 guest VM OS
https://bugs.launchpad.net/bugs/1829696
You received this bug notification because you are a member of qemu-devel-ml, 
which is subscribed to QEMU.