Hello guys,
I've found a strange issue with qemu lately.
I have some script which run 256 KVM VMs on a linux server. While I run VMs by qemu everything was good, but when I switched to libvirt the server performance decreased heavily.
I started investigate that and found following:
When I run my VMs with -hda option - everything works perfect.
When I run my VMs with -drive/-device options performance decreases heavily.
Libvirt starts VMs with -drive/-device options so I have that bad performance.
I have no idea how it can be real: -hda (old option) works perfect, -drive/-device (new option) works badly. I thought those options are interchangeable. Are they not?
Below you can find my configurations:
Command line to run qemu with -hda option:
/usr/bin/kvm \
-pidfile $pid_file \
-daemonize \
-display none \
-m 3072 \
-smp cores=1 \
-hda $img \
-serial telnet:127.0.0.1:$port_1,server,nowait \
-serial telnet:127.0.0.1:$port_2,server,nowait \
-net tap,ifname=$tap_0,vlan=1,script=no,downscript=no \
-net nic,model=e1000,vlan=1,macaddr=$mac_1
Command line to run qemu with -device/-drive options:
/usr/bin/kvm \
-pidfile $pid_file \
-daemonize \
-display none \
-m 3072 \
-smp cores=1 \
-drive file=$img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none \
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \
-serial telnet:127.0.0.1:$port_1,server,nowait \
-serial telnet:127.0.0.1:$port_2,server,nowait \
-net tap,ifname=$tap_0,vlan=1,script=no,downscript=no \
-net nic,model=e1000,vlan=1,macaddr=$mac_1
I checked the issue on ubuntu 12.04, 14.04 and 15.04 beta2. The issue are present on the all.
When it works good I have following in my htop:
CPU: Load average is about 3
Memory: 344195/386877MB
When it works badly I have following in my htop:
CPU: Load average is about 258
Memory: 109914/386881MB
vmstat shows
address@hidden:~/ios_xrv_test# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
6 129 0 191338944 194296 10591492 0 0 84 20 2 1 1 1 93 5
1 132 0 191300608 194296 10591492 0 0 11715 918 61126 136692 3 1 59 36
1 128 0 191291088 194296 10591492 0 0 12376 1383 62597 142102 3 1 61 35
3 131 0 191275904 194296 10591492 0 0 13502 1732 63313 144123 3 1 61 35
5 131 0 191228624 194296 10591492 0 0 12848 1262 62208 139809 3 1 60 36
-------------
address@hidden:~/ios_xrv_test# qemu-system-x86_64 --version
QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9), Copyright (c) 2003-2008 Fabrice Bellard
address@hidden:~/ios_xrv_test# uname -a
Linux lab05 3.19.0-12-generic #12-Ubuntu SMP Fri Apr 3 04:03:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
address@hidden:~/ios_xrv_test# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Vivid Vervet (development branch)
Release: 15.04
Codename: vivid
------ or
address@hidden:~/ios_xrv_test# qemu-system-x86_64 --version
QEMU emulator version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Bellard
address@hidden:~/ios_xrv_test# uname -a
Linux lab01 3.13.0-48-generic #80~precise1-Ubuntu SMP Thu Mar 12 19:30:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
address@hidden:~/ios_xrv_test# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise
----- or
address@hidden:~/ios_xrv_libvirt# qemu-system-x86_64 --version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.10), Copyright (c) 2003-2008 Fabrice Bellard
address@hidden:~/ios_xrv_libvirt# uname -a
Linux lab10 3.16.0-34-generic #45~14.04.1-Ubuntu SMP Tue Mar 24 11:14:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
address@hidden:~/ios_xrv_libvirt# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty
------
All servers have 384Gb RAM
Can you please help me with that?
Thanks
--
With best regards, Pavel Shirshov