[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Qemu-discuss] qemu with librbd vs krbd - huge performance difference

From: Nikola Ciprich
Subject: [Qemu-discuss] qemu with librbd vs krbd - huge performance difference
Date: Tue, 4 Sep 2018 08:48:01 +0200
User-agent: Mutt/1.5.21 (2010-09-15)


I've seen this discussed few times without any real conclusion,
so I'd like to try asking again..

we're  seeing very bad performance on ceph volumes mapped to qemu using librbd,
while block access to krbd is multiple times better (when comparing fast 
storage iops).

rough numbers I'm getting:

qemu, block access to KRBD mapped volume:
read: > ~100K IOPS
write: > ~100K IOPS

qemu: librbd access to volume:
read: ~25K IOPS
write: ~20K IOPS

when I run FIO on host, using librbd, I'm getting ~140K IOPS on reads, ~50K IOPS
on write.

trying different cache modes does not have any important impact. I'm using 

no special settings, host is 4.14.60 x86_64, qemu-kvm-2.12.0, guest has  16 
cores assigned.

fio cmdline:

on host:
fio --readwrite=randread/randwrite --randrepeat=1 --ioengine=rbd --direct=1 
--numjobs=16 --gtod_reduce=1 --name=test --pool=${pool} --rbdname=${vol} 
--invalidate=0 --bs=4k --iodepth=64 --time_based --runtime=60 --group_reporting

in guest:
fio --readwrite=randread/randwrite --filename=/dev/sdx --randrepeat=1 
--ioengine=libaio --direct=1 --numjobs=16 --gtod_reduce=1 --name=test 
--invalidate=0 --bs=4k --iodepth=64 --time_based --runtime=60 --group_reporting

trying numjobs=1 leads to similar results (qemu krbd much faster then librbd)

no special rbd features enabled on volume, apart from layering.

any ideas on why the difference might be so huge?

thanks a lot in advance



Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799

mobil servis: +420 737 238 656
email servis: address@hidden

reply via email to

[Prev in Thread] Current Thread [Next in Thread]