[Qemu-devel] Re: [PATCH v5 0/4] virtio: Use ioeventfd for virtqueue notify

[Stefan Hajnoczi]

Here are 4k sequential read results (cache=none) to check whether we
see an ioeventfd performance regression with virtio-blk.

The idea is to use a small blocksize with an I/O pattern (sequential
reads) that is cheap and executes quickly.  Therefore we're doing many
iops and the cost virtqueue kick/notify is especially important.
We're not trying to stress the disk, we're trying to make the
difference in ioeventfd=on/off apparent.

I did 2 runs for both ioeventfd=off and ioeventfd=on.  The results are
similar: 1% and 2% degradation in MB/s or iops.  We'd have to do more
runs to see if the degradation is statistically significant, but the
percentage value is so low that I'm satisfied.

Are you happy to merge virtio-ioeventfd v6 + your fixups?

Full results below:

x86_64-softmmu/qemu-system-x86_64 -m 1024 -drive
if=none,file=rhel6.img,cache=none,id=system -device
virtio-blk-pci,drive=system -drive
if=none,file=/dev/volumes/storage,cache=none,id=storage -device
virtio-blk-pci,drive=storage -cpu kvm64,+x2apic -vnc :0

fio jobfile:
[global]
ioengine=libaio
buffered=0
rw=read
bs=4k
iodepth=1
runtime=2m

[job1]
filename=/dev/vdb

ioeventfd=off:
job1: (groupid=0, jobs=1): err= 0: pid=2692
  read : io=2,353MB, bw=20,080KB/s, iops=5,019, runt=120001msec
    slat (usec): min=20, max=1,424, avg=34.86, stdev= 7.62
    clat (usec): min=1, max=11,547, avg=162.02, stdev=42.95
    bw (KB/s) : min=16600, max=20328, per=100.03%, avg=20084.25, stdev=241.88
  cpu          : usr=1.14%, sys=13.40%, ctx=604918, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=602391/0, short=0/0
     lat (usec): 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
     lat (usec): 100=0.01%, 250=99.89%, 500=0.07%, 750=0.01%, 1000=0.02%
     lat (msec): 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%

Run status group 0 (all jobs):
   READ: io=2,353MB, aggrb=20,079KB/s, minb=20,561KB/s,
maxb=20,561KB/s, mint=120001msec, maxt=120001msec

Disk stats (read/write):
  vdb: ios=601339/0, merge=0/0, ticks=112092/0, in_queue=111815, util=93.38%

ioeventfd=on:
job1: (groupid=0, jobs=1): err= 0: pid=2692
  read : io=2,299MB, bw=19,619KB/s, iops=4,904, runt=120001msec
    slat (usec): min=9, max=2,257, avg=40.43, stdev=11.65
    clat (usec): min=1, max=28,000, avg=161.12, stdev=61.46
    bw (KB/s) : min=15720, max=19984, per=100.02%, avg=19623.26, stdev=290.76
  cpu          : usr=1.49%, sys=19.34%, ctx=591398, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=588578/0, short=0/0
     lat (usec): 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
     lat (usec): 100=0.01%, 250=99.86%, 500=0.09%, 750=0.01%, 1000=0.02%
     lat (msec): 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%

Run status group 0 (all jobs):
   READ: io=2,299MB, aggrb=19,619KB/s, minb=20,089KB/s,
maxb=20,089KB/s, mint=120001msec, maxt=120001msec

Disk stats (read/write):
  vdb: ios=587592/0, merge=0/0, ticks=110373/0, in_queue=110125, util=91.97%

Stefan