Re: [Qemu-block] [RFC v6] RBD: Add support readv,writev for rbd

[Jaze Lee]

I test on cache=none with sata disk, because i do not have ssd in my pc,
and I do not find any performance improved.

I do not know why the performance is not be improved....


The test case is:
boot vm from rbd, and attatch one rbd volume with 100G size to it. The
qemu and the osd is on the same host.

The fio is
[global]
ioengine=libaio
filename=/dev/vdb
thread
group_reporting
direct=1
iodepth=32

[write_32k]
bs=32k
rw=write
runtime=30
stonewall

[rand_write_4k]
bs=4k
rw=randwrite
runtime=30
stonewall

the result is


# origin
address@hidden test]# fio test.fio
write_32k: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
ioengine=libaio, iodepth=32
rand_write_4k: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K,
ioengine=libaio, iodepth=32
fio-2.2.8
Starting 2 threads
Jobs: 1 (f=1): [_(1),w(1)] [67.0% done] [0KB/1342KB/0KB /s] [0/335/0
iops] [eta 00m:30s]
write_32k: (groupid=0, jobs=1): err= 0: pid=1842: Wed Feb 22 16:13:51 2017
  write: io=746496KB, bw=24851KB/s, iops=776, runt= 30039msec
    slat (usec): min=2, max=853, avg= 8.45, stdev=10.68
    clat (msec): min=21, max=557, avg=41.19, stdev=43.43
     lat (msec): min=21, max=557, avg=41.20, stdev=43.43
    clat percentiles (msec):
     |  1.00th=[   23],  5.00th=[   23], 10.00th=[   23], 20.00th=[   23],
     | 30.00th=[   24], 40.00th=[   24], 50.00th=[   24], 60.00th=[   24],
     | 70.00th=[   25], 80.00th=[   58], 90.00th=[   91], 95.00th=[  108],
     | 99.00th=[  247], 99.50th=[  306], 99.90th=[  465], 99.95th=[  553],
     | 99.99th=[  562]
    bw (KB  /s): min= 4622, max=43722, per=100.00%, avg=25227.04, stdev=9684.02
    lat (msec) : 50=79.46%, 100=14.34%, 250=5.32%, 500=0.82%, 750=0.06%
  cpu          : usr=0.48%, sys=0.65%, ctx=2278, majf=0, minf=6
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=23328/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rand_write_4k: (groupid=1, jobs=1): err= 0: pid=1847: Wed Feb 22 16:13:51 2017
  write: io=45668KB, bw=1519.8KB/s, iops=379, runt= 30063msec
    slat (usec): min=1, max=682, avg= 8.05, stdev=12.10
    clat (msec): min=8, max=549, avg=84.24, stdev=54.92
     lat (msec): min=8, max=549, avg=84.25, stdev=54.92
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   18], 10.00th=[   19], 20.00th=[   39],
     | 30.00th=[   72], 40.00th=[   79], 50.00th=[   86], 60.00th=[   93],
     | 70.00th=[   95], 80.00th=[  102], 90.00th=[  112], 95.00th=[  161],
     | 99.00th=[  326], 99.50th=[  367], 99.90th=[  486], 99.95th=[  486],
     | 99.99th=[  553]
    bw (KB  /s): min=  396, max= 4548, per=100.00%, avg=1530.29, stdev=860.64
    lat (msec) : 10=0.09%, 20=17.75%, 50=2.93%, 100=54.92%, 250=21.83%
    lat (msec) : 500=2.48%, 750=0.02%
  cpu          : usr=0.23%, sys=0.45%, ctx=5636, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.7%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=11417/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=746496KB, aggrb=24850KB/s, minb=24850KB/s, maxb=24850KB/s,
mint=30039msec, maxt=30039msec

Run status group 1 (all jobs):
  WRITE: io=45668KB, aggrb=1519KB/s, minb=1519KB/s, maxb=1519KB/s,
mint=30063msec, maxt=30063msec

Disk stats (read/write):
  vdb: ios=122/34652, merge=0/0, ticks=69/1911324, in_queue=1913516, util=99.54%



#the scatter io is

address@hidden test]# fio test.fio
write_32k: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
ioengine=libaio, iodepth=32
rand_write_4k: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K,
ioengine=libaio, iodepth=32
fio-2.2.8
Starting 2 threads
Jobs: 1 (f=1): [_(1),w(1)] [67.0% done] [0KB/1288KB/0KB /s] [0/322/0
iops] [eta 00m:30s]
write_32k: (groupid=0, jobs=1): err= 0: pid=1841: Wed Feb 22 16:21:51 2017
  write: io=746496KB, bw=24837KB/s, iops=776, runt= 30056msec
    slat (usec): min=2, max=183, avg= 8.48, stdev= 9.17
    clat (msec): min=21, max=474, avg=41.22, stdev=42.54
     lat (msec): min=21, max=474, avg=41.23, stdev=42.54
    clat percentiles (msec):
     |  1.00th=[   23],  5.00th=[   23], 10.00th=[   23], 20.00th=[   24],
     | 30.00th=[   24], 40.00th=[   24], 50.00th=[   24], 60.00th=[   24],
     | 70.00th=[   25], 80.00th=[   58], 90.00th=[   95], 95.00th=[  108],
     | 99.00th=[  225], 99.50th=[  314], 99.90th=[  453], 99.95th=[  474],
     | 99.99th=[  474]
    bw (KB  /s): min= 5466, max=43786, per=100.00%, avg=25117.63, stdev=9332.73
    lat (msec) : 50=79.86%, 100=13.43%, 250=5.92%, 500=0.78%
  cpu          : usr=0.41%, sys=0.76%, ctx=2614, majf=0, minf=6
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=23328/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rand_write_4k: (groupid=1, jobs=1): err= 0: pid=1846: Wed Feb 22 16:21:51 2017
  write: io=44980KB, bw=1496.7KB/s, iops=374, runt= 30055msec
    slat (usec): min=1, max=489, avg= 8.07, stdev= 9.00
    clat (msec): min=10, max=514, avg=85.51, stdev=57.00
     lat (msec): min=10, max=514, avg=85.52, stdev=57.00
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   18], 10.00th=[   19], 20.00th=[   43],
     | 30.00th=[   71], 40.00th=[   83], 50.00th=[   86], 60.00th=[   93],
     | 70.00th=[   95], 80.00th=[  102], 90.00th=[  119], 95.00th=[  176],
     | 99.00th=[  322], 99.50th=[  359], 99.90th=[  510], 99.95th=[  515],
     | 99.99th=[  515]
    bw (KB  /s): min=  389, max= 4792, per=100.00%, avg=1515.82, stdev=850.12
    lat (msec) : 20=18.51%, 50=2.05%, 100=54.64%, 250=21.91%, 500=2.77%
    lat (msec) : 750=0.12%
  cpu          : usr=0.27%, sys=0.44%, ctx=5926, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.7%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=11245/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=746496KB, aggrb=24836KB/s, minb=24836KB/s, maxb=24836KB/s,
mint=30056msec, maxt=30056msec

Run status group 1 (all jobs):
  WRITE: io=44980KB, aggrb=1496KB/s, minb=1496KB/s, maxb=1496KB/s,
mint=30055msec, maxt=30055msec

Disk stats (read/write):
  vdb: ios=122/34509, merge=0/0, ticks=74/1911712, in_queue=1913542, util=99.58%


I also use systemstap thread_indent to trace the function time consumed.
The no-scatter.txt file is the original, and the scatter.txt is the scatter-io.




2017-02-21 19:30 GMT+08:00 Stefan Hajnoczi <address@hidden>:
> On Tue, Feb 21, 2017 at 03:43:59PM +0800, Jaze Lee wrote:
>> If I test , which cache mode i should choose ?
>>
>> writeback? or none? or default?
>
> Both cache=none and cache=writeback should be tested since block/rbd.c
> changes behavior depending on BDRV_O_NOCACHE.



-- 
谦谦君子

no-scatter.txt
Description: Text document

scatter.txt
Description: Text document