randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64 ... randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64 Starting 16 processes randrw: (groupid=0, jobs=1): err= 0: pid=5400 read : io=2,042MB, bw=20,460KB/s, iops=5,114, runt=102203msec slat (usec): min=12, max=22,671, avg=175.37, stdev=231.46 clat (usec): min=3, max=6,003K, avg=2380.80, stdev=38172.74 bw (KB/s) : min= 0, max=69632, per=1.01%, avg=3319.37, stdev=8556.85 write: io=6,126MB, bw=61,378KB/s, iops=15,344, runt=102203msec slat (usec): min=11, max=23,406, avg=175.63, stdev=225.77 clat (usec): min=3, max=6,003K, avg=2316.07, stdev=31731.75 bw (KB/s) : min= 0, max=209680, per=0.49%, avg=4772.01, stdev=18459.42 cpu : usr=2.54%, sys=88.49%, ctx=18173, majf=0, minf=292 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=522760/1568248, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec): 250=0.01%, 500=1.07%, 750=1.56%, 1000=5.44% lat (msec): 2=57.00%, 4=26.19%, 10=8.19%, 20=0.51%, 50=0.02% lat (msec): 100=0.01%, 250=0.01%, 750=0.01%, >=2000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5403 read : io=2,042MB, bw=21,219KB/s, iops=5,304, runt= 98540msec slat (usec): min=12, max=1,037K, avg=175.41, stdev=1452.13 clat (usec): min=12, max=2,265K, avg=2222.05, stdev=14539.38 bw (KB/s) : min= 0, max=40920, per=1.04%, avg=3399.56, stdev=8589.12 write: io=6,126MB, bw=63,660KB/s, iops=15,915, runt= 98540msec slat (usec): min=12, max=1,037K, avg=175.82, stdev=1452.46 clat (usec): min=3, max=2,265K, avg=2210.09, stdev=14167.54 bw (KB/s) : min= 0, max=123904, per=0.55%, avg=5396.55, stdev=19414.89 cpu : usr=2.24%, sys=91.12%, ctx=10329, majf=0, minf=279 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=522738/1568270, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec): 250=0.01%, 500=0.08%, 750=0.93%, 1000=5.74% lat (msec): 2=59.91%, 4=25.46%, 10=7.34%, 20=0.48%, 50=0.03% lat (msec): 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01% lat (msec): >=2000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5405 read : io=2,041MB, bw=23,166KB/s, iops=5,791, runt= 90229msec slat (usec): min=12, max=26,837, avg=164.06, stdev=192.74 clat (usec): min=88, max=1,061K, avg=2002.88, stdev=4616.16 bw (KB/s) : min= 0, max=35328, per=1.18%, avg=3855.86, stdev=9105.04 write: io=6,127MB, bw=69,532KB/s, iops=17,383, runt= 90229msec slat (usec): min=12, max=26,838, avg=165.07, stdev=197.00 clat (usec): min=5, max=1,063K, avg=2022.56, stdev=6095.61 bw (KB/s) : min= 0, max=106512, per=0.52%, avg=5084.68, stdev=18985.13 cpu : usr=2.47%, sys=94.18%, ctx=9418, majf=0, minf=224 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=522557/1568451, short=0/0 lat (usec): 10=0.01%, 20=0.01%, 100=0.01%, 250=0.01%, 500=0.03% lat (usec): 750=0.32%, 1000=4.12% lat (msec): 2=64.10%, 4=26.07%, 10=5.15%, 20=0.19%, 50=0.01% lat (msec): 500=0.01%, 1000=0.01%, 2000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5406 read : io=2,040MB, bw=20,823KB/s, iops=5,205, runt=100313msec slat (usec): min=10, max=21,390, avg=182.87, stdev=265.77 clat (usec): min=2, max=166K, avg=2225.22, stdev=2174.95 bw (KB/s) : min= 0, max=47000, per=0.98%, avg=3196.89, stdev=8344.89 write: io=6,128MB, bw=62,557KB/s, iops=15,639, runt=100313msec slat (usec): min=10, max=24,630, avg=183.91, stdev=266.89 clat (usec): min=42, max=1,137K, avg=2231.98, stdev=2514.15 bw (KB/s) : min= 0, max=140096, per=0.48%, avg=4694.76, stdev=18152.16 cpu : usr=2.61%, sys=94.53%, ctx=12908, majf=0, minf=286 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=522197/1568811, short=0/0 lat (usec): 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.18% lat (usec): 750=1.40%, 1000=5.72% lat (msec): 2=57.76%, 4=26.26%, 10=7.41%, 20=1.16%, 50=0.10% lat (msec): 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5408 read : io=2,042MB, bw=21,579KB/s, iops=5,394, runt= 96893msec slat (usec): min=12, max=24,417, avg=177.97, stdev=245.24 clat (usec): min=4, max=606K, avg=2160.41, stdev=2076.76 bw (KB/s) : min= 0, max=39824, per=0.95%, avg=3118.12, stdev=8213.33 write: io=6,126MB, bw=64,744KB/s, iops=16,185, runt= 96893msec slat (usec): min=11, max=24,418, avg=178.20, stdev=239.99 clat (usec): min=2, max=794K, avg=2157.34, stdev=2056.01 bw (KB/s) : min= 0, max=119616, per=0.48%, avg=4760.83, stdev=18210.70 cpu : usr=2.89%, sys=94.53%, ctx=9526, majf=0, minf=290 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=522706/1568302, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01% lat (usec): 500=0.08%, 750=0.66%, 1000=5.12% lat (msec): 2=60.12%, 4=25.23%, 10=8.23%, 20=0.51%, 50=0.04% lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5411 read : io=2,045MB, bw=23,417KB/s, iops=5,854, runt= 89409msec slat (usec): min=12, max=24,150, avg=163.38, stdev=188.59 clat (usec): min=2, max=961K, avg=1997.13, stdev=5359.34 bw (KB/s) : min= 0, max=33176, per=1.20%, avg=3921.81, stdev=9249.26 write: io=6,123MB, bw=70,131KB/s, iops=17,532, runt= 89409msec slat (usec): min=11, max=24,151, avg=163.63, stdev=187.56 clat (usec): min=2, max=963K, avg=1997.59, stdev=5377.38 bw (KB/s) : min= 0, max=98808, per=0.59%, avg=5801.44, stdev=20325.41 cpu : usr=2.50%, sys=94.71%, ctx=8925, majf=0, minf=225 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=523433/1567575, short=0/0 lat (usec): 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.03% lat (usec): 750=0.26%, 1000=4.09% lat (msec): 2=66.49%, 4=23.59%, 10=5.29%, 20=0.23%, 50=0.01% lat (msec): 500=0.01%, 1000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5415 read : io=2,047MB, bw=21,052KB/s, iops=5,263, runt= 99562msec slat (usec): min=12, max=23,013, avg=179.49, stdev=231.02 clat (usec): min=5, max=990K, avg=2232.38, stdev=7309.81 bw (KB/s) : min= 0, max=40664, per=1.05%, avg=3450.76, stdev=8443.34 write: io=6,121MB, bw=62,956KB/s, iops=15,738, runt= 99562msec slat (usec): min=11, max=23,017, avg=179.92, stdev=228.20 clat (usec): min=2, max=991K, avg=2232.22, stdev=7402.44 bw (KB/s) : min= 0, max=120560, per=0.48%, avg=4713.87, stdev=17808.57 cpu : usr=2.55%, sys=93.26%, ctx=12195, majf=0, minf=293 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=524006/1567002, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec): 250=0.01%, 500=0.10%, 750=0.81%, 1000=4.50% lat (msec): 2=56.69%, 4=29.72%, 10=7.60%, 20=0.53%, 50=0.03% lat (msec): 100=0.01%, 500=0.01%, 1000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5418 read : io=2,043MB, bw=20,860KB/s, iops=5,215, runt=100302msec slat (usec): min=12, max=427K, avg=181.53, stdev=869.43 clat (usec): min=58, max=974K, avg=2238.65, stdev=6606.73 bw (KB/s) : min= 0, max=72088, per=1.04%, avg=3394.31, stdev=8585.12 write: io=6,125MB, bw=62,528KB/s, iops=15,632, runt=100302msec slat (usec): min=12, max=427K, avg=181.22, stdev=540.35 clat (usec): min=3, max=974K, avg=2251.61, stdev=6893.62 bw (KB/s) : min= 0, max=213368, per=0.51%, avg=4994.41, stdev=18717.14 cpu : usr=2.41%, sys=92.93%, ctx=12565, majf=0, minf=293 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=523081/1567927, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec): 250=0.01%, 500=0.76%, 750=2.30%, 1000=4.84% lat (msec): 2=56.04%, 4=26.96%, 10=8.49%, 20=0.55%, 50=0.05% lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5419 read : io=2,041MB, bw=21,657KB/s, iops=5,414, runt= 96514msec slat (usec): min=12, max=19,996, avg=177.09, stdev=231.33 clat (usec): min=3, max=112K, avg=2147.31, stdev=1724.55 bw (KB/s) : min= 0, max=40432, per=1.07%, avg=3499.30, stdev=8651.00 write: io=6,127MB, bw=65,005KB/s, iops=16,251, runt= 96514msec slat (usec): min=11, max=20,001, avg=177.60, stdev=228.69 clat (usec): min=3, max=983K, avg=2147.52, stdev=1869.10 bw (KB/s) : min= 0, max=119984, per=0.49%, avg=4768.17, stdev=18249.18 cpu : usr=2.80%, sys=94.82%, ctx=9438, majf=0, minf=289 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=522545/1568463, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01% lat (usec): 500=0.04%, 750=0.45%, 1000=4.45% lat (msec): 2=61.29%, 4=25.70%, 10=7.42%, 20=0.59%, 50=0.06% lat (msec): 100=0.01%, 250=0.01%, 1000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5421 read : io=2,043MB, bw=21,508KB/s, iops=5,376, runt= 97253msec slat (usec): min=11, max=24,839, avg=176.88, stdev=233.19 clat (usec): min=2, max=1,028K, avg=2179.00, stdev=6253.87 bw (KB/s) : min= 0, max=35480, per=1.02%, avg=3334.78, stdev=8350.75 write: io=6,125MB, bw=64,495KB/s, iops=16,123, runt= 97253msec slat (usec): min=10, max=24,838, avg=177.39, stdev=229.57 clat (usec): min=3, max=1,029K, avg=2172.26, stdev=5624.04 bw (KB/s) : min= 0, max=107000, per=0.47%, avg=4657.42, stdev=17796.07 cpu : usr=2.58%, sys=93.92%, ctx=9420, majf=0, minf=293 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=522919/1568089, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec): 250=0.01%, 500=0.07%, 750=0.41%, 1000=4.05% lat (msec): 2=59.96%, 4=27.26%, 10=7.84%, 20=0.37%, 50=0.03% lat (msec): 100=0.01%, 250=0.01%, 750=0.01%, 2000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5422 read : io=2,046MB, bw=20,865KB/s, iops=5,216, runt=100431msec slat (usec): min=12, max=18,936, avg=178.24, stdev=220.92 clat (usec): min=3, max=1,085K, avg=2277.03, stdev=10543.06 bw (KB/s) : min= 0, max=64048, per=0.98%, avg=3214.49, stdev=8315.93 write: io=6,122MB, bw=62,416KB/s, iops=15,604, runt=100431msec slat (usec): min=11, max=18,936, avg=179.54, stdev=238.08 clat (usec): min=3, max=1,218K, avg=2252.51, stdev=9355.08 bw (KB/s) : min= 0, max=188896, per=0.47%, avg=4623.42, stdev=17907.56 cpu : usr=2.40%, sys=92.13%, ctx=14663, majf=0, minf=289 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=523881/1567127, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01% lat (usec): 500=0.60%, 750=2.00%, 1000=5.03% lat (msec): 2=55.68%, 4=28.08%, 10=7.99%, 20=0.54%, 50=0.04% lat (msec): 100=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5423 read : io=2,043MB, bw=21,866KB/s, iops=5,466, runt= 95692msec slat (usec): min=10, max=25,313, avg=172.80, stdev=218.96 clat (usec): min=32, max=1,748K, avg=2109.43, stdev=4990.48 bw (KB/s) : min= 0, max=42856, per=1.14%, avg=3741.56, stdev=8800.77 write: io=6,125MB, bw=65,539KB/s, iops=16,384, runt= 95692msec slat (usec): min=10, max=25,312, avg=173.74, stdev=225.83 clat (usec): min=3, max=1,750K, avg=2122.99, stdev=7473.72 bw (KB/s) : min= 0, max=124768, per=0.57%, avg=5562.20, stdev=19413.97 cpu : usr=2.46%, sys=93.24%, ctx=9598, majf=0, minf=296 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=523110/1567898, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01% lat (usec): 500=0.06%, 750=0.49%, 1000=4.28% lat (msec): 2=59.46%, 4=28.76%, 10=6.62%, 20=0.30%, 50=0.02% lat (msec): 500=0.01%, 2000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5426 read : io=2,042MB, bw=21,241KB/s, iops=5,310, runt= 98463msec slat (usec): min=12, max=328K, avg=181.08, stdev=512.92 clat (usec): min=2, max=329K, avg=2194.43, stdev=2135.67 bw (KB/s) : min= 0, max=41096, per=1.01%, avg=3321.97, stdev=8418.27 write: io=6,126MB, bw=63,704KB/s, iops=15,926, runt= 98463msec slat (usec): min=12, max=328K, avg=181.45, stdev=511.49 clat (usec): min=3, max=806K, avg=2190.69, stdev=2246.42 bw (KB/s) : min= 0, max=123336, per=0.42%, avg=4136.03, stdev=16998.12 cpu : usr=2.38%, sys=95.15%, ctx=10125, majf=0, minf=301 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=522875/1568133, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01% lat (usec): 500=0.08%, 750=1.02%, 1000=5.56% lat (msec): 2=58.21%, 4=25.73%, 10=8.73%, 20=0.61%, 50=0.03% lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5430 read : io=2,044MB, bw=23,139KB/s, iops=5,784, runt= 90434msec slat (usec): min=9, max=17,771, avg=166.73, stdev=205.87 clat (usec): min=3, max=28,581, avg=2011.96, stdev=1287.60 bw (KB/s) : min= 0, max=33024, per=1.21%, avg=3977.11, stdev=9220.77 write: io=6,124MB, bw=69,349KB/s, iops=17,337, runt= 90434msec slat (usec): min=8, max=17,772, avg=167.18, stdev=200.49 clat (usec): min=2, max=322K, avg=2012.38, stdev=1318.95 bw (KB/s) : min= 0, max=101904, per=0.56%, avg=5468.81, stdev=19634.52 cpu : usr=2.67%, sys=95.33%, ctx=9582, majf=0, minf=223 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=523142/1567866, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 100=0.01%, 250=0.01%, 500=0.04% lat (usec): 750=0.30%, 1000=4.09% lat (msec): 2=64.07%, 4=25.85%, 10=5.29%, 20=0.33%, 50=0.01% lat (msec): 100=0.01%, 250=0.01%, 500=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5433 read : io=2,041MB, bw=21,631KB/s, iops=5,407, runt= 96622msec slat (usec): min=12, max=24,818, avg=177.45, stdev=239.57 clat (usec): min=2, max=54,795, avg=2148.21, stdev=1634.62 bw (KB/s) : min= 0, max=44208, per=1.15%, avg=3781.51, stdev=9009.51 write: io=6,127MB, bw=64,934KB/s, iops=16,233, runt= 96622msec slat (usec): min=11, max=24,820, avg=178.45, stdev=235.05 clat (usec): min=3, max=827K, avg=2153.44, stdev=1880.85 bw (KB/s) : min= 0, max=131424, per=0.51%, avg=5039.91, stdev=18868.19 cpu : usr=2.46%, sys=95.35%, ctx=9448, majf=0, minf=292 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=522502/1568506, short=0/0 lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01% lat (usec): 500=0.07%, 750=0.91%, 1000=5.80% lat (msec): 2=59.72%, 4=24.46%, 10=8.39%, 20=0.59%, 50=0.05% lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% randrw: (groupid=0, jobs=1): err= 0: pid=5435 read : io=2,043MB, bw=21,489KB/s, iops=5,372, runt= 97369msec slat (usec): min=12, max=19,281, avg=177.99, stdev=228.71 clat (usec): min=52, max=51,356, avg=2165.59, stdev=1640.70 bw (KB/s) : min= 0, max=40880, per=1.17%, avg=3832.59, stdev=8911.65 write: io=6,125MB, bw=64,412KB/s, iops=16,102, runt= 97369msec slat (usec): min=11, max=21,294, avg=179.67, stdev=243.44 clat (usec): min=23, max=917K, avg=2169.18, stdev=1955.65 bw (KB/s) : min= 0, max=117336, per=0.47%, avg=4608.21, stdev=17816.78 cpu : usr=2.60%, sys=94.74%, ctx=10485, majf=0, minf=295 IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0% issued r/w: total=523084/1567924, short=0/0 lat (usec): 50=0.01%, 100=0.01%, 250=0.01%, 500=0.07%, 750=0.51% lat (usec): 1000=4.43% lat (msec): 2=60.54%, 4=26.37%, 10=7.22%, 20=0.81%, 50=0.03% lat (msec): 100=0.01%, 500=0.01%, 1000=0.01% Run status group 0 (all jobs): READ: io=32,686MB, aggrb=320MB/s, minb=20,950KB/s, maxb=23,979KB/s, mint=89409msec, maxt=102203msec WRITE: io=98,002MB, aggrb=959MB/s, minb=62,850KB/s, maxb=71,813KB/s, mint=89409msec, maxt=102203msec Disk stats (read/write): sdb: ios=524732/1572507, merge=0/0, ticks=103623/284056, in_queue=386938, util=90.65% sdc: ios=524007/1573232, merge=0/0, ticks=136650/387572, in_queue=526039, util=95.12% sdf: ios=524172/1573069, merge=1/0, ticks=103816/328857, in_queue=432885, util=92.30% sdg: ios=524563/1572702, merge=0/0, ticks=143133/430861, in_queue=574566, util=95.14% sdj: ios=524697/1572620, merge=0/0, ticks=105559/287185, in_queue=393271, util=91.25% sdk: ios=524189/1573176, merge=0/1, ticks=108667/340430, in_queue=449542, util=92.97% sdn: ios=524898/1572471, merge=0/0, ticks=117013/362832, in_queue=480406, util=94.40% sdo: ios=525975/1571417, merge=0/0, ticks=127692/400875, in_queue=530214, util=95.18% sdr: ios=525067/1572325, merge=0/1, ticks=123347/364557, in_queue=487667, util=95.68% sds: ios=523876/1573487, merge=0/1, ticks=126661/401779, in_queue=529278, util=96.14% sdv: ios=525253/1572170, merge=0/0, ticks=104001/300690, in_queue=406165, util=92.12% sdw: ios=525082/1572239, merge=1/16, ticks=281593/787233, in_queue=1059862, util=97.55% sdz: ios=525599/1571785, merge=0/1, ticks=155029/453394, in_queue=608861, util=95.61% sdaa: ios=511876/1536335, merge=0/0, ticks=113820/306599, in_queue=420507, util=89.85% sdad: ios=524086/1573304, merge=0/2, ticks=164113/491454, in_queue=656369, util=95.99% sdae: ios=512678/1535528, merge=0/0, ticks=117502/347213, in_queue=465071, util=91.67%