Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?

[Stefan Hajnoczi]

On Fri, Oct 25, 2013 at 05:01:54PM +0200, Jack Wang wrote:
> We've seen guest block io lost in a VM.any response will be helpful
> 
> environment is:
> guest os: Ubuntu 1304
> running busy database workload with xfs on a disk export with virtio-blk
> 
> the exported vdb has very high infight io over 300. Some times later a
> lot io process in D state, looks a lot requests is lost in below storage
> stack.

Is the image file on a local file system or are you using a network
storage system (e.g. NFS, Gluster, Ceph, Sheepdog)?

If you run "vmstat 5" inside the guest, do you see "bi"/"bo" block I/O
activity?  If that number is very low or zero then there may be a
starvation problem.  If that number is reasonable then the workload is
simply bottlenecked on disk I/O.

virtio-blk only has 128 descriptors available so it's not possible to
have 300 requests pending at the virtio-blk layer.

If you suspect QEMU, try building qemu.git/master from source in case
the bug has already been fixed.

If you want to trace I/O requests, you might find this blog post on
writing trace analysis scripts useful:
http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html

Stefan