Re: [Qemu-devel] [RFC 1/2] pci-dma-api-v1

[Andrea Arcangeli]

On Sat, Nov 29, 2008 at 09:48:22PM +0200, Avi Kivity wrote:
> Since Andrea's patches contain emulation for aio readv/writev, the 
> performance degradation will not occur (though we will not see the benefit 
> either).

It will not occur especially if you #define DEBUG_BOUNCE. The way I
emulated bdrv_aio_readv/writev didn't involve a bounce, but it's
serially submitting the I/O so that it truly runs zerocopy when
DEBUG_BOUNCE is not defined ;).

IF you enable DEBUG_BOUNCE then the bounce layer will be forced on,
and the iovec will be linearized and the DMA command executed on the
hardware will be allowed to be as large as MAX_DMA_BOUNCE_BUFFER like
before. So until we have a real bdrv_aio_readv/writev #defining
DEBUG_BOUNCE is a must with cache=off.

> I doubt you can get measure malloc overhead with anything less a thousand 
> disks (even there, other overheads are likely to drown that malloc).

I also eliminated any sign of malloc in the direct path with a small
cache layer that caches as many pci dma sg params (with the max iovcnt
seen so far embedded into it) as the max number of simultaneous
in-flight I/O seen for the whole runtime. With a max param of 10 (so
only if there are more than 10 simultaneous sg dma I/O operations in
flight, malloc will have to run). Only params with iov arrays with a
iovcnt < 2048 are cached. So worst case ram waste is limited, and it's
auto-tuning at runtime to remain near zero for single disk setups etc..