Re: [Qemu-devel] is there a limit on the number of in-flight I/O operations?

[Andrey Korolyov]

On Fri, Aug 28, 2015 at 3:31 AM, Josh Durgin <address@hidden> wrote:
> On 08/27/2015 09:49 AM, Stefan Hajnoczi wrote:
>>
>> On Mon, Aug 25, 2014 at 03:50:02PM -0600, Chris Friesen wrote:
>>>
>>> The only limit I see in the whole call chain from
>>> virtio_blk_handle_request() on down is the call to
>>> bdrv_io_limits_intercept() in bdrv_co_do_writev().  However, that doesn't
>>> provide any limit on the absolute number of inflight operations, only on
>>> operations/sec.  If the ceph server cluster can't keep up with the
>>> aggregate
>>> load, then the number of inflight operations can still grow indefinitely.
>>
>>
>> We probably shouldn't rely on QEMU I/O throttling to keep memory usage
>> reasonable.
>
>
> Agreed.
>
>> Instead rbd should be adjusted to support iovecs as you suggested.  That
>> way no bounce buffers are needed.
>
>
> Yeah, this is pretty simple to do. Internally librbd has iovec-equivalents.
> I'm not sure this is the main source of extra memory usage
> though.
>
> I suspect the main culprit here is rbd cache letting itself burst too
> large, rather than the bounce buffers.
>
> Andrey, does this still occur with caching off?

No, complete cache disablement helped so far perfectly, what cannot be
said for wt/wb modes. As I mentioned, the consumption curve is a
reactive by its nature, so I`ll try to catch the burst under valgrind
but cannot promise anything.