On Thu, May 14, 2015 at 4:42 PM, Andrey Korolyov <address@hidden> wrote:
On Wed, Aug 27, 2014 at 9:43 AM, Chris Friesen
<address@hidden> wrote:
On 08/25/2014 03:50 PM, Chris Friesen wrote:
I think I might have a glimmering of what's going on. Someone please
correct me if I get something wrong.
I think that VIRTIO_PCI_QUEUE_MAX doesn't really mean anything with
respect to max inflight operations, and neither does virtio-blk calling
virtio_add_queue() with a queue size of 128.
I think what's happening is that virtio_blk_handle_output() spins,
pulling data off the 128-entry queue and calling
virtio_blk_handle_request(). At this point that queue entry can be
reused, so the queue size isn't really relevant.
In virtio_blk_handle_write() we add the request to a MultiReqBuffer and
every 32 writes we'll call virtio_submit_multiwrite() which calls down
into bdrv_aio_multiwrite(). That tries to merge requests and then for
each resulting request calls bdrv_aio_writev() which ends up calling
qemu_rbd_aio_writev(), which calls rbd_start_aio().
rbd_start_aio() allocates a buffer and converts from iovec to a single
buffer. This buffer stays allocated until the request is acked, which
is where the bulk of the memory overhead with rbd is coming from (has
anyone considered adding iovec support to rbd to avoid this extra
copy?).
The only limit I see in the whole call chain from
virtio_blk_handle_request() on down is the call to
bdrv_io_limits_intercept() in bdrv_co_do_writev(). However, that
doesn't provide any limit on the absolute number of inflight
operations,
only on operations/sec. If the ceph server cluster can't keep up with
the aggregate load, then the number of inflight operations can still
grow indefinitely.
Chris
I was a bit concerned that I'd need to extend the IO throttling code to
support a limit on total inflight bytes, but it doesn't look like that
will
be necessary.
It seems that using mallopt() to set the trim/mmap thresholds to 128K is
enough to minimize the increase in RSS and also drop it back down after
an
I/O burst. For now this looks like it should be sufficient for our
purposes.
I'm actually a bit surprised I didn't have to go lower, but it seems to
work
for both "dd" and dbench testcases so we'll give it a try.
Chris
Bumping this...
For now, we are rarely suffering with an unlimited cache growth issue
which can be observed on all post-1.4 versions of qemu with rbd
backend in a writeback mode and certain pattern of a guest operations.
The issue is confirmed for virtio and can be re-triggered by issuing
excessive amount of write requests without completing returned acks
from a emulator` cache timely. Since most applications behave in a
right way, the oom issue is very rare (and we developed an ugly
workaround for such situations long ago). If anybody is interested in
fixing this, I can send a prepared image for a reproduction or
instructions to make one, whichever is preferable.
Thanks!
A gentle bump: for at least rbd backend with writethrough/writeback
cache it is possible to achieve unlimited growth with lot of large
unfinished ops, what can be considered as a DoS. Usually it is
triggered by poorly written applications in the wild, like proprietary
KV databases or MSSQL under Windows, but regular applications,
primarily OSS databases, can trigger the RSS growth for hundreds of
megabytes just easily. There is probably no straight way to limit
in-flight request size by re-chunking it, as supposedly malicious
guest can inflate it up to very high numbers, but it`s fine to crash
such a guest, saving real-world stuff with simple in-flight op count
limiter looks like more achievable option.