Re: [Qemu-devel] [RFC 2/3] QMP: rate limit BLOCK_IO_ERROR

[Daniel P. Berrange]

On Mon, Aug 11, 2014 at 01:07:51PM +0200, Markus Armbruster wrote:
> "Daniel P. Berrange" <address@hidden> writes:
> > For the BLOCK IO ERROR events this does not work because the events are
> > device and operation specific.
> >
> >   QAPI_EVENT_BLOCK_IO_ERROR dev=ide0-hd1 op=read action=stop
> >   QAPI_EVENT_BLOCK_IO_ERROR dev=scsi1-hd2 op=write action=stop
> >   QAPI_EVENT_BLOCK_IO_ERROR dev=ide0-hd1 op=write action=stop
> >
> > with throttling the app wll only receive
> >
> >   QAPI_EVENT_BLOCK_IO_ERROR dev=ide0-hd1 op=write action=stop
> >
> > which means it will have an *incorrect* view of the system state because
> > the info about  scsi1-hd2 is irretrievably lost, likewise info about the
> > read operation of ide0-hd1.
> 
> Even when the event is lost, the information should not be lost.  There
> should be a way to poll for it (libvirt needs that anyway, to cope with
> possible event loss during a libvirt restart).

Yes, that's true.

> > If you want to throttle BLOCK IO ERROR events, then you need to make the
> > monitor throttling more intelligent, so that it hashes on all the contextual
> > state. In this case you'd have to throttle based on (event, dev, op) to get
> > correct application behaviour.
> 
> I think there's more than one  to skin this cat:
> 
> 1. Don't throttle.  Client can rely on events as long as it keeps the
>    QMP connection alive.  Client should poll after establishing the QMP
>    connection.

A malicious guest OS can flood libvirt with events in this way. Of course
even if we throttle, a compromised QEMU can still flood libvirt. The only
fail-safe protection is for libvirt to detect flooding and throttle the
rate at which it talks to the (malicious) QEMU.

> 2. Throttle more smartly, so that events only get dropped when they're
>    semantically superseded.  I figure that's what you proposed in your
>    last paragraph.

Yep, that's what I was suggesting.

> 3. Throttle, but accumulate the information carried by the event, i.e.
>    any dropped events' data is sent with the next non-dropped event.

I fear this could get rather ugly - fields which are currently scalar
quantities would need to become lists or hashes.

> 4. Throttle without smarts or accumulation.
> 
>    a. The event's additional information may be incomplete, thus
>       worthless.  Client needs too poll after getting an event.

Might as well just not bother sending any additionl info in events
if we took this path.

> 
>    b. Add a flag "throttling has dropped some events".  The additional
>       information is incomplete when the flag is set.  Client needs to
>       poll then.

This is a reasonable idea too.

> Backward compatibility considerations may narrow our choice.

I think  1, 2 or 4b are viable from a general design POV, but only 1 or 2
are viable from a back-compat POV, unless there was an explicit command
that client apps issued to turn on the throttling in 4b instead of it
being on by default.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|