Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/23/2010 09:24 AM, Avi Kivity wrote:
> On 04/23/2010 04:48 PM, Anthony Liguori wrote:
> > On 04/23/2010 07:48 AM, Avi Kivity wrote:
> > > On 04/22/2010 09:49 PM, Anthony Liguori wrote:
> > > > > real API. Say, adding a device libvirt doesn't know about or 
> > > > > stopping the VM
> > > > > while libvirt thinks it's still running or anything like that.
> > > >   Another problem is issuing Monitor commands that could confuse 
> > > > libvirt's
> > > >
> > > > We need to make libvirt and qemu smarter.
> > > >
> > > > We already face this problem today with multiple libvirt users.  
> > > > This is why sophisticated management mechanisms (like LDAP) have 
> > > > mechanisms to do transactions or at least a series of atomic 
> > > > operations.
> > >
> > > And people said qmp/json was overengineered...
> > >
> > > But seriously, transactions won't help anything.  qemu maintains 
> > > state, and when you have two updaters touching a shared variable not 
> > > excepting each other to, things break, no matter how much locking 
> > > there is.
> >
> > Let's consider some concrete examples.  I'm using libvirt and QMP and 
> > in QMP, I want to hot unplug a device.
> >
> > Today, I do this by listing the pci devices, and issuing a pci_del 
> > that takes a PCI address.  This is intrinsically racy though because 
> > in the worst case scenario, in between when I enumerate pci devices 
> > and do the pci_del in QMP, in libvirt, I've done a pci_del and then a 
> > pci_add within libvirt of a completely different device.
>
> Obviously you should do the pci_del through libvirt.  Once libvirt 
> supports an API, use it.

It was just an example...

> > There are a few ways to solve this, the simplest being that we give 
> > devices unique ids that are never reused and instead of pci_del 
> > taking a pci bus address, it takes a device id.  That would address 
> > this race.
> >
> > You can get very far by just being clever about unique ids and 
> > notifications.  There are some cases where a true RMW may be required 
> > but I can't really think of one off hand.  The way LDAP addresses 
> > this is that it has a batched operation and a simple set of boolean 
> > comparison operations.  This lets you execute a batched operation 
> > that will do a RMW.
>
> I'm sure we can be very clever, but I'd rather direct this cleverness 
> to qemu core issues, not to the QMP (which in turn requires that users 
> be clever to use it correctly).  QMP is a low bandwidth protocol, so 
> races will never show up in testing.  We're laying mines here for 
> users to step on that we will never encounter ourselves.
>
> > >   The only way that separate monitors could work is if they touch 
> > > completely separate state, which is difficult to ensure if you 
> > > upgrade your libvirt.
> >
> > I don't think this is as difficult of a problem as you think it is.  
> > If you look at Active Directory and the whole set of management tools 
> > based on it, they certainly allow concurrent management 
> > applications.  You can certainly get into trouble still but with just 
> > some careful considerations, you can make two management applications 
> > work together 90% of the time without much fuss on the applications 
> > part.
>
> Maybe.  We'll still have issues.  For example, sVirt: if a QMP command 
> names a labeled resource, the non-libvirt user will have no way of 
> knowing how to label it.

This is orthogonal to QMP and has to do strictly with how libvirt 
prepares a resource for qemu.

> Much better to exact a commitment from libvirt to track all QMP (and 
> command line) capabilities.  Instead of adding cleverness to QMP, add 
> APIs to libvirt.

Let's step back for a minute because I think we're missing the forest 
through the trees.

We're trying to address a few distinct problems:

1) Allow libvirt users to access features of qemu that are not exposed 
through libvirt

2) Provide a means for non-libvirt users to interact with qemu

3) Provide a unified and interoperable view of the world for non-libvirt 
and libvirt users

For (1), we all agree that the best case scenario would be for libvirt 
to support every qemu feature.  I think we can also all agree though 
that this is not really practical and certainly not practical for 
developers since there is a development cost associated with libvirt 
support (to model an API appropriately).

The new API proposed addresses (1) by allowing a user to drill down to 
the QMP context.  It's a good solution IMHO and I think we all agree 
that there's an inherent risk to this that users will have to evaluate 
on a case-by-case basis.  It's a good stop-gap though.

(2) is largely addressed by QMP and a config file.  I'd like to see a 
nice C library, but I think a lot of other folks are happy with JSON 
support in higher level languages.

(3) is the place where there are still potential challenges.  I think at 
the very least, our goal should be to enable conversion from (2) and (1) 
to be as easy as possible.  That's why I have proposed implementing a C 
library for the JSON transport because we could plumb that through the 
new libvirt API.  This would allow a user to very quickly port an 
application from QMP to libvirt.  In order to do this, we need the 
libvirt API to expose a dedicated monitor because we'll need to be able 
to manipulate events and negotiate features.

Beyond simple porting, there's a secondary question of having 
non-libvirt apps co-exist with libvirt apps.  I think it's a good long 
term goal, but I don't think we should worry too much about it now.

Regards,

Anthony Liguori