[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Making QEMU easier for management tools and applications

From: Kevin Wolf
Subject: Re: Making QEMU easier for management tools and applications
Date: Wed, 8 Jan 2020 14:38:42 +0100
User-agent: Mutt/1.12.1 (2019-06-15)

Am 08.01.2020 um 12:40 hat Christophe de Dinechin geschrieben:
> > On 8 Jan 2020, at 11:43, Kevin Wolf <address@hidden> wrote:
> > Am 07.01.2020 um 18:11 hat Christophe de Dinechin geschrieben:
> >> So I think that it might help, in the long run, to start defining the
> >> language in question in some abstract way, and then to have rules
> >> for how to transform that abstract language into concrete bindings.
> > 
> > I think this abstract language is QAPI.
> Currently, yes, that’s the closest we have to a well-defined language,
> I pointed it out in my mail, and there is qapi-code-gen.txt to prove it.

Oh, did you? I must have missed that point, sorry.

> So it would certainly make sense for the language I’m describing to
> be a forward evolution of QAPI, or something that is trivially transformed
> to / from the existing json schema.

I guess I need to be more specific about what I mean with "QAPI". Of
course, many things you can't currently do in QAPI because they are not
implemented. When I'm talking about that QAPI should be used more, I'm
mostly talking about applying its core concepts and infrastructure with
the obvious extensions in some places, not about the existing

So I guess "a forward evoluion of QAPI" is what I really have in mind.

> > The problem is that we're not even close to using QAPI for everything.
> That’s the symptom of the problem. The problem as I see it is that
> we could not. at least not with today’s QAPI (maybe I’m wrong).
> A good example is shortcut command-line options. I don’t think
> you can, today, derive a user-friendly command-line syntax from
> qapi-schema.json, and even less so one that is compatible with what
> we have today.

Well, you can't because it's not implemented. Implementing it from
scratch without compatibility requirements would be relatively easy, and
we're in fact going to try it out with the qemu-storage-daemon.

But there is nothing in defining a command line option that would
inherently be different in structure from defining a QMP command, so I
would consider providing a command line option that is described by a
QAPI type not a problem, even though it's not implemented yet.

Compatibility is much harder, which is not the least because the
existing command line options are bad (primarily: using inconsistent
syntax). The question is if we should make bad options work unchanged in
the new infrastructure or if it wouldn't be better to replace them with
something that behaves the same as the rest.

> > Adding a new language on top of QAPI instead isn't going to make the
> > conversion process any faster.
> I fully agree that whatever language we select has to have an easy,
> systematic forward path from QAPI, but also from the other
> existing meta-languages, like the .def files used for options,
> or build configuration (there is a hint of the connection between
> the two in the option ‘if’ member in the .json files).
> In other words, the language I have in mind would be something
> that I could auto-generate from, say, the current qapi-schema.json
> and the qemu-options.def. Maybe in the end, that would be simply
> by having qemu-options.def being used to build qemu-options.json,
> and then add the relevant entries in qemu-options.json. Maybe
> there is a better way.

I would rather say that .def should go away and its content should
be converted to a JSON schema that becomes the new source format rather
than an intermediate generated file. There is nothing in .def files that
couldn't be represented in the schema.

> >> This definition itself is not obvious (at least not to me). For
> >> example, do we have, anywhere but in the C code, the specification
> >> of how one can add a disk to qemu, and what it means?
> >> Say, looking at qemu-options.def, how do I tell that -hda has
> >> anything to do with -device or -blockdev or -help?
> > 
> > BlockdevOptions in the QAPI schema is what tells you how it _really_
> > works.
> Not really, IMO, in the sense that it gives me no clue as to how -hda
> or -device relate to it.
> An important point that you raise through your response, though, is
> that the QAPI schema is only an IDL (interface definition language).
> In other words, it can tell me what a BlockdevOptions look like on
> the wire, but at least in the current state, it cannot tell me what happens
> with it.
> This is probably a good thing (it’s already complicated enough as is),
> but it’s worth pointing out that, even sticking to a declarative language,
> one could add constraints, as hinted by HAS_ARG in the .def file
> (but we could have a richer constraints language, e.g. describing
> -m option so that there can be at most one, the arg is a number,
> can’t be negative or floating-point, and so on.

To a certain degree, QAPI does just that, by supporting different data
types for options. If necessary, I'm sure the type system could be
extended, but it's not clear to me to which degree we actually need

Just drawing parallels from what we do for QMP commands, I imagine
something like this to describe the -m option as it actually exists in
qemu-options.def today:

# @m:
# Configure guest RAM.
# @size: initial amount of guest memory
# @slots: number of hotplug slots (default: none)
# @maxmem: maximum amount of guest memory (default: none)
{ 'cmdline-option': 'm',
  'arguments': {
    'size': 'size',
    'slots': '*int',
    'maxmem': '*size'
  'default-key': 'size',
  'repeat': 'OVERRIDE' }

HAS_ARG is automatically covered simply by 'arguments' being non-empty.

(Markus probably already has a prototype with slightly different syntax,
but the idea should be similar.)

> And of course, we could decide to not stick to a declarative language,
> but to have a way to have some, possibly limited, processing
> described in the language. I’m thinking notably of what is called
> “desugaring” in other threads.

We could. But is it actually worth inventing a new programming language?
I think this is something that should be done in C code even in the
future. I think what we're looking for is a way to describe interfaces,
not implementations.

Even just for adding more sophisticated constraints, like in the example
above that maxmem >= size, it's questionable whether doing this in the
schema provides enough value for actually implementing it there.

> >> A big issue, though, is that of compatibility. Doing the above starting
> >> from scratch does not seem that complicated. Doing it in a way that
> >> preserves a minimum of interoperability with earlier-generation
> >> software is another ordeal.
> > 
> > Indeed, this is the major reason why QAPI isn't as pervasive as it
> > should be.
> Probably, but also because QAPI does not address the needs of
> some of the things a slightly more general language could do.

So what are the things that you would like to describe, but can't see an
easy way to actually describe them with QAPI concepts?

> >> So I think that Daniel is right. We may need at some point to start
> >> a NEMU-style offshoot that does not attempt to be compatible,
> >> but explores describing an increasing surface of the API using a
> >> new meta-language from which we can generate, in a consistent
> >> way, at least:
> >> 
> >> - C bindings
> >> - Command-line options
> >> - Shell bindings (or “HMP”)
> >> - JSON schema or qom description
> >> - Bindings in other languages (Rust, Go, Python)
> >> - Networked versions of the API (socket, REST)
> >> - Client-side code e.g. for libvirt.
> >> - Serialization / deserialization, e.g. for configuration files
> >> - Documentation, including man page and API docs
> >> - Command-line help
> > 
> > I think the only thing in this list that can't obviously be covered
> > easily by QAPI is QOM. Or rather, it's covered by passing through
> > key=value lists without describing their structure
> That’s close enough to me. (In my mind, that part was already “done”
> by QAPI, even if in the convoluted way you describe)

That's not really close enough. If you are happy with this level of
abstraction, you can define any command line option to be either a flag
or take a string as its argument and be done. This is obviously not very
helpful because it says nothing about the structure of that string. In
the same way, QAPI can't say anything about the structure of a QOM
object, and I think that's a problem.

> > - which, as far as I
> > understand, is mainly because QOM properties aren't necessarily static,
> > so we can't provide a statically defined interface for them. Probably
> > solvable in QEMU, but not without a major effort.
> Or maybe extend the language so that it’s internal semantics
> knows about this aspect of QOM?

My point is that for example you can't generate a C struct (which is
statically defined) from something that has a dynamic structure. The
best you could do is fall back to key=value even in the C source, both
key and value being strings. But then you still have to parse these
strings manually, so it doesn't actually help much compared to a state
without C bindings.

Maybe what could be done is covering at least the static properties and
then having key=value for the dynamic part (which should be the

> >> - Relations, e.g. how we represent “contains”, “derives from”, “needs”,
> >> “one of”, “one or several”, “attaches to”…
> >> - States, e.g. how do we represent the machine configuration,
> >> or the desired new disk setting
> >> - Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
> >> “start”, “notify”, etc. and how we describe the kind of input they need.
> >> - Possibly more subtle things like support for transactions, 
> >> commit/rollback,
> >> i.e. “I want to add connect a virtual nic to some host vf, but if anything
> >> along the way fails, I’d like all the cleanup to happen automatically)
> > 
> > This sounds like a different approach from our current QAPI command set
> Well, except for purposefully trying to use a different wording to avoid
> the risk of getting your mind stuck in one of the particular existing
> meta-languages in QEMU, the approach is not very different.

I didn't necessarily mean relations/state/verbs, which obviously exist,
but the examples that seemed to express things in a deliberately
different way from what we have today.

> - Transactions do not exist today that I know of, although we see
> signs of them in discussions about the fact that this options destroys
> that back end but that option does not.

We have a 'transaction' QMP command, but they are not an integral part
of the language. Considering the trouble to implement transactional
commands, I suppose we don't even want it to be a fundamental part of
the language.

> > Does it actually provide more functionality, though?
> It’s not intended to provide more, but to require less to do the same thing.

Though we always need to keep in mind that if requiring less for future
additions requires a huge effort now, the investment may pay off only in
a very distant future (if the abstraction we build even survives until


reply via email to

[Prev in Thread] Current Thread [Next in Thread]