Re: Making QEMU easier for management tools and applications

[Christophe de Dinechin]

Hi Stefan,



> On 20 Dec 2019, at 17:13, Stefan Hajnoczi <address@hidden> wrote:
> 
> Hi,
> QEMU presents a command-line interface and QMP monitor for
> applications to interact with.  Applications actually need API
> bindings in their programming language.  Bindings avoid reimplementing
> code to spawn a QEMU process and interact with QMP.  QEMU is kind of
> lazy and de facto relies on libvirt for API bindings.
> 
> Is it time for better QEMU APIs?

I hear that often ;-)

After catching up with this thread, I think that there is a common,
intuitive sense that a number of topics are interconnected and
relate to the QEMU API, from command-line to language bindings
to sandboxing to configuration files, and even to rewriting in Rust…

A good question is: why do we believe that? Thinking about it, it’s
not entirely obvious, is it, that a C API has anything to do with a
configuration file, is it?

(I _believe_ that it is true, but I’m trying to explore why our intuition
tells us so)

My gut feeling at this point is that we are not talking about an API,
we are first talking about a _language_, with relatively complicated
and loosely defined semantics. There are signs of this language
everywhere, from some custom object-model to several distinct
syntaxen (from ever-more-complex command-line options to JSON).
In order to see what I mean with “relatively complicated semantics”,
see the recent discussions about -hda vs. -drive vs. -device vs. -blockdev…

It does not help that a large fraction of this language is used to describe
graphs, e.g. network or bus topology. Textual languages make it
notoriously difficult to describe graphs, forcing us to add intermediate
user-provided names e.g. to interconnect a guest device and the
associated host resource, or to precisely define how guest NICs
connect to various virtual networks or host interface cards.

We then have several dialects of what should be, at least conceptually,
the same language, including, but not limited, to an XML dialect, a virsh
dialect, a command-line dialect, a qmp dialect, a JSON dialect that is
partially incorporated in several of the above, etc…

Of note, several tools, including virsh, git, OpenShift or dnf, have started
moving away from the simple “list of switches” towards having sub-commands.
Some, like virsh, even switch to “shell mode” if called without arguments.
So maybe qemu should enter “shell mode” if given no argument, and
accept a command file as input… If only there was no backward
compatibility issue…

So I think that it might help, in the long run, to start defining the
language in question in some abstract way, and then to have rules
for how to transform that abstract language into concrete bindings.
This definition itself is not obvious (at least not to me). For
example, do we have, anywhere but in the C code, the specification
of how one can add a disk to qemu, and what it means?
Say, looking at qemu-options.def, how do I tell that -hda has
anything to do with -device or -blockdev or -help?

I think that the following piece of code from vl.c is revealing:

            case QEMU_OPTION_hda:
            case QEMU_OPTION_hdb:
            case QEMU_OPTION_hdc:
            case QEMU_OPTION_hdd:
                drive_add(IF_DEFAULT, popt->index - QEMU_OPTION_hda, optarg,
                          HD_OPTS);
                break;
            case QEMU_OPTION_blockdev:
                {
                    Visitor *v;
                    BlockdevOptionsQueueEntry *bdo;

                    v = qobject_input_visitor_new_str(optarg, "driver",
                                                      &error_fatal);

                    bdo = g_new(BlockdevOptionsQueueEntry, 1);
                    visit_type_BlockdevOptions(v, NULL, &bdo->bdo,
                                               &error_fatal);
                    visit_free(v);
                    loc_save(&bdo->loc);
                    QSIMPLEQ_INSERT_TAIL(&bdo_queue, bdo, entry);
                    break;
                }
            case QEMU_OPTION_drive:
                if (drive_def(optarg) == NULL) {
                    exit(1);
                }
                break;

Here, we have three cases related to disks in a way or another,
and three entirely different ways of doing things.

AFAICT, qemu already created several meta-languages to define
several aspects of the API, from qemu-options.def to qapi-schema.json.
But maybe at some point we need to go meta once more, and define
a language defining the API from which we could automatically
derive the various bindings, including FFI-style bindings for Rust and Go,
as well as some internal data structures. Ideally, that meta-definition
is something that could be shared between libvirt and qemu so that they
literally speak the same language. Or that could be used to automatically
build a REST interface.

A big issue, though, is that of compatibility. Doing the above starting
from scratch does not seem that complicated. Doing it in a way that
preserves a minimum of interoperability with earlier-generation
software is another ordeal.

So I think that Daniel is right. We may need at some point to start
a NEMU-style offshoot that does not attempt to be compatible,
but explores describing an increasing surface of the API using a
new meta-language from which we can generate, in a consistent
way, at least:

- C bindings
- Command-line options
- Shell bindings (or “HMP”)
- JSON schema or qom description
- Bindings in other languages (Rust, Go, Python)
- Networked versions of the API (socket, REST)
- Client-side code e.g. for libvirt.
- Serialization / deserialization, e.g. for configuration files
- Documentation, including man page and API docs
- Command-line help

At the most fundamental level, I think we need to describe:

- Values, e.g. how we represent names, sizes, paths, etc, possibly
with some user-friendly aspects, e.g. path shortcuts, memory units,
spelling shortcuts (e.g. being able to consistently say -blo for -blockdev
if that’s the shortest option that matches)
- Relations, e.g. how we represent “contains”, “derives from”, “needs”,
“one of”, “one or several”, “attaches to”…
- States, e.g. how do we represent the machine configuration,
or the desired new disk setting
- Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
“start”, “notify”, etc. and how we describe the kind of input they need.
- Possibly more subtle things like support for transactions, commit/rollback,
i.e. “I want to add connect a virtual nic to some host vf, but if anything
along the way fails, I’d like all the cleanup to happen automatically)


> 
> 1. We have qapi-schema.json.  Let's render to HTML and publish
> versioned documentation online.

Added “Documentation” above to make sure this is preserved.
I agree with the assessment that the existing commenting state
is not good enough to make this that valuable at this point.
Notably missing are all the relation aspects.


> 
> 2. scripts/qmp/ contains command-line tools for QMP communication.
> They could use some polish and then be shipped.

Shouldn’t we first identify what they do and how to use them? ;-)

> 
> 3. python/qemu/ contains Python modules for managing a QEMU process
> and QMP communication.  This should be packaged in distros and
> available on PyPI.

That is probably somewhat independent from the rest, although
in my “meta” model, we should certainly be able to also generate
some python bindings.

> 
> 4. Go and Rust bindings would also be useful.  There is
> https://github.com/intel/govmm but I think it makes sense to keep it
> in qemu.git and provide an interface similar to our Python modules.

Same comment as above.

> 
> 5. A jailer is needed to isolate the QEMU process and vhost-user
> device backends using seccomp, Linux namespaces, and maybe
> SELinux/AppArmor.  We used to be able to rely on libvirt for QEMU
> security, but it's becoming a common task for any device backend and
> IMO should be its own launcher tool.

I believe that this “jailer” comment is interesting. It’s not really
related to a refactoring of APIs, it looks more like an API that we
forgot because libvirt did it, and we now realize that this may not
be the right place for some use cases.

> 
> 6. A configuration file format is sorely needed so that guest
> configuration can be persisted and easily launched.  Today you have to
> create a shell script that executes a QEMU command-line, but this is
> suboptimal because sharing executable scripts is dangerous from a
> security perspective and is hard to parse or modify programmatically.

Agreed. That’s one reason why I started talking about a “language” above.


> In many of these areas we already have a partial solution.  It just
> needs more work.  I think it would be worth the effort and the mental
> shift to really providing APIs that are easy to use by applications.
> 
> What do you think?

Overall, I think this is not a simple problem.

> 
> Have I missed things that are needed?

The main one for me is the meta-language I talked about above.

> 
> Have I included things that are unnecessary?

No, but some of the things that you included surprised me in an “API”
discussion, and that’s what launched me in this “meta-language” direction.


> 
> Stefan
>