Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Fri, Dec 20, 2019 at 04:13:59PM +0000, Stefan Hajnoczi wrote:
> Hi,
> QEMU presents a command-line interface and QMP monitor for
> applications to interact with.  Applications actually need API
> bindings in their programming language.  Bindings avoid reimplementing
> code to spawn a QEMU process and interact with QMP.  QEMU is kind of
> lazy and de facto relies on libvirt for API bindings.
> 
> Is it time for better QEMU APIs?
> 
> 1. We have qapi-schema.json.  Let's render to HTML and publish
> versioned documentation online.

You have patches to publish this to the website already which is
good for git master. We still need to get this published for
stable releases, ideally we would have up to publish it for our
pre-existing stable releases back a year or two too.

At the same time though this is insufficient. It is *really* hard
to understand how QAPI schema maps to either JSON or CLI args for
either QMP or the CLI. Most docs examples just talk in terms of
-drive. I think the qapi schema docs could benefit from more
work to provide real world example usage inline.


> 2. scripts/qmp/ contains command-line tools for QMP communication.
> They could use some polish and then be shipped.

The qmp-shell in particular is interesting. Libvirt has a similar
need for a simpler way to deal with QMP and "virsh qemu-monitor-command"
has notable overlap with functionality of qmp-shell.

I wonder if there's a way to spin this off so that we have have a
standard QMP client shell that can either talk direct to a UNIX
socket, or indirect via libvirt's QMP command passthrough APIs.

> 3. python/qemu/ contains Python modules for managing a QEMU process
> and QMP communication.  This should be packaged in distros and
> available on PyPI.
>
> 4. Go and Rust bindings would also be useful.  There is
> https://github.com/intel/govmm but I think it makes sense to keep it
> in qemu.git and provide an interface similar to our Python modules.

Presumably you mean bindings for the QMP commands ?  This will
quickly expand to mean bindings for every aspect of QEMU configuration
that's currently QAPI-ified. It will quickly become a very big job
to design, develop & maintain, especially if you want to support
multiple languages in a natural way.

> 5. A jailer is needed to isolate the QEMU process and vhost-user
> device backends using seccomp, Linux namespaces, and maybe
> SELinux/AppArmor.  We used to be able to rely on libvirt for QEMU
> security, but it's becoming a common task for any device backend and
> IMO should be its own launcher tool.

Based on experiance in libvirt, this is an even larger job than (4),
as the feature set here is huge.  Much of it directly ties into the
config problem, as to deal with SELinux / namespace setup the code
needs to understand what resources to provide access to. This
requires a way to express 100% coverage of all QEMU configuration
in use & analyse it to determine what resources it implies. So this
ties strongly into QAPI-ification completion.


On the libvirt side, we've a strong long term desire to move our
codebase from C to either Go or Rust. In doing this refactoring
and rewriting, we would quite likely end up with self-contained
packages / modules / APIs that can be used independantly of the
main libvirt API. Thus there's probably scope here to collaborate
between libvirt & QEMU communities to develop new APIs.

Some of this code could be directly QEMU related, but other parts
of the code are likely to be generic & not tied to QEMU at all.
For example libvirt's SELinux, namespace, cgroup setup code is used
by LXC too. Thus from the libvirt POV, some libraries might be
deliverable by QEMU project, but others might be deliverable by
libvirt project. Or might be scope for a indepent new project to
deliver pieces.

Essentially we can consider a stack

     +--------------+
     |  Libvirt API |
     +--------------+
           |
           V
     +--------------+
     | Helper APIs  |
     +--------------+
           |
           V
     +--------------+
     |  QEMU        |
     +--------------+

Right now, everything in "Helper APIs" is hidden inside libvirt.
Some of that conceptually ought to be deliverable by QEMU. Some
of that ought to be spun out as separate deliverables by libvirt
or another independant project.

> 6. A configuration file format is sorely needed so that guest
> configuration can be persisted and easily launched.  Today you have to
> create a shell script that executes a QEMU command-line, but this is
> suboptimal because sharing executable scripts is dangerous from a
> security perspective and is hard to parse or modify programmatically.

A configuration file is just one aspect of our broader configuration
problem.

Our CLI in particular suffers from trying to be "all things to all people"
and IMHO is a direct cause of the perception of QEMU as being overly
complex, hard to understand & burdened by legacy support. Of course we
didn't set out intentionally on this path, things just grew organically
we were not succesfully enough at tackling technical debt over time.

To pick on blockdev, as it has had the most growth, IIUC we have

  $QEMU FILENAME
  $QEMU -hda FILENAME
  $QEMU -drive if=ide,file=FILENAME
  $QEMU -drive if=none,file=FILENAME,id=disk0 -device virtio-blk,drive=disk0
  $QEMU -blockdev driver=raw,file.filename=FILENAME,id=disk0 -device 
virtio-blk,drive=disk0

IIUC, in (all/most) cases "FILENAME" can be either a plain filename,
or a "json:{....}" string, so that's giving us possibly as many as
10 ways to configure disks, with increasing generality.

If you were designing QEMU from a clean slate it is unlikely we'd
plan to have 10 syntaxes. A clean slate would probably provide
2 different syntaxes at most - one simple but convenient, and one
complex but fully expressive.

For every one there's going to be some set of users who think
their chosen syntax is ideal, and will thus be unhappy if we
cull one choice.  For the health of QEMU long term though, I
think we're going to have to accept the need to make some people
temporarily unhappy in order to become sustaininable over the
long term.


This ultimately all stems from the fact that the CLI design has
aimed to provide 100% coverage of all QEMU features for mgmt apps,
while at the same time as providing convenient syntax for humans.
Except it has actually failed in the former, because there's some
stuff you can only configured after launching QEMU via QMP commands.

We can reduce some of the maint burden by mapping "convenient syntax"
onto the more expressive (usually QAPI modelled) syntax internally,
but I think that alone is insufficient.

We should step back & consider the bigger question of what we want
QEMU's approach to configuration to be.

With my libvirt hat on, we have to support JSON for a large portion
of config because we need to use QMP hotplug. Thus if we could eliminate
the need to use the CLI syntax entirely, it would remove / simplify
libvirt's code base to work exclusively in terms of JSON.

With this in mind, I think we could reasonably declare a high level
goal that

 - CLI arg syntax is *exclusively* for human targetted convenient syntax
 - CLI arg syntax will *not* aim for 100% feature coverage
 - CLI arg syntax will not guarantee strict back compat but will not
   gratituitously break human usage
 - JSON syntax will provide 100% feature coverage for mmgmt apps and humans
 - JSON syntax will not be exposed via ARGV
 - JSON syntax will be usable via a config file
 - JSON syntax will be usable via QMP
 - JSON syntax will provide strict back compat, but with deprecations
   policy followed

This is more or less what our long term goal with QAPI has always
been. The sticking point has always been the tension with CLI arg
syntax back-compat. We need to decide how to cut the knot so that
we can complete the QAPI-ification in a forseeable amount of time.

> In many of these areas we already have a partial solution.  It just
> needs more work.  I think it would be worth the effort and the mental
> shift to really providing APIs that are easy to use by applications.
> 
> What do you think?

Personally I think QEMU configuration is the single biggest pain point
for dealing with QEMU that we're not investing enough resources in.

> Have I missed things that are needed?
> 
> Have I included things that are unnecessary?

Many of the things you list above are conceptually nice, but imply a
lot of extra work for maintainers, as they are expanding the scope of
the QEMU project. There's a risk this could divert resources away from
tackling technical debt in things we're already maintaining. The flip
side of course is that some of the things might attract new contributors
to the project, because new concepts are inevitably more interesting to
some people than tackling technical debt. IOW both are important and
we need to find a balance between them.

One of the serious long term issues we've had in QEMU is our inability
to finish jobs that we start. The configuration problem is the biggest
example or something we started JSON-ifying 10 years ago, but are still
an unknowable amount of time away from finishing.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|