[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Making QEMU easier for management tools and applications

From: Stefan Hajnoczi
Subject: Re: Making QEMU easier for management tools and applications
Date: Thu, 2 Jan 2020 14:22:27 +0000

On Tue, Dec 24, 2019 at 01:00:35PM +0000, Daniel P. Berrangé wrote:
> On Fri, Dec 20, 2019 at 04:13:59PM +0000, Stefan Hajnoczi wrote:
> > Hi,
> > QEMU presents a command-line interface and QMP monitor for
> > applications to interact with.  Applications actually need API
> > bindings in their programming language.  Bindings avoid reimplementing
> > code to spawn a QEMU process and interact with QMP.  QEMU is kind of
> > lazy and de facto relies on libvirt for API bindings.
> > 
> > Is it time for better QEMU APIs?
> > 
> > 1. We have qapi-schema.json.  Let's render to HTML and publish
> > versioned documentation online.
> You have patches to publish this to the website already which is
> good for git master. We still need to get this published for
> stable releases, ideally we would have up to publish it for our
> pre-existing stable releases back a year or two too.
> At the same time though this is insufficient. It is *really* hard
> to understand how QAPI schema maps to either JSON or CLI args for
> either QMP or the CLI. Most docs examples just talk in terms of
> -drive. I think the qapi schema docs could benefit from more
> work to provide real world example usage inline.
> > 2. scripts/qmp/ contains command-line tools for QMP communication.
> > They could use some polish and then be shipped.
> The qmp-shell in particular is interesting. Libvirt has a similar
> need for a simpler way to deal with QMP and "virsh qemu-monitor-command"
> has notable overlap with functionality of qmp-shell.
> I wonder if there's a way to spin this off so that we have have a
> standard QMP client shell that can either talk direct to a UNIX
> socket, or indirect via libvirt's QMP command passthrough APIs.

A standard (and easier-to-use and powerful) QMP shell would be nice.

> > 3. python/qemu/ contains Python modules for managing a QEMU process
> > and QMP communication.  This should be packaged in distros and
> > available on PyPI.
> >
> > 4. Go and Rust bindings would also be useful.  There is
> > https://github.com/intel/govmm but I think it makes sense to keep it
> > in qemu.git and provide an interface similar to our Python modules.
> Presumably you mean bindings for the QMP commands ?  This will
> quickly expand to mean bindings for every aspect of QEMU configuration
> that's currently QAPI-ified. It will quickly become a very big job
> to design, develop & maintain, especially if you want to support
> multiple languages in a natural way.

No, I mean just launching QEMU and communicating over QMP.

I think creating bindings for QMP commands is too much effort.  It would
duplicate the QAPI schema documentation because we'd need native API
documentation too.

I think clients should work at the QMP JSON level.

> > 5. A jailer is needed to isolate the QEMU process and vhost-user
> > device backends using seccomp, Linux namespaces, and maybe
> > SELinux/AppArmor.  We used to be able to rely on libvirt for QEMU
> > security, but it's becoming a common task for any device backend and
> > IMO should be its own launcher tool.
> Based on experiance in libvirt, this is an even larger job than (4),
> as the feature set here is huge.  Much of it directly ties into the
> config problem, as to deal with SELinux / namespace setup the code
> needs to understand what resources to provide access to. This
> requires a way to express 100% coverage of all QEMU configuration
> in use & analyse it to determine what resources it implies. So this
> ties strongly into QAPI-ification completion.

I don't have a clear design in mind yet but I've learnt a little from
writing the virtiofsd namespace isolation and seccomp code.  The task is
really hard for a general-purpose QEMU binary that supports all
features, because configuring the sandbox with the minimal attack
surface requires knowledge of the feature set being used in this QEMU
invocation :(.

> On the libvirt side, we've a strong long term desire to move our
> codebase from C to either Go or Rust. In doing this refactoring
> and rewriting, we would quite likely end up with self-contained
> packages / modules / APIs that can be used independantly of the
> main libvirt API. Thus there's probably scope here to collaborate
> between libvirt & QEMU communities to develop new APIs.
> Some of this code could be directly QEMU related, but other parts
> of the code are likely to be generic & not tied to QEMU at all.
> For example libvirt's SELinux, namespace, cgroup setup code is used
> by LXC too. Thus from the libvirt POV, some libraries might be
> deliverable by QEMU project, but others might be deliverable by
> libvirt project. Or might be scope for a indepent new project to
> deliver pieces.

I think duplication is in the nature of these isolation mechanisms.
systemd also supports most of the features necessary for isolation but
neither QEMU nor libvirt take advantage of systemd and it's not easy to
do so either.  I think it will be necessary to be pragmatic and look at
it on a case-by-case basis.  There will be some duplication.

> Essentially we can consider a stack
>      +--------------+
>      |  Libvirt API |
>      +--------------+
>            |
>          V
>      +--------------+
>      | Helper APIs  |
>      +--------------+
>            |
>          V
>      +--------------+
>      |  QEMU        |
>      +--------------+
> Right now, everything in "Helper APIs" is hidden inside libvirt.
> Some of that conceptually ought to be deliverable by QEMU. Some
> of that ought to be spun out as separate deliverables by libvirt
> or another independant project.

I would like QEMU to directly expose APIs without the need for another
level of APIs, where this makes sense.  It's inefficient and slow to
develop stable APIs in QEMU and then repeat that work again in higher
layers of the stack.  To some extent this is unavoidable but I think
QEMU is close to providing usable APIs and tools itself in the areas
I've outlined in this email thread.

By doing this we'll create a better experience for Kata Containers and
other programs that directly use QEMU.

> > 6. A configuration file format is sorely needed so that guest
> > configuration can be persisted and easily launched.  Today you have to
> > create a shell script that executes a QEMU command-line, but this is
> > suboptimal because sharing executable scripts is dangerous from a
> > security perspective and is hard to parse or modify programmatically.
> A configuration file is just one aspect of our broader configuration
> problem.
> Our CLI in particular suffers from trying to be "all things to all people"
> and IMHO is a direct cause of the perception of QEMU as being overly
> complex, hard to understand & burdened by legacy support. Of course we
> didn't set out intentionally on this path, things just grew organically
> we were not succesfully enough at tackling technical debt over time.
> To pick on blockdev, as it has had the most growth, IIUC we have
>   $QEMU -drive if=ide,file=FILENAME
>   $QEMU -drive if=none,file=FILENAME,id=disk0 -device virtio-blk,drive=disk0
>   $QEMU -blockdev driver=raw,file.filename=FILENAME,id=disk0 -device 
> virtio-blk,drive=disk0
> IIUC, in (all/most) cases "FILENAME" can be either a plain filename,
> or a "json:{....}" string, so that's giving us possibly as many as
> 10 ways to configure disks, with increasing generality.
> If you were designing QEMU from a clean slate it is unlikely we'd
> plan to have 10 syntaxes. A clean slate would probably provide
> 2 different syntaxes at most - one simple but convenient, and one
> complex but fully expressive.
> For every one there's going to be some set of users who think
> their chosen syntax is ideal, and will thus be unhappy if we
> cull one choice.  For the health of QEMU long term though, I
> think we're going to have to accept the need to make some people
> temporarily unhappy in order to become sustaininable over the
> long term.

Agreed :).

> This ultimately all stems from the fact that the CLI design has
> aimed to provide 100% coverage of all QEMU features for mgmt apps,
> while at the same time as providing convenient syntax for humans.
> Except it has actually failed in the former, because there's some
> stuff you can only configured after launching QEMU via QMP commands.
> We can reduce some of the maint burden by mapping "convenient syntax"
> onto the more expressive (usually QAPI modelled) syntax internally,
> but I think that alone is insufficient.
> We should step back & consider the bigger question of what we want
> QEMU's approach to configuration to be.
> With my libvirt hat on, we have to support JSON for a large portion
> of config because we need to use QMP hotplug. Thus if we could eliminate
> the need to use the CLI syntax entirely, it would remove / simplify
> libvirt's code base to work exclusively in terms of JSON.
> With this in mind, I think we could reasonably declare a high level
> goal that
>  - CLI arg syntax is *exclusively* for human targetted convenient syntax
>  - CLI arg syntax will *not* aim for 100% feature coverage
>  - CLI arg syntax will not guarantee strict back compat but will not
>    gratituitously break human usage
>  - JSON syntax will provide 100% feature coverage for mmgmt apps and humans
>  - JSON syntax will not be exposed via ARGV
>  - JSON syntax will be usable via a config file
>  - JSON syntax will be usable via QMP
>  - JSON syntax will provide strict back compat, but with deprecations
>    policy followed
> This is more or less what our long term goal with QAPI has always
> been. The sticking point has always been the tension with CLI arg
> syntax back-compat. We need to decide how to cut the knot so that
> we can complete the QAPI-ification in a forseeable amount of time.

Sounds good.

> > In many of these areas we already have a partial solution.  It just
> > needs more work.  I think it would be worth the effort and the mental
> > shift to really providing APIs that are easy to use by applications.
> > 
> > What do you think?
> Personally I think QEMU configuration is the single biggest pain point
> for dealing with QEMU that we're not investing enough resources in.
> > Have I missed things that are needed?
> > 
> > Have I included things that are unnecessary?
> Many of the things you list above are conceptually nice, but imply a
> lot of extra work for maintainers, as they are expanding the scope of
> the QEMU project. There's a risk this could divert resources away from
> tackling technical debt in things we're already maintaining. The flip
> side of course is that some of the things might attract new contributors
> to the project, because new concepts are inevitably more interesting to
> some people than tackling technical debt. IOW both are important and
> we need to find a balance between them.
> One of the serious long term issues we've had in QEMU is our inability
> to finish jobs that we start. The configuration problem is the biggest
> example or something we started JSON-ifying 10 years ago, but are still
> an unknowable amount of time away from finishing.

I think now is the right time to do it because there is awareness and
willingness to find solutions both in libvirt and QEMU.  I'm optimistic
that progress can be made here.

The next step is a more detailed plan so that multiple people can work
on these tasks.


Attachment: signature.asc
Description: PGP signature

reply via email to

[Prev in Thread] Current Thread [Next in Thread]