Redesign of QEMU startup & initial configuration

[Markus Armbruster]

= Motivation =

QEMU startup and initial configuration were designed many years ago for
a much, much simpler QEMU.  They have since changed beyond recognition
to adapt to new needs.  There was no real redesign.  Adaption to new
needs has become more and more difficult.  A recent example for
"difficult" is Damien's "[RFC PATCH v3 0/5] QMP support for
cold-plugging devices".

I think it's time for a redesign.


= What users want for initial configuration =

1. QMP only

   Management applications need to use QMP for monitoring anyway.  They
   may want to use it for initial configuration, too.  Libvirt does.

   They still need to bootstrap a QMP monitor, and for that, CLI is fine
   as long as it's simple and stable.

2. CLI and configuration files

   Human users want a CLI and configuration files.

   CLI is good for quick tweaks, and to explore.

   For more permanent, non-trivial configuration, configuration files
   are more suitable, because they are easier to read, edit, and
   document than long command lines.


= What we have for initial configuration =

Half of 1. and half of 2., satisfying nobody's needs.

Management applications need to do a lot of non-trivial initial
configuration with the CLI.

Human users struggle with inconsistent syntax, insufficiently expressive
configuration files, and huge command lines.


= Why startup matters for initial configuration =

Startup and initial configuration are joined at the hip: the way startup
works limits how initial configuration can work.

Consider the startup we had before -prefconfig:

0. Basic, configuration-independent initialization

1. Parse and partially process CLI

2. Startup with the remaining CLI processing mixed in

3. Startup complete, run main loop

Note:

* CLI is processed out of order.

  Few people understand the order, and only while starting at the code.
  Pretty much every time we mess with the order, we break something.

* QMP monitors become available only at 3, i.e. after startup is
  complete.  Precludes use of QMP for initial configuration.

The current startup with -preconfig permits a bit of initial
configuration with QMP, at the cost of complicating startup code and
interfaces.  More in "Appendix: Why further incremental change feels
impractical".


= Ways forward =

0. This is too hard, let's go shopping

   I can't really blame people for not wanting to mess with startup.  It
   *is* hard, and there's plenty of other useful stuff to do.

   However, I believe we have to mess with it, i.e. this is not actually
   a way forward.

1. Improve what we have step by step, managing incompatible changes

   We tried.  It has eaten substantial time and energy, it has
   complicated code and interfaces further, and we're still nowhere near
   where we need to be.  Does not feel like a practical way forward to
   me.

   If you don't believe me, read the long version in "Appendix: Why
   further incremental change feels impractical".

2. Start over with a QMP-only QEMU, keep the traditional QEMU forever

   A QMP-only QEMU isn't just a matter of providing an alternate main()
   where we ditch most of the CLI for QMP replacements.  It takes a
   rework of the startup code as "phase transitions" triggered by QMP
   commands.  Similar to how -preconfig triggers some startup work.

   To reduce code duplication, we'll probably want to rework the
   traditional QEMU's startup code some, too.

3. Start over with the aim to replace the traditional QEMU

   I don't want an end game where we have two QEMUs, one with bad CLI
   and limited QMP, and one without real CLI and decent QMP.  I want one
   QEMU with decent CLI and decent QMP.

   I propose that a QEMU with decent CLI and decent QMP isn't really
   harder than a QMP-only QEMU.  CLI is just another transport that
   happens to be one-way, i.e. the QMP commands encoded as CLI options
   can't return a value.  Same for configuration files.  I believe the
   parts that are actually hard are shared with QMP-only: QAPIfying
   initial configuration and reworking the startup code as phase
   transitions.

   When that decent QEMU is ready, we can deprecate and later drop the
   traditional one.


= A design proposal =

0. Basic, configuration-independent initialization

1. Start main loop

2. Feed it CLI left to right.  Each option runs a handler just like each
   QMP command does.

   Options that read a configuration file inject the file into the feed.

   Options that create a monitor create it suspended.

   Options may advance the phase / run state, they may require certain
   phase(s), and semantics may depend on the phase.

3. When we're done with CLI, resume any monitors we created.

   As a convenience, provide an easy way to auto-advance phase / run
   state to "guest runs" right here.  The traditional way would be to
   auto-advance unless the user gives -S.

4. Monitors now feed commands to the main loop.  Commands may advance
   the phase / run state, and they may require certain phase(s), and
   semantics may depend on the phase.

   In particular, command "cont" advances phase / run state to "guest
   runs".

Users can do as much or as little with the CLI as they want.  A
management application might want to set up a QMP monitor and no more.


= A way to get to the proposed design =

Create an alternate QEMU binary with the CLI taken out and shot:

0. Basic, configuration-independent initialization

2. Startup

3. Startup complete, run main loop

Maybe hardcode a few things so we can actually run something without any
configuration.  To be ditched when we can configure again.

Rework 2. Startup as a sequence of state transitions we can later
trigger from commands / options.

Create a quasi-monitor so we can feed QMP commands to the main loop.

Make QMP command "cont" transition to "guest runs".

Feed "cont" to the main loop, and drop 2. Startup.

Create QAPI/CLI infrastructure, so we can define CLI options like QMP
commands.  I have (dusty and likely quite bit-rotten) RFC patches.

Feed CLI options to the main loop, and only then feed "cont".

Create option -S to suppress feeding "cont".

Create / reuse / rework QMP commands and CLI options to create QMP
monitors.  Monitors created by CLI start suspended, and get resumed only
when we're done with the CLI.  This is so monitor commands can't race
with the CLI.

Create CLI option to read a configuration file, and feed its contents to
the main loop.  Recursively.  We'll likely want nicer syntax than CLI or
QMP for configuration files.

At this point, we have all the infrastructure we need.  What's still
missing is filling gaps in QMP and in CLI.

I believe "QMP only" would only be marginally easier.  Basically, it
processes (a minimal) CLI before the main loop, not in it.  We could
leave it unQAPIfied.  I can't quite see offhand how to do configuration
files sanely.

Stretch goals:

* Throw in QAPIfying HMP.

* Dust with syntactic sugar (sparingly, please) to make CLI/HMP easier
  to use for humans.


= Initial exploration, a.k.a. talk's cheap, show me the code =

I'm going to post "[PATCH RFC 00/11] vl: Explore redesign of startup"
shortly.


= Appendix: Why further incremental change feels impractical =

Recall from "Why startup matters for initial configuration" how the
traditional startup code made initial configuration with QMP impossible.

We added -preconfig (v3.0.0 in 2018) to support a bit of it.  If given,
we delay part of the startup until QMP command x-exit-preconfig, like
this:

0. Basic, configuration-independent initialization

1. Parse and partially process CLI left to right

2. Some startup with some of the remaining CLI processing mixed in

3. Run main loop

3.1. Execute QMP commands that are allowed before x-exit-preconfig

3.2. x-exit-preconfig: Remaining startup with the remaining CLI
     processing mixed in

3.3. Startup complete

Note:

* (Out of order) CLI processing is interleaved with (in-order) QMP
  execution.

  One word: ugh!

* QMP becomes available earlier, but still not early enough to support
  "QMP only".

* The initial implementation of -preconfig was a hack.  Getting to the
  current implementation took effort.

  Actually, *any* non-trivial change in this area now takes more effort
  than it should.  See also "pretty much every time [...] we break
  something" above.

We can try to push more and more startup work from 2. to 3 until "QMP
only" becomes possible.

Damien's "[RFC PATCH v3 0/5] QMP support for cold-plugging devices" is a
modest step in this direction.  Since the work to be pushed needs to
move to a different point than x-exit-preconfig, we need another command
x-machine-init to provide a suitable point.

At this point, we complicated startup code and interfaces (and arguably
compromised CLI sanity further), with more of the same needed to get
anywhere near "QMP only".

There is plenty of doubt on the interfaces going around, and that's why
-preconfig is still not a stable interface.

Crafting a big change in small steps sounds great.  It isn't when we
have to make things worse before they can get better, and every step is
painfully slow because it's just too hard, and the end state we aim for
isn't really what we want.