[Qemu-devel] [PATCH v6 00/11] enable numa configuration before machine_init() from QMP

[Igor Mammedov]

v5->v6:
  * add exit-preconfig QMP command instead of overloading meaning of 'cont' 
command
  * add doc text to qemu-tech.texi about -S and --preconfig
  * add numa configuration example into commit message of 10/11
  * limit set-numa-node QMP command to preconfig mode
v4->v5:
  * rebase on top of OOB changes that's now in master
  * s/qobject_to_qdict(/qobject_to(QDict,/
  * s/-preconfig/--preconfig/
  * s/2.12/2.13/
  * s/parse_NumaOptions/set_numa_options/
  * drop if (err) guard around error_propagate()
  * move QCO_ALLOWED_IN_PRECONFIG and do_qmp_dispatch() runstate check
    from the later patch 'qapi: introduce new cmd option  
"allowed-in-preconfig"'
    to here for better bissectability
  * add TODO comment to '{ RUN_STATE_PRECONFIG, RUN_STATE_INMIGRATE }' 
transition
  * check for incoming && preconfig outside of option parsing loop
  * add 'use QMP instead" to error message, suggesting user the right interface 
to use
  * allow query-command-line-options in preconfig state
  * make sure that allowed-in-preconfig could be set only to True
  * move out QCO_ALLOWED_IN_PRECONFIG check in do_qmp_dispatch() to
    earlier 'cli: add --preconfig option' patch
  * 2 extra test patches
     'tests: let qapi-schema tests detect allowed-in-preconfig'
     'tests: add allowed-in-preconfig-test for qapi-schema'
v3->v4:
  * replace 'runstates' list in  QMP command with a single
    boolean 'ption allowed-in-preconfig' like it's done with
    'allow-oob'. Which allows to simplify intrusive QAPI
    changes quite a lot. (Eric Blake <address@hidden>)
  * Make sure HMP is disbled for real, v3 was just printing
    error mesage but allowing command to be executed
    ("Dr. David Alan Gilbert" <address@hidden>)
  * Improve "cli: add -preconfig option" commit message,
    explain a bit more on semantics of new state/option.
  * ithe rest of minor fixups suggested at v3 review
    (Eric Blake <address@hidden>)
    PS:
       havn't impl. test for new option in
         tests/qapi-schema/qapi-schema-test.json yet,
       can do it on top if approach is acceptable.
v1->v3:
  * introduce PRECONFIG runstate with -preconfig option.
    it's cleaner to manage transitions and do checks
    than reusing existing PRELAUNCH state.
  * extend QAPI schema commands with 'runstates' keyword,
    so that it would be possible to specify in command definition
    when it is valid to execute.
    (python changes a bit hackery, since I have little to
     no idea how it should work)
  * add preconfig QMP and set-numa-node tests
  * make mutually exclusive -preconfig and -incoming options,
    for simplicity sake. Shouldn't be problem as target can
    be starter with pure CLI, since mapping on source is
    already known.
  * Drop HMP part and leave only QMP in preconfig state.


Currently it's problematic to configure NUMA mapping for CPUs using "-numa cpu="
option, without restarting QEMU, first is to query board specific CPU layout
with query-hotpluggable-cpus QMP command (which is function of -M and -smp CLI
options). Mgmt side isn't happy with restarting QEMU to query configuration
parameters first and then run actual instance, so it keeps using old cpu_index
based option '-numa cpus' instead of new socket/core/thread-id based one.

Introduce a new '--preconfig' CLI option which allows pausing QEMU before
machine_init() is run (preconfig state). So it would be possible to query
possible CPUs layout, configure NUMA mapping based on query result with
set-numa-node QMP command and continue executing  without restarting QEMU.

Difference between new --preconfig pause point vs -S is that the later pauses
QEMU after machine is constructed and ready to run guest code or in process of
incoming migration (essentially machine is in some running state (with paused
VCPUs) and any action on it is considered as hotplug). At this point it's hard
to configure or reconfigure parameters that affect machine_init() and later
stages. While the new --preconfig option pauses QEMU instance before
machine_init() and would allow to configure parameters as if doing it from CLI
but in interactve manner using QMP interface, which would allow introspecting
and configuring QEMU instance without restarting it. 

Initially only limited set of commands (that are ready to work with non
initialized machine or without it) are allowed in preconfig state:

   #existing commands:
       qmp_capabilities
       query-qmp-schema
       query-commands
       query-command-line-options
       query-status
       query-hotpluggable-cpus
   #new commands
       exit-preconfig
       set-numa-node

Which commands are allowed at preconfig state, is defined by QAPI Schema
with help of optional 'allowed-in-preconfig' flag in command definition,
which allows users to get list of commands it can run with help of
query-qmp-schema command.

Once user done with configuration at preconfig state, it could be exited with
exit-preconfig command, in which case QEMU would proceed with VM intialization
and either start guest execution or pause in prelaunch state if -S CLI option
were specified along with --prelaunch on startup.

PS:
Later we can modify other commands to run early, for example device_add. So
that devices would be added in cold-plug context, instead of getting device
'hotplugged' with followed up fixups and workarounds to make it look like
cold-plugged.

Example of configuration CPU's NUMA mapping with --preconfig:
$QEMU -smp 2 --preconfig ...

QMP:
# get CPUs layout for current target/machine/CLI
-> {'execute': 'query-hotpluggable-cpus' }      
<- {'return': [
       {'props': {'core-id': 0, 'thread-id': 0, 'socket-id': 1}, ... },
       {'props': {'core-id': 0, 'thread-id': 0, 'socket-id': 0}, ... }
   ]}

# configure 1st node
-> {'execute': 'set-numa-node', 'arguments': { 'type': 'node', 'nodeid': 0 } }  
    
<- {'return': {}}
-> {'execute': 'set-numa-node', 'arguments': { 'type': 'cpu',       
       'node-id': 0, 'core-id': 0, 'thread-id': 0, 'socket-id': 1, }
   }
<- {'return': {}}

# configure 2nd node
-> {'execute': 'set-numa-node', 'arguments': { 'type': 'node', 'nodeid': 1 } }
-> {'execute': 'set-numa-node', 'arguments': { 'type': 'cpu',      
       'node-id': 1, 'core-id': 0, 'thread-id': 0, 'socket-id': 0 }
   }
<- {'return': {}}

# [optional] verify configuration
-> {'execute': 'query-hotpluggable-cpus' }      
<- {'return': [
       {'props': {'core-id': 0, 'thread-id': 0, 'node-id': 0, 'socket-id': 1}, 
... },
       {'props': {'core-id': 0, 'thread-id': 0, 'node-id': 1, 'socket-id': 0}, 
... }
   ]}


Git tree:
    https://github.com/imammedo/qemu.git qmp_preconfig_v6

 
CC: address@hidden
CC: address@hidden
CC: address@hidden
CC: address@hidden

Igor Mammedov (11):
  numa: postpone options post-processing till machine_run_board_init()
  numa: split out NumaOptions parsing into set_numa_options()
  cli: add --preconfig option
  hmp: disable monitor in preconfig state
  qapi: introduce new cmd option "allowed-in-preconfig"
  tests: let qapi-schema tests detect allowed-in-preconfig
  tests: add allowed-in-preconfig-test for qapi-schema
  tests: extend qmp test with preconfig checks
  qmp: permit query-hotpluggable-cpus in preconfig state
  qmp: add set-numa-node command
  tests: functional tests for QMP command set-numa-node

 include/qapi/qmp/dispatch.h                      |  1 +
 include/sysemu/numa.h                            |  2 +
 include/sysemu/sysemu.h                          |  1 +
 tests/libqtest.h                                 |  9 +++
 docs/devel/qapi-code-gen.txt                     | 10 +++-
 hw/core/machine.c                                |  5 +-
 monitor.c                                        | 11 +++-
 numa.c                                           | 70 ++++++++++++++++--------
 qapi/introspect.json                             |  5 +-
 qapi/misc.json                                   | 48 ++++++++++++++--
 qapi/qmp-dispatch.c                              |  8 +++
 qapi/run-state.json                              |  8 ++-
 qemu-options.hx                                  | 13 +++++
 qemu-tech.texi                                   | 40 ++++++++++++++
 qmp.c                                            | 10 ++++
 scripts/qapi/commands.py                         | 12 ++--
 scripts/qapi/common.py                           | 19 ++++---
 scripts/qapi/doc.py                              |  4 +-
 scripts/qapi/introspect.py                       |  7 ++-
 tests/Makefile.include                           |  1 +
 tests/libqtest.c                                 |  7 +++
 tests/numa-test.c                                | 61 +++++++++++++++++++++
 tests/qapi-schema/allowed-in-preconfig-test.err  |  1 +
 tests/qapi-schema/allowed-in-preconfig-test.exit |  1 +
 tests/qapi-schema/allowed-in-preconfig-test.json |  2 +
 tests/qapi-schema/allowed-in-preconfig-test.out  |  0
 tests/qapi-schema/doc-good.out                   |  4 +-
 tests/qapi-schema/ident-with-escape.out          |  2 +-
 tests/qapi-schema/indented-expr.out              |  4 +-
 tests/qapi-schema/qapi-schema-test.json          |  3 +
 tests/qapi-schema/qapi-schema-test.out           | 22 ++++----
 tests/qapi-schema/test-qapi.py                   |  8 +--
 tests/qmp-test.c                                 | 37 +++++++++++++
 tests/test-qmp-cmds.c                            |  4 ++
 vl.c                                             | 35 +++++++++++-
 35 files changed, 403 insertions(+), 72 deletions(-)
 create mode 100644 tests/qapi-schema/allowed-in-preconfig-test.err
 create mode 100644 tests/qapi-schema/allowed-in-preconfig-test.exit
 create mode 100644 tests/qapi-schema/allowed-in-preconfig-test.json
 create mode 100644 tests/qapi-schema/allowed-in-preconfig-test.out

-- 
2.7.4