[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[RFC PATCH 00/13] Add a plugin to support out-of-band live migration for
From: |
Lei Rao |
Subject: |
[RFC PATCH 00/13] Add a plugin to support out-of-band live migration for VFIO pass-through device |
Date: |
Tue, 24 May 2022 14:18:35 +0800 |
Migration of a VFIO passthrough device can be supported by using a device
specific kernel driver to save/restore the device state thru device specific
interfaces. But this approach doesn't work for devices that lack a state
migration interface, e.g. NVMe.
On the other hand, Infrastructure Process Unit (IPU) or Data Processing Unit
(DPU) vendors may choose to implement an out-of-band interface from the SoC to
help manage the state of such non-migratable devices e.g. via gRPC or JSON-RPC
protocols.
This RFC attempts to support such out-of-band migration interface by introducing
the concept of migration backends in vfio. The existing logic around vfio
migration uAPI is now called the 'local' backend while a new 'out-of-band'
backend is further introduced allowing vfio to redirect VMState ops to an
external plugin.
Currently, the backend migration Ops is defined close to SaveVMHandlers. We also
considered whether there is value of abstracting it in a lower level e.g. close
to vfio migration uAPI but no clear conclusion. Hence this is one part which
we'd like to hear suggestions.
This proposal adopts a plugin mechanism (an example can be found in [1]) given
that IPU/DPU vendors usually implement proprietary migration interfaces without
a standard. But we are also open if an alternative option makes better sense,
e.g. via loadable modules (with Qemu supporting gRPC or JSON-RPC support) or an
IPC mechanism similar to vhost-user.
The following graph describes the overall component relationship:
+----------------------------------------------------+
| QEMU |
| +------------------------------------------------+ |
| | VFIO Live Migration Framework | |
| | +--------------------------------------+ | |
| | | VFIOMigrationOps | | |
| | +-------^---------------------^--------+ | |
| | | | | |
| | +-------v-------+ +-------v--------+ | |
| | | LM Backend Via| | LM Backend Via | | |
| | | Device Fd | | Plugins | | |
| | +-------^-------+ | +----------+ | |
| | | | |Plugin Ops+----+-+------------+
| | | +-----+----------+ | | |
| | | | | +---------v----------+
| +------------+-----------------------------------+ | | Vendor Specific |
| | | | Plugins(.so) |
+--------------+-------------------------------------+ +----------+---------+
UserSpace | |
----------------+--------------------------------------------- |
Kernel | |
| |
+----------v----------------------+ |
| Kernel VFIO Driver | |
| +-------------------------+ | |
| | | | | Network
| | Vendor-Specific Driver | | |
| | | | |
| +----------^--------------+ | |
| | | |
+---------------+-----------------+ |
| |
| |
---------------------+----------------------------------------- |
Hardware | |
| +-----+-----+-----+----+-----+ |
+----------v------+ | VF0 | VF1 | VF2 | ...| VFn | |
| Traditional | +-----+-----+-----+----+-----+ |
| PCIe Devices | | | |
+-----------------+ | +--------+------------+ | |
| | | Agent |<-+----+
| | +------------+ |
| | | |
| | SOC | |
| +---------------------+ |
| IPU |
+----------------------------+
Two command-line parameters (x-plugin-path and x-plugin-arg) are introduced to
enable the out-of-band backend. If specified, vfio will attempt to use the
out-of-band backend.
The following is an example of VFIO command-line parameters for OOB-Approach:
-device
vfio-pci,id=$ID,host=$bdf,x-enable-migration,x-plugin-path=$plugin_path,x-plugin-arg=$plugin_arg
[1] https://github.com/raolei-intel/vfio-lm-plugin-example.git
Lei Rao (13):
vfio/migration: put together checks of migration initialization
conditions
vfio/migration: move migration struct allocation out of
vfio_migration_init
vfio/migration: move vfio_get_dev_region_info out of
vfio_migration_probe
vfio/migration: Separated functions that relate to the In-Band
approach
vfio/migration: rename functions that relate to the In-Band approach
vfio/migration: introduce VFIOMigrationOps layer in VFIO live
migration framework
vfio/migration: move the statistics of bytes_transferred to generic
VFIO migration layer
vfio/migration: split migration handler registering from
vfio_migration_init
vfio/migration: move the functions of In-Band approach to a new file
vfio/pci: introduce command-line parameters to specify migration
method
vfio/migration: add a plugin layer to support out-of-band live
migration
vfio/migration: add some trace-events for vfio migration plugin
vfio/migration: make the region and plugin member of struct
VFIOMigration to be a union
docs/devel/vfio-migration-plugin.rst | 165 +++++++
hw/vfio/meson.build | 2 +
hw/vfio/migration-local.c | 456 +++++++++++++++++++
hw/vfio/migration-plugin.c | 266 +++++++++++
hw/vfio/migration.c | 577 ++++++------------------
hw/vfio/pci.c | 2 +
hw/vfio/trace-events | 9 +-
include/hw/vfio/vfio-common.h | 37 +-
include/hw/vfio/vfio-migration-plugin.h | 21 +
9 files changed, 1096 insertions(+), 439 deletions(-)
create mode 100644 docs/devel/vfio-migration-plugin.rst
create mode 100644 hw/vfio/migration-local.c
create mode 100644 hw/vfio/migration-plugin.c
create mode 100644 include/hw/vfio/vfio-migration-plugin.h
--
2.32.0
- [RFC PATCH 00/13] Add a plugin to support out-of-band live migration for VFIO pass-through device,
Lei Rao <=
- [RFC PATCH 03/13] vfio/migration: move vfio_get_dev_region_info out of vfio_migration_probe, Lei Rao, 2022/05/24
- [RFC PATCH 02/13] vfio/migration: move migration struct allocation out of vfio_migration_init, Lei Rao, 2022/05/24
- [RFC PATCH 04/13] vfio/migration: Separated functions that relate to the In-Band approach, Lei Rao, 2022/05/24
- [RFC PATCH 01/13] vfio/migration: put together checks of migration initialization conditions, Lei Rao, 2022/05/24
- [RFC PATCH 05/13] vfio/migration: rename functions that relate to the In-Band approach, Lei Rao, 2022/05/24
- [RFC PATCH 06/13] vfio/migration: introduce VFIOMigrationOps layer in VFIO live migration framework, Lei Rao, 2022/05/24
- [RFC PATCH 07/13] vfio/migration: move the statistics of bytes_transferred to generic VFIO migration layer, Lei Rao, 2022/05/24
- [RFC PATCH 08/13] vfio/migration: split migration handler registering from vfio_migration_init, Lei Rao, 2022/05/24
- [RFC PATCH 09/13] vfio/migration: move the functions of In-Band approach to a new file, Lei Rao, 2022/05/24
- [RFC PATCH 12/13] vfio/migration: add some trace-events for vfio migration plugin, Lei Rao, 2022/05/24