[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [RFC PATCH 00/13] Add a plugin to support out-of-band live migration
|
From: |
Dong, Eddie |
|
Subject: |
RE: [RFC PATCH 00/13] Add a plugin to support out-of-band live migration for VFIO pass-through device |
|
Date: |
Wed, 1 Jun 2022 17:09:25 +0000 |
> -----Original Message-----
> From: Qemu-devel <qemu-devel-
> bounces+eddie.dong=intel.com@nongnu.org> On Behalf Of Alex Williamson
> Sent: Thursday, May 26, 2022 11:44 AM
> To: Rao, Lei <Lei.Rao@intel.com>
> Cc: Tian, Kevin <kevin.tian@intel.com>; Dong, Eddie
> <eddie.dong@intel.com>; Zeng, Jason <jason.zeng@intel.com>;
> quintela@redhat.com; dgilbert@redhat.com; Li, Yadong
> <yadong.li@intel.com>; Liu, Yi L <yi.l.liu@intel.com>; qemu-
> devel@nongnu.org
> Subject: Re: [RFC PATCH 00/13] Add a plugin to support out-of-band live
> migration for VFIO pass-through device
>
> On Tue, 24 May 2022 14:18:35 +0800
> Lei Rao <lei.rao@intel.com> wrote:
>
> > Migration of a VFIO passthrough device can be supported by using a
> > device specific kernel driver to save/restore the device state thru
> > device specific interfaces. But this approach doesn't work for devices
> > that lack a state migration interface, e.g. NVMe.
> >
> > On the other hand, Infrastructure Process Unit (IPU) or Data
> > Processing Unit
> > (DPU) vendors may choose to implement an out-of-band interface from
> > the SoC to help manage the state of such non-migratable devices e.g.
> > via gRPC or JSON-RPC protocols.
> >
> > This RFC attempts to support such out-of-band migration interface by
> > introducing the concept of migration backends in vfio. The existing
> > logic around vfio migration uAPI is now called the 'local' backend while a
> new 'out-of-band'
> > backend is further introduced allowing vfio to redirect VMState ops to
> > an external plugin.
> >
> > Currently, the backend migration Ops is defined close to
> > SaveVMHandlers. We also considered whether there is value of
> > abstracting it in a lower level e.g. close to vfio migration uAPI but
> > no clear conclusion. Hence this is one part which we'd like to hear
> suggestions.
> >
> > This proposal adopts a plugin mechanism (an example can be found in
> > [1]) given that IPU/DPU vendors usually implement proprietary
> > migration interfaces without a standard. But we are also open if an
> > alternative option makes better sense, e.g. via loadable modules (with
> > Qemu supporting gRPC or JSON-RPC support) or an IPC mechanism similar
> to vhost-user.
>
> AFAIU, QEMU is not interested in supporting plugin modules, especially
> proprietary ones. I don't see that a case has really been made that this
> cannot be done in-band, through a vfio-pci variant driver, possibly making
> use of proprietary interfaces to a userspace agent if necessary (though
> please don't ask such to be accepted in-tree for the kernel either). A vfio-
> user device server might also host such proprietary, device specific agents
> while supporting the standard, in-band migration interface. Thanks,
>
Thanks Alex. Removing plug-in module is not a problem.
Do you mean to implement the migration and protocol handling inside Qemu-client
(probably vfio-client after the VFIO-user is merged)? Or to build as part of
libvfio-user? We can also build it as a separate process of Qemu-server as part
of Multi-Process Qemu.
In here, the protocol between host CPU and SoC is based on gRPC, which support
Rust code in client (Host CPU side here) more than C/C++. Do you have any
suggestion to better support Rust code with Qemu C/C++ code?
Thx Eddie
> Alex
>
> >
> > The following graph describes the overall component relationship:
> >
> > +----------------------------------------------------+
> > | QEMU |
> > | +------------------------------------------------+ |
> > | | VFIO Live Migration Framework | |
> > | | +--------------------------------------+ | |
> > | | | VFIOMigrationOps | | |
> > | | +-------^---------------------^--------+ | |
> > | | | | | |
> > | | +-------v-------+ +-------v--------+ | |
> > | | | LM Backend Via| | LM Backend Via | | |
> > | | | Device Fd | | Plugins | | |
> > | | +-------^-------+ | +----------+ | |
> > | | | | |Plugin Ops+----+-+------------+
> > | | | +-----+----------+ | | |
> > | | | | |
> > +---------v----------+
> > | +------------+-----------------------------------+ | | Vendor Specific
> > |
> > | | | | Plugins(.so)
> > |
> > +--------------+-------------------------------------+
> > +----------+---------+
> > UserSpace | |
> > ----------------+--------------------------------------------- |
> > Kernel | |
> > | |
> > +----------v----------------------+ |
> > | Kernel VFIO Driver | |
> > | +-------------------------+ | |
> > | | | | |
> > Network
> > | | Vendor-Specific Driver | | |
> > | | | | |
> > | +----------^--------------+ | |
> > | | | |
> > +---------------+-----------------+ |
> > | |
> > | |
> > ---------------------+----------------------------------------- |
> > Hardware | |
> > | +-----+-----+-----+----+-----+ |
> > +----------v------+ | VF0 | VF1 | VF2 | ...| VFn | |
> > | Traditional | +-----+-----+-----+----+-----+ |
> > | PCIe Devices | | | |
> > +-----------------+ | +--------+------------+ | |
> > | | | Agent |<-+----+
> > | | +------------+ |
> > | | | |
> > | | SOC | |
> > | +---------------------+ |
> > | IPU |
> > +----------------------------+
> >
> > Two command-line parameters (x-plugin-path and x-plugin-arg) are
> > introduced to enable the out-of-band backend. If specified, vfio will
> > attempt to use the out-of-band backend.
> >
> > The following is an example of VFIO command-line parameters for OOB-
> Approach:
> >
> > -device
> > vfio-pci,id=$ID,host=$bdf,x-enable-migration,x-plugin-path=$plugin_pat
> > h,x-plugin-arg=$plugin_arg
> >
> > [1] https://github.com/raolei-intel/vfio-lm-plugin-example.git
> >
> > Lei Rao (13):
> > vfio/migration: put together checks of migration initialization
> > conditions
> > vfio/migration: move migration struct allocation out of
> > vfio_migration_init
> > vfio/migration: move vfio_get_dev_region_info out of
> > vfio_migration_probe
> > vfio/migration: Separated functions that relate to the In-Band
> > approach
> > vfio/migration: rename functions that relate to the In-Band approach
> > vfio/migration: introduce VFIOMigrationOps layer in VFIO live
> > migration framework
> > vfio/migration: move the statistics of bytes_transferred to generic
> > VFIO migration layer
> > vfio/migration: split migration handler registering from
> > vfio_migration_init
> > vfio/migration: move the functions of In-Band approach to a new file
> > vfio/pci: introduce command-line parameters to specify migration
> > method
> > vfio/migration: add a plugin layer to support out-of-band live
> > migration
> > vfio/migration: add some trace-events for vfio migration plugin
> > vfio/migration: make the region and plugin member of struct
> > VFIOMigration to be a union
> >
> > docs/devel/vfio-migration-plugin.rst | 165 +++++++
> > hw/vfio/meson.build | 2 +
> > hw/vfio/migration-local.c | 456 +++++++++++++++++++
> > hw/vfio/migration-plugin.c | 266 +++++++++++
> > hw/vfio/migration.c | 577 ++++++------------------
> > hw/vfio/pci.c | 2 +
> > hw/vfio/trace-events | 9 +-
> > include/hw/vfio/vfio-common.h | 37 +-
> > include/hw/vfio/vfio-migration-plugin.h | 21 +
> > 9 files changed, 1096 insertions(+), 439 deletions(-) create mode
> > 100644 docs/devel/vfio-migration-plugin.rst
> > create mode 100644 hw/vfio/migration-local.c create mode 100644
> > hw/vfio/migration-plugin.c create mode 100644
> > include/hw/vfio/vfio-migration-plugin.h
> >
>
- RE: [RFC PATCH 00/13] Add a plugin to support out-of-band live migration for VFIO pass-through device,
Dong, Eddie <=