RE: [RFC PATCH 00/13] Add a plugin to support out-of-band live migration for VFIO pass-through device

[Dong, Eddie]

> -----Original Message-----
> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Wednesday, June 1, 2022 11:01 AM
> To: Dong, Eddie <eddie.dong@intel.com>
> Cc: Rao, Lei <lei.rao@intel.com>; Tian, Kevin <kevin.tian@intel.com>; Zeng,
> Jason <jason.zeng@intel.com>; quintela@redhat.com; dgilbert@redhat.com;
> Li, Yadong <yadong.li@intel.com>; Liu, Yi L <yi.l.liu@intel.com>; qemu-
> devel@nongnu.org
> Subject: Re: [RFC PATCH 00/13] Add a plugin to support out-of-band live
> migration for VFIO pass-through device
> 
> On Wed, 1 Jun 2022 17:09:25 +0000
> "Dong, Eddie" <eddie.dong@intel.com> wrote:
> 
> > > -----Original Message-----
> > > From: Qemu-devel <qemu-devel-
> > > bounces+eddie.dong=intel.com@nongnu.org> On Behalf Of Alex
> > > bounces+Williamson
> > > On Tue, 24 May 2022 14:18:35 +0800
> > > Lei Rao <lei.rao@intel.com> wrote:
> > > > This proposal adopts a plugin mechanism (an example can be found
> > > > in
> > > > [1]) given that IPU/DPU vendors usually implement proprietary
> > > > migration interfaces without a standard. But we are also open if
> > > > an alternative option makes better sense, e.g. via loadable
> > > > modules (with Qemu supporting gRPC or JSON-RPC support) or an IPC
> > > > mechanism similar
> > > to vhost-user.
> > >
> > > AFAIU, QEMU is not interested in supporting plugin modules,
> > > especially proprietary ones.  I don't see that a case has really
> > > been made that this cannot be done in-band, through a vfio-pci
> > > variant driver, possibly making use of proprietary interfaces to a
> > > userspace agent if necessary (though please don't ask such to be
> > > accepted in-tree for the kernel either).  A vfio- user device server
> > > might also host such proprietary, device specific agents while
> > > supporting the standard, in-band migration interface.  Thanks,
> > >
> >
> > Thanks Alex. Removing plug-in module is not a problem.
> >
> > Do you mean to implement the migration and protocol handling inside
> > Qemu-client (probably vfio-client after the VFIO-user is merged)? Or
> > to build as part of libvfio-user? We can also build it as a separate
> > process of Qemu-server as part of Multi-Process Qemu.
> 
> AIUI, the QEMU "client" in a vfio-user configuration is simply QEMU itself.
> The vfio-user "server" provides the actual device implementation, which
> could support different license models, depending on what libraries or
> existing code is incorporated to implement that server.  The QEMU remote
> machine type is simply a QEMU-based implementation of a vfio-user server.
> The vfio-user server is analogous to a vfio-pci variant driver in the
> kernel/ioctl interface model.  The vfio-user client should be device agnostic,
> possibly with similar exceptions we have today via device specific quirk
> handling for the vfio kernel interface.
> 
> > In here, the protocol between host CPU and SoC is based on gRPC, which
> > support Rust code in client (Host CPU side here) more than C/C++. Do
> > you have any suggestion to better support Rust code with Qemu C/C++
> > code?
> 
> I'm not qualified to provide suggestions regarding Rust code integration with
> QEMU, but I think that's only required if the device specific migration
> support is on the "client".  As above, I don't think that's the correct model,
> the vfio migration protocol is meant to support any device specific
> requirements on the device end of the connection, ie. the "server" end for
> vfio-user, which can be an entirely separate, non-QEMU based process.  I
> think there are also ways to write kernel drivers in Rust, so possibly a 
> kernel
> interface vfio-pci variant driver could also work.  Thanks,
> 


Alex:
        Thanks for your suggestion. Yes, agree Qemu (client) is, by nature, 
neutral to physical device knowledge.
        With more thinking, it seems that:
        1: Solution to have a separate kernel driver:   
                The way the host CPU talking with the SoC chip of the device is 
going thru TCP/IP network, plus high level protocol (gRPC or Json..). This is 
going to be very complicated and might be hard to be accepted by community.

        2: Implement as a full qemu-server device model.
                This way works if we implement a full device model in 
vfio-user, but given that the device (NVME for now) works in VFIO passthru mode 
for performance, the issue Kevin Tian raised in another email is a real concern 
too.

        3: Implement partial (or supplemental) feature in Qemu-server device 
model.
                This solution defines a Qemu/VFIO migration interface between 
the client and server for migration.  Client migration-proxy uses hardware 
transparent interface to talk with the remote-migration server. The remote 
server may manage device-specific (protocol specific to be more precisely) 
components to talk with different hardware backend. In this case, we rely on 
today's VFIO module to manage the device and manipulate the device thru kernel 
driver. During migration, it will use the migration-proxy to get/set state 
etc...   
                Users can configure the additional Qemu command line parameter 
to choose a remote migration-proxy for a VFIO device. 


        Can you suggest?

Thx Eddie