Re: Out-of-Process Device Emulation session at KVM Forum 2020

[Stefan Hajnoczi]

On Mon, Nov 02, 2020 at 11:00:12AM +0800, Jason Wang wrote:
> 
> On 2020/10/30 下午7:13, Stefan Hajnoczi wrote:
> > > I still don't get why it must be opaque.
> > If the device state format needs to be in the VMM then each device
> > needs explicit enablement in each VMM (QEMU, cloud-hypervisor, etc).
> > 
> > Let's invert the question: why does the VMM need to understand the
> > device state of a_passthrough_  device?
> 
> 
> It's not a 100% passthrough device if you want to support live migration.
> E.g the device state save and restore is not under the control of drivers in
> the guest.

VFIO devices are already not pure passthrough (even without mdev) since
the PCI bus is emulated and device-specific quirks may be implemented.
Adding device state save/load does not change anything here.

> And if I understand correctly, it usually requires device emulation or
> mediation in either userspace or kernel to support e.g dirty page tracking
> and other things.

Breaking down the options further:

1. VFIO on physical PCI devices. Here I see two approaches:

   a. An mdev vendor driver implements the migration region described in
      Kirti's patch series. Individual device state fields are
      marshalled by the driver.

   b. The VFIO PCI core parses a PCI Capability that indicates migration
      support on the physical device and filters it out. The remainder
      of the device is passed through. The device state representation
      is saved/loaded by the physical hardware when the VFIO PCI core
      receives the ioctl and notifies the hardware. QEMU and host
      kernel code does not marshall individual device state fields.

      In the future it may be desirable to also expose the PCI
      Capability so that the guest is able to snapshot and restore the
      device. That could be useful for checkpointing AI/HPC workloads on
      GPUs, for example. I don't think there is a fundamental reason why
      device state save/load needs to be handled by the host except that
      VM live migration is supposed to be transparent to guests and
      cannot rely on guest cooperation.

2. vfio-user device backends. The device backend implements the
   save/load.

signature.asc
Description: PGP signature