Re: [PATCH v2 00/11] vfio/migration: Implement VFIO migration protocol v2

[Avihai Horon]

On 6/8/2022 12:32 AM, Alex Williamson wrote:
> External email: Use caution opening links or attachments
>
>
> On Tue, 7 Jun 2022 20:44:23 +0300
> Avihai Horon <avihaih@nvidia.com> wrote:
>
> > On 5/30/2022 8:07 PM, Avihai Horon wrote:
> > > Hello,
> > >
> > > Following VFIO migration protocol v2 acceptance in kernel, this series
> > > implements VFIO migration according to the new v2 protocol and replaces
> > > the now deprecated v1 implementation.
> > >
> > > The main differences between v1 and v2 migration protocols are:
> > > 1. VFIO device state is represented as a finite state machine instead of
> > >      a bitmap.
> > >
> > > 2. The migration interface with kernel is done using VFIO_DEVICE_FEATURE
> > >      ioctl and normal read() and write() instead of the migration region
> > >      used in v1.
> > >
> > > 3. Migration protocol v2 currently doesn't support the pre-copy phase of
> > >      migration.
> > >
> > > Full description of the v2 protocol and the differences from v1 can be
> > > found here [1].
> > >
> > > Patches 1-3 are prep patches fixing bugs and adding QEMUFile function
> > > that will be used later.
> > >
> > > Patches 4-6 refactor v1 protocol code to make it easier to add v2
> > > protocol.
> > >
> > > Patches 7-11 implement v2 protocol and remove v1 protocol.
> > >
> > > Thanks.
> > >
> > > [1]
> > > https://lore.kernel.org/all/20220224142024.147653-10-yishaih@nvidia.com/
> > >
> > > Changes from v1: 
> > > https://lore.kernel.org/all/20220512154320.19697-1-avihaih@nvidia.com/
> > > - Split the big patch that replaced v1 with v2 into several patches as
> > >     suggested by Joao, to make review easier.
> > > - Change warn_report to warn_report_once when container doesn't support
> > >     dirty tracking.
> > > - Add Reviewed-by tag.
> > >
> > > Avihai Horon (11):
> > >     vfio/migration: Fix NULL pointer dereference bug
> > >     vfio/migration: Skip pre-copy if dirty page tracking is not supported
> > >     migration/qemu-file: Add qemu_file_get_to_fd()
> > >     vfio/common: Change vfio_devices_all_running_and_saving() logic to
> > >       equivalent one
> > >     vfio/migration: Move migration v1 logic to vfio_migration_init()
> > >     vfio/migration: Rename functions/structs related to v1 protocol
> > >     vfio/migration: Implement VFIO migration protocol v2
> > >     vfio/migration: Remove VFIO migration protocol v1
> > >     vfio/migration: Reset device if setting recover state fails
> > >     vfio: Alphabetize migration section of VFIO trace-events file
> > >     docs/devel: Align vfio-migration docs to VFIO migration v2
> > >
> > >    docs/devel/vfio-migration.rst |  77 ++--
> > >    hw/vfio/common.c              |  21 +-
> > >    hw/vfio/migration.c           | 640 ++++++++--------------------------
> > >    hw/vfio/trace-events          |  25 +-
> > >    include/hw/vfio/vfio-common.h |   8 +-
> > >    migration/migration.c         |   5 +
> > >    migration/migration.h         |   3 +
> > >    migration/qemu-file.c         |  34 ++
> > >    migration/qemu-file.h         |   1 +
> > >    9 files changed, 252 insertions(+), 562 deletions(-)
> > Ping.
> Based on the changelog, this seems like a mostly cosmetic spin and I
> don't see that all of the discussion threads from v1 were resolved to
> everyone's satisfaction.  I'm certainly still uncomfortable with the
> pre-copy behavior and I thought there were still some action items to
> figure out whether an SLA is present and vet the solution with
> management tools.  Thanks,

Yes.
OK, so let's clear things up and reach an agreement before I prepare the 
v3 series.

There are three topics that came up in previous discussion:

1. [PATCH v2 01/11] vfio/migration: Fix NULL pointer dereference bug.
   Juan gave his Reviewed-by but he wasn't sure about qemu_file_* usage
   outside migration thread.
   This code existed before and I fixed a NULL pointer dereference that
   I encountered.
   I suggested that later we can refactor VMChangeStateHandler to
   return error.
   I prefer not to do this refactor right now because I am not sure
   it's as straightforward change as it might seem - if some notifier
   fails and we abort do_vm_stop/vm_prepare_start in the middle, can
   this leave the VM in some unstable state?
   We plan to leave it as is and not do the refactor as part of this
   series.
   Are you ok with this?

2. [PATCH v2 02/11] vfio/migration: Skip pre-copy if dirty page
   tracking is not supported.
   As previously discussed, this patch doesn't consider the configured
   downtime limit.
   One way to fix it is to allow such migration only when "no SLA" (no
   downtime limit) is set. AFAIK today there is no way that one can set
   "no SLA".
   If we go with this option, we change normal flow of migration
   (skipping pre-copy) and might need to change management tools.

Instead, what about letting QEMU VFIO code mark all pages dirty (instead 
of kernel)?
This way we don’t skip pre-copy and we get the same behavior we have now 
of perpetual dirtying all RAM, which respects SLA.
If we go with this option, do we need to block migration when IOMMU is 
sPAPR TCE?
Until now migration would be blocked because sPAPR TCE doesn't report 
dirty_pages_supported cap, but going with this option we will allow 
migration even when dirty_pages_supported cap is not set (and let QEMU 
dirty all pages).

3. [PATCH v2 03/11] migration/qemu-file: Add qemu_file_get_to_fd().
   Juan expressed his concern about the amount of data that will go
   through main migration thread.

This is already the case in v1 protocol - VFIO devices send all their 
data in the main migration thread. Note that like in v1 protocol, here 
as well the data is sent in small sized chunks, each with a header.
This patch just aims to eliminate an extra copy.

We plan to leave it as is. Is this ok?

Thanks.