Re: [PATCH 01/18] vfio/migration: Add VFIO migration pre-copy support

[Alex Williamson]

On Wed, 1 Feb 2023 13:28:40 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Tue, Jan 31, 2023 at 09:15:03PM -0700, Alex Williamson wrote:
> 
> > > IMHO this is generally the way forward to do multi-device as well,
> > > remove the MMIO from all the address maps: VFIO, SW access, all of
> > > them. Nothing can touch MMIO except for the vCPU.  
> > 
> > Are you suggesting this relative to migration or in general?    
> 
> I would suggest a general qemu p2p on/off option.
> 
> > P2P is a feature with tangible benefits and real use cases.  Real
> > systems seem to be moving towards making P2P work better, so it
> > would seem short sighted to move to and enforce only configurations
> > w/o P2P in QEMU generally.    
> 
> qemu needs to support it, but it should be a user option option.
> 
> Every system I've been involved with for enabling P2P into a VM has
> been a total nightmare. This is not something you just turn on and it
> works great :\ The whole thing was carefully engineered right down to
> the BIOS to be able to work safely.
> 
> P2P in baremetal is much easier compared to P2P inside a VM.
> 
> > Besides, this would require some sort of deprecation, so are we
> > intending to make users choose between migration and P2P?  
> 
> Give qemu an option 'p2p on/p2p off' and default it to on for
> backwards compatability.
> 
> If p2p on and migration devices don't support P2P states then
> migration is disabled. The user made this choice when they bought
> un-capable HW.
> 
> Log warnings to make it more discoverable. I think with the cdev
> patches we can make it so libvirt can query the device FD for
> capabilities to be even cleaner.
> 
> If user sets 'p2p off' then migration works with all HW.
> 
> p2p on/off is a global switch. With p2p off nothing, no HW or SW or
> hybrid device, can touch the MMIO memory.
> 
> 'p2p off' is a valuable option in its own right because this stuff
> doesn't work reliably and is actively dangerous. Did you know you can
> hard crash the bare metal from a guest on some platforms with P2P
> operations? Yikes. If you don't need to use it turn it off and don't
> take the risk.

If we're honest, there are a number of cases of non-exceptional faults
that an assigned device can generate that the platform might escalate
to fatal errors.

> Arguably for this reason 'p2p off' should trend toward the default as
> it is much safer.

Safety in the hands of the userspace to protect the host though?
Shouldn't the opt-in be at the kernel level whether to allow p2p
mappings?  I don't have an issue if QEMU were to mirror this by
creating a RAM-only AddressSpace for devices which would be used when
p2p is disable (it'd save us some headaches for various unaligned
devices as well), but we shouldn't pretend that actually protects the
host.  OTOH, QEMU could feel confident supporting migration of devices
w/o support of the migration P2P states with that restriction.

> > Are we obliged to start with that hardware?  I'm just trying to think
> > about whether a single device restriction is sufficient to prevent any
> > possible P2P or whether there might be an easier starting point for
> > more capable hardware.  There's no shortage of hardware that could
> > support migration given sufficient effort.  Thanks,  
> 
> I think multi-device will likely have some use cases, so I'd like to
> see a path to have support for them. For this series I think it is
> probably fine since it is already 18 patches.

It might be fine for this series because it hasn't yet proposed to make
migration non-experimental, but it's unclear where the goal post is
that we can actually make that transition.

If we restrict migration to a single vfio device, is that enough?
Theoretically it's not, but perhaps in practice... maybe?

Therefore, do we depend on QEMU implementing a new RAM-only AddressSpace
for devices?  What's the path to making it the default?  Maybe there
are other aspects of the VM from which we can infer a preference
towards migration support, ex. 'host' CPU type.

Another option, as previously mentioned, is to start with requiring P2P
migration support both at the device and QEMU, where we only restrict
the set of devices that could initially support migration without
modifying existing behavior of the VM.

In any case, it seems we're a bit further from being able to declare
this as supported than some had hoped.  Thanks,

Alex