Re: [RFC v2] VFIO Migration

[Stefan Hajnoczi]

On Thu, Nov 05, 2020 at 04:52:20PM +0100, Cornelia Huck wrote:
> On Thu, 5 Nov 2020 15:09:02 +0000
> Stefan Hajnoczi <stefanha@redhat.com> wrote:
> 
> (...)
> 
> <did not fully read through the v1 thread, so apologies if I missed
> something>
> 
> > VFIO/mdev Devices
> > -----------------
> > TODO this is a first draft, more thought needed around enumerating supported
> > parameters, representing default values, etc
> > 
> > The following mdev type sysfs attrs are available for managing device
> > instances:
> > 
> >   /sys/.../<parent-device>/mdev_supported_types/<type-id>/
> >       create - writing a UUID to this file instantiates a device
> >       migration/ - migration related files
> >           model - unique device model string, e.g. vendor-a.com/my-nic
> > 
> > Device models supported by an mdev driver can be enumerated by reading the
> > migration/model attr for each <type-id>.
> 
> IIUC, we're grouping together all users of a specific mdev_type, but
> support a variety of sub-configurations? Does that include parameters
> or not? If not, shouldn't we already be covered by mdev_type?

I will include an explanation of how mdev types relate to migration
parameters in the next revision of this document.

> > 
> > The following mdev device sysfs attrs relate to a specific device instance:
> > 
> >   /sys/.../<parent-device>/<uuid>/
> >       mdev_type/ - symlink to mdev type sysfs attrs, e.g. to fetch 
> > migration/model
> >       migration/ - migration related files
> >           applied - Write "1" to apply current migration parameter values or
> >                     "0" to reset migration parameter values to their 
> > defaults.
> >                     Parameters can only be applied or reset while the mdev 
> > is
> >                     not opened.
> >           params/ - migration parameters
> >               <my-param> - read/write migration parameter "my-param"
> >               ...
> > 
> > When the device is created the migration/applied attr is "0". Migration
> > parameters are accessible in migration/params/ and read 0 bytes because they
> > are at their default values.  At the point opening the mdev device will fail
> > because migration parameters must be applied first. Migration parameters 
> > can be
> > set to the desired values or left at their defaults. "1" must be written to
> > migration/applied before opening the mdev device.
> > 
> > If writing to a migration/params/<param> attr or setting migration/applied 
> > to
> > "1" fails, then the device implementation does not support the migration
> > parameters.
> > 
> > An open mdev device typically does not allow migration parameters to be 
> > changed
> > at runtime. However, certain migration/params attrs may allow writes at
> > runtime. Usually these migration parameters only affect the device state
> > representation and not the hardware interface. This makes it possible to
> > upgrade or downgrade the device state representation at runtime so that
> > migration is possible to newer or older device implementations.
> > 
> > An existing mdev device instance can be reused by closing the mdev device 
> > and
> > writing "0" to migration/applied. This resets parameters to their defaults 
> > so
> > that a new list of migration parameters can be applied.
> > 
> > The migration parameter list for an mdev device that is in operation can be
> > read from migration/params/. Parameters that read 0 bytes are at their 
> > default
> > value.
> 
> I'm trying to figure out what that means for the mdevs I'm most
> familiar with, ccw and ap. Both of them currently support a single
> mdev_type.
> 
> For ccw, there are some things that I could imagine as parameters, like
> the device number, or channel paths. Maybe we could include the channel
> path type (FICON, ...) in the migration device model? We should not
> include device numbers etc. in the device model.

That sounds good. Usually the host-specifics (which host device number
is being passed through) are not guest-visible and shouldn't be
migration parameters. Anything that affects the guest-visible hardware
interface or device state representation needs to be a migration
parameter.

> For ap, we have matrices covering tuples (APQNs) derived from a
> cross-product of card/domains configure via sysfs attributes. I think
> later modification of these is desired. I think we also might be able
> to mix-and-match different types within the same matrix, so not sure if
> we can put these into any device model. In fact, I'm a bit at a loss
> how the device model for ap would look like (other than simply
> 'matrix'). Can we deal with dynamic parameters?

Migration parameters are static. If you might need migration parameters
foo1, foo2, foo3, foo4, foo5 at runtime then they can be defined
statically but default to off.

Also, this migration compatibility scheme is progressive rather than a
binary "full compatibility checking" vs "no compatibility checking"
choice. QEMU relies on the user or management tool to set up compatible
source and destinations with a few sanity checks in cases where QEMU
developers thought it was helpful. So QEMU is somewhere in the middle of
the spectrum. I'm not trying to force anyone to express everything in
migration parameters. Public device models (e.g. if we device one for
virtio-net-pci) will probably be towards the "full compatibility
checking" side of the spectrum so that variations between device
implementations can be detected and handled. A proprietary device model
might be fine with just a single hardware-revision=N parameter that is
incremented every time something changes. It can be as simple or complex
as needed.

Stefan

signature.asc
Description: PGP signature