[RFC v4] VFIO Migration

[Stefan Hajnoczi]

v4:
 * Replaced the complex migration parameter concept with a simpler version
   string as suggested by multiple people. The version string is similar to a
   versioned machine type or CPU model in QEMU. Device creation parameters are
   no longer taken into consideration, but it's likely that allowing devices to
   define a small set of variants (e.g. single-queue virtio-net, 64-queue
   virtio-net) will be sufficient.

v3:
 * Introduce migration info JSON to describe migration parameters
 * Rework mdev sysfs interface
 * Propose standard interface for vfio-user device emulation programs

VFIO Migration
==============
This document describes how to ensure migration compatibility for VFIO devices,
including mdev and vfio-user devices.

Overview
--------
VFIO devices can save and load a *device state*. Saving a device state produces
a snapshot of a VFIO device that can be loaded again at a later point in time
to resume the device from the snapshot.

The process of saving a device state and loading it later is called
*migration*. The device state may be loaded by the same device instance that
saved it or by a new instance, possibly running on a different machine.

A VFIO/mdev driver together with the physical device provides the functionality
of a device. Alternatively, a vfio-user device emulation program can provide
the functionality of a device. These are called *device implementations*.

The device implementation where a migration originates is called the *source*
and the device implementation that a migration targets is called the
*destination*.

Although it is possible to migrate device state without regard to migration
compatibility, this is prone to failure except in trivial cases. Device
implementations vary in feature availability and resource capacity so that it
is difficult to be confident that a migration of a complex device will succeed.
Furthermore, without migration compatibility checking it is possible that
migration appears to succeed but leaves the device in an inoperable state,
leading to data loss or corruption.

This document describes how to establish migration compatibility between the
source and destination. A check can be performed before migrating and can
therefore be used to select a suitable migration destination. When
compatibility has been established, the probability of migrating successfully
is high and a successful migration does not leave the device inoperable due to
silent migration problems.

Hardware Interface Compatibility
--------------------------------
VFIO devices have a *hardware interface* consisting of device regions and
interrupts. The hardware interface can vary between device instances. Examples
include:
* Functionality - hardware register blocks that are only present on certain
  device instances.
* Feature availability - features that are advertised through feature bits,
  device hardware revisions, or similar.
* Resource capacity - size of display framebuffers, number of queues, queue
  size, etc.

These examples demonstrate aspects of the hardware interface that must not
change unexpectedly across migration. Were they to differ between source and
destination, the chance of device driver malfunction would be high because the
layout of the hardware interface would change or assumptions the device driver
makes about available functionality would be violated.

Hardware interfaces sometimes support reporting an event when a change occurs.
In those cases it may be possible to changes in the hardware interface across
migration. In most other cases migration must not result in a visible change in
the hardware interface.

Device State Representation
---------------------------
Device state contains both data accessible through the device's hardware
interface and device-internal state needed to restore device operation.

The *device state representation* defines the binary data layout of the device
state. The device state representation is specific to each device and is beyond
the scope of this document, but aspects pertaining to migration compatibility
are discussed here.

Migration compatibility includes both migrating from old to new software
versions and from new to old software versions.

A device state representation can only be changed in a limited number of ways
without breaking migration compatibility. Some device state representations may
allow the addition of optional new fields that are ignored by old software
versions and therefore do not break compatibility.

Version Strings
---------------
*Version strings* uniquely identify hardware interfaces and device state
representations for the purpose of migration compatibility. Migration is only
possible between two devices that have the same version string.

Simple devices may define a sequence of version strings like "v1", "v2", etc.
Complex devicees that offer several variants need to define parallel sequences
such as "q1-v1", "q1-v2", etc and "q64-v1", "q64-v2", etc for a device that
offers both single-queue and 64-queue variants.

A version string is an ASCII string that does not contain newline characters
('\n').

Device Models
-------------
The combination of the hardware interface, device state representation, and
version strings definitions is called a *device model*. Device models are
identified by a unique ASCII string starting with a domain name and followed by
path components separated with backslashes ('/'). Examples include
vendor-a.com/my-nic, gitlab.com/user/my-device, virtio-spec.org/pci/virtio-net,
and qemu.org/pci/10ec/8139.

The device model string is not changed as the device evolves. Instead, version
strings are added to express new variants of the device.

The device model is not tied to a specific device implementation. The same
device model could be implemented as a VFIO/dev driver or as a vfio-user device
emulation program.

Multiple device implementations can support the same device model. Doing so
means that the device implementations can offer migration compatiblity because
they support the same hardware interface, device state representation, and
version strings.

Multiple device models can exist for the same hardware interface, each with a
different device state representation and version strings. This makes it
possible to fork and independently develop device models.

Orchestrating Migrations
------------------------
The following conditions must be met to establish migration compatibility:

1. The destination supports the source's device model.

2. The destination supports the source's version string.

The migration compatibility check can be performed without initiating a
migration. Therefore, this process can be used to select the migration
destination.

The following steps perform the migration:

1. Configure the destination so it is prepared to load the device state,
   including setting the device model and version string. This may involve
   instantiating a new device instance or resetting an existing device instance
   to a configuration that is compatible with the source.

   The details of how to do this for VFIO/mdev drivers and vfio-user device
   backend programs is described below.

2. Save the device state on the source and load it on the destination.

3. If migration succeeds then the destination resumes operation and the source
   must not resume operation. If the migration fails then the source resumes
   operation and the destination must not resume operation.

VFIO mdev Drivers
-----------------
The following mdev type sysfs attrs are available for managing device
instances::

  /sys/.../<parent-device>/mdev_supported_types/<type-id>/
    create - writing a UUID to this file instantiates a device
    migration_device_models - read-only list of device model and version
                              strings. Format:
                                <model-1>:\n
                                <version-1>\n
                                <version-2>\n
                                ...
                                <model-2>:\n
                                <version-1>\n
                                <version-2>\n

Device models supported by an mdev driver and supported version strings can be
read from the migration_device_models attr. An mdev type may support more than
one device model. Multiple mdev types may support the same device model, for
example, when they offer different resource capacity or feature availability
configurations. Typically the supported version strings will be different
between mdev types.

For example, a graphics card that supports 4 GB and 8 GB device instances would
provide gfx-4GB and gfx-8GB mdev types::

  /sys/.../<parent-device>/mdev_supported_types/
    gfx-4GB/
      create - writing a UUID to this file instantiates a device
      migration_device_models - Contents:
        org.project/graphics-card:
        4GB-v1
        4GB-v2
    gfx-8GB/
      create - writing a UUID to this file instantiates a device
      migration_device_models - Contents:
        org.project/graphics-card:
        8GB-v1
        8GB-v2

The following mdev device sysfs attrs relate to a specific device instance::

  /sys/.../<parent-device>/<uuid>/
    mdev_type/ - symlink to mdev type sysfs attrs, e.g. to fetch migration/model
    migration/ - migration related files
      device_model - read/write device model string
      version - read/write version string

When the device is created migration/device_model and migration/version contain
default values chosen by the driver. They can be set to any value listed in the
migration_device_models attr. Setting the device_model attr resets the version
attr to a default value chosen by the driver.

Changes to the device_model and version attrs may fail once the mdev device has
been opened. The mdev driver may allow changes as long as the hardware
interface of the open device remains compatible. This makes it possible to
upgrade or downgrade to a device model and version with an incompatible device
state representation.

vfio-user Device Emulation Programs
-----------------------------------
Device emulation programs often support a simple invocation model where running
the program creates a single device instance. The lifecycle of the device
instance is tied to the lifecycle of the process. Such device emulation
programs are described below.

More complex device emulation programs may host multiple devices. The interface
for configuring these device emulation programs is not standardized. Therefore,
migrating these devices is beyond the scope of this document.

The migration information JSON is printed to standard output by a vfio-user
device emulation program as follows:

.. code:: bash

  $ my-device --print-migration-info-json
  {
    "models": {
      "<model>": {
        "versions": ["<version>", ...]
      }
    }
  }

The `models` object contains a member for each supported device model. The
`versions` member is a list of all supported version strings.

The device is instantiated by launching the destination process with the
device model and version from the source:

.. code:: bash

  $ my-device --migration-model <model> --migration-version <version> ...

There is no standard interface for changing the device model and version at
runtime.

signature.asc
Description: PGP signature