docs/devel/vfio-migration.rst | 68 ++++++++++++++++-------------------
1 file changed, 30 insertions(+), 38 deletions(-)
diff --git a/docs/devel/vfio-migration.rst
index 9ff6163c88..1d50c2fe5f 100644
@@ -7,46 +7,39 @@ the guest is running on source host and restoring
this saved state on the
destination host. This document details how saving and restoring of
devices is done in QEMU.
-Migration of VFIO devices consists of two phases: the optional
-and the stop-and-copy phase. The pre-copy phase is iterative and
-accommodate VFIO devices that have a large amount of data that needs
-transferred. The iterative pre-copy phase of migration allows for
the guest to
-continue whilst the VFIO device state is transferred to the
-helps to reduce the total downtime of the VM. VFIO devices can
choose to skip
-the pre-copy phase of migration by returning pending_bytes as zero
+Migration of VFIO devices currently consists of a single
+During the stop-and-copy phase the guest is stopped and the entire
+data is transferred to the destination.
+The pre-copy phase of migration is currently not supported for VFIO
+Support for VFIO pre-copy will be added later on.
A detailed description of the UAPI for VFIO device migration can be
-the comment for the ``vfio_device_migration_info`` structure in the
+the comment for the ``vfio_device_mig_state`` structure in the
VFIO implements the device hooks for the iterative approach as
-* A ``save_setup`` function that sets up the migration region and
- flag in the VFIO device state.
+* A ``save_setup`` function that sets up migration on the source.
-* A ``load_setup`` function that sets up the migration region on the
- destination and sets _RESUMING flag in the VFIO device state.
+* A ``load_setup`` function that sets the VFIO device on the
+ _RESUMING state.
* A ``save_live_pending`` function that reads pending_bytes from
driver, which indicates the amount of data that the vendor driver
has yet to
save for the VFIO device.
-* A ``save_live_iterate`` function that reads the VFIO device's data
- vendor driver through the migration region during iterative phase.
* A ``save_state`` function to save the device config space if it
-* A ``save_live_complete_precopy`` function that resets _RUNNING
flag from the
- VFIO device state and iteratively copies the remaining data for
- device until the vendor driver indicates that no data remains
- is zero).
+* A ``save_live_complete_precopy`` function that sets the VFIO
+ _STOP_COPY state and iteratively copies the data for the VFIO
+ the vendor driver indicates that no data remains.
* A ``load_state`` function that loads the config section and the data
- sections that are generated by the save functions above
+ sections that are generated by the save functions above.
* ``cleanup`` functions for both save and load that perform any
- related cleanup, including unmapping the migration region
+ related cleanup.
The VFIO migration code uses a VM state change handler to change
@@ -71,13 +64,13 @@ tracking can identify dirtied pages, but any page
pinned by the vendor driver
can also be written by the device. There is currently no device or
support for dirty page tracking in hardware.
-By default, dirty pages are tracked when the device is in pre-copy
as well as
-stop-and-copy phase. So, a page pinned by the vendor driver will be
-the destination in both phases. Copying dirty pages in pre-copy
-QEMU to predict if it can achieve its downtime tolerances. If QEMU
-pre-copy phase keeps finding dirty pages continuously, then it
-that even in stop-and-copy phase, it is likely to find dirty pages
-predict the downtime accordingly.
+By default, dirty pages are tracked during pre-copy as well as
+phase. So, a page pinned by the vendor driver will be copied to the
+in both phases. Copying dirty pages in pre-copy phase helps QEMU to
+it can achieve its downtime tolerances. If QEMU during pre-copy
+finding dirty pages continuously, then it understands that even in
+phase, it is likely to find dirty pages and can predict the downtime
QEMU also provides a per device opt-out option
which disables querying the dirty bitmap during pre-copy phase. If
it is set to
@@ -111,23 +104,22 @@ Live migration save path
migrate_init spawns migration_thread
Migration thread then calls each device's
- (RUNNING, _SETUP, _RUNNING|_SAVING)
+ (RUNNING, _SETUP, _RUNNING)
- (RUNNING, _ACTIVE, _RUNNING|_SAVING)
+ (RUNNING, _ACTIVE, _RUNNING)
If device is active, get pending_bytes by
If total pending_bytes >= threshold_size, call
- Data of VFIO device for pre-copy phase is copied
Iterate till total pending bytes converge and are less than
On migration completion, vCPU stops and calls
- each active device. The VFIO device is then transitioned into
- (FINISH_MIGRATE, _DEVICE, _SAVING)
+ each active device. The VFIO device is then transitioned into
+ (FINISH_MIGRATE, _DEVICE, _STOP_COPY)
For the VFIO device, iterate in .save_live_complete_precopy until
pending data is 0
- (FINISH_MIGRATE, _DEVICE, _STOPPED)
+ (FINISH_MIGRATE, _DEVICE, _STOP)
- (FINISH_MIGRATE, _COMPLETED, _STOPPED)
+ (FINISH_MIGRATE, _COMPLETED, _STOP)
Migraton thread schedules cleanup bottom half and exits
Live migration resume path
@@ -136,7 +128,7 @@ Live migration resume path
Incoming migration calls .load_setup for each device
- (RESTORE_VM, _ACTIVE, _STOPPED)
+ (RESTORE_VM, _ACTIVE, _STOP)
For each device, .load_state is called for that device
(RESTORE_VM, _ACTIVE, _RESUMING)