RE: [PATCH v2 1/1] docs/devel: Add VFIO device migration documentation

[Tian, Kevin]

> From: Tarun Gupta (SW-GPU) <targupta@nvidia.com>
> Sent: Tuesday, March 16, 2021 9:35 PM
> 
> 
> >> +
> >> +* A ``save_live_iterate`` function that reads the VFIO device's data from
> the
> >> +  vendor driver through the migration region during iterative phase.
> >> +
> >> +* A ``save_live_complete_precopy`` function that resets _RUNNING flag
> >> from the
> >> +  VFIO device state, saves the device config space, if any, and 
> >> iteratively
> >
> > and if any,
> 
> I didn't get this. I intended to say that it will save the device config
> space only if it is present.
> So, used "saves device config space, if any".

I misread it, with the impression that 'if any' is for 'iteratively copy'.

> >> +
> >> +System memory dirty pages tracking
> >> +----------------------------------
> >> +
> >> +A ``log_sync`` memory listener callback marks those system memory
> pages
> >> +as dirty which are used for DMA by the VFIO device. The dirty pages
> bitmap
> >> is
> >> +queried per container. All pages pinned by the vendor driver through
> >> +vfio_pin_pages() external API have to be marked as dirty during
> migration.
> >
> > why mention kernel internal functions in an userspace doc?
> 
> I'll remove the mention of vfio_pin_pages() and just mention "external API".
> 
> >
> >> +When there are CPU writes, CPU dirty page tracking can identify dirtied
> >> pages,
> >> +but any page pinned by the vendor driver can also be written by device.
> >> There
> >> +is currently no device which has hardware support for dirty page tracking.
> >
> > no device or IOMMU support
> 
> Right, will update it.
> 
> >
> >> So
> >> +all pages which are pinned by vendor driver are considered as dirty.
> >
> > Similarly, why do we care about how the kernel identifies whether a page is
> > dirty. It could be dirtied due to pinning, or due to IOMMU dirty bit, or due
> > to IOMMU page fault. Here we'd better just focus on user-tangible effect,
> > e.g. a large/non-converging dirty map might be returned then how to
> handle
> > such situation...
> 
> Since VFIO migration feature is not just implemented in userspace but
> also involves implementation in kernel space as well, have documented
> here what is implemented as of now.

but userpace only needs to care about the implications of the uAPI, instead
of the internal detail in the kernel. You might take above pinning detail as
one example to explain that the size of the dirty page set might be big and
static, and then explain how userspace should cope with this situation. but 
don't describe it as the assumed fact.

Thanks
Kevin