[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH v2 1/1] docs/devel: Add VFIO device migration documentation
From: |
Dr. David Alan Gilbert |
Subject: |
Re: [PATCH v2 1/1] docs/devel: Add VFIO device migration documentation |
Date: |
Tue, 16 Mar 2021 15:46:57 +0000 |
User-agent: |
Mutt/2.0.5 (2021-01-21) |
* Tian, Kevin (kevin.tian@intel.com) wrote:
> > From: Qemu-devel <qemu-devel-bounces+kevin.tian=intel.com@nongnu.org>
> > On Behalf Of Dr. David Alan Gilbert
> >
> > * Daniel P. Berrangé (berrange@redhat.com) wrote:
> > > On Thu, Mar 11, 2021 at 12:50:09AM +0530, Tarun Gupta wrote:
> > > > Document interfaces used for VFIO device migration. Added flow of state
> > changes
> > > > during live migration with VFIO device. Tested by building docs with the
> > new
> > > > vfio-migration.rst file.
> > > >
> > > > v2:
> > > > - Included the new vfio-migration.rst file in index.rst
> > > > - Updated dirty page tracking section, also added details about
> > > > 'pre-copy-dirty-page-tracking' opt-out option.
> > > > - Incorporated comments around wording of doc.
> > > >
> > > > Signed-off-by: Tarun Gupta <targupta@nvidia.com>
> > > > Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
> > > > ---
> > > > MAINTAINERS | 1 +
> > > > docs/devel/index.rst | 1 +
> > > > docs/devel/vfio-migration.rst | 135
> > ++++++++++++++++++++++++++++++++++
> > > > 3 files changed, 137 insertions(+)
> > > > create mode 100644 docs/devel/vfio-migration.rst
> > >
> > >
> > > > +Postcopy
> > > > +========
> > > > +
> > > > +Postcopy migration is not supported for VFIO devices.
> > >
> > > What is the problem here and is there any plan for how to address it ?
> >
> > There's no equivalent to userfaultfd for accesses to RAM made by a
> > device.
> > There's some potential for this to be doable with an IOMMU or the like,
> > but:
> > a) IOMMUs and devices aren't currently happy at recovering from
> > failures
> > b) the fragementation you get during a postcopy probably isn't pretty
> > when you get to build IOMMU tables.
>
> To overcome such limitations one may adopt a prefault-and-pull scheme if
> the vendor driver has the capability to track pending DMA buffers in the
> migration process (with additional uAPI changes in VFIO or userfaultfd),
> as discussed here:
>
> https://static.sched.com/hosted_files/kvmforum2019/7a/kvm-forum-postcopy-final.pdf
Did that get any further?
I can imagine that might be tricikier for a GPU than a network card; the
shaders in a GPU are pretty random as to what they go off and access, so
I can't see how you could prefault.
Dave
> >
> > > Postcopy is essentially the only migration mechanism that can reliably
> > > complete, so it really should be considered the default approach to
> > > migration for all mgmt apps wanting to do migration, except in special
> > > cases. IOW, if we want VFIO migration to be viable, we need postcopy
> > > support.
> >
> > There's lots of other things postcopy doesn't work with; so hmm.
> >
>
> Agree. Also given the amount of work even for pre-copy migration, it makes
> more sense to do things step-by-step.
>
> Thanks
> Kevin
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
RE: [PATCH v2 1/1] docs/devel: Add VFIO device migration documentation, Tian, Kevin, 2021/03/11
Re: [PATCH v2 1/1] docs/devel: Add VFIO device migration documentation, Cornelia Huck, 2021/03/15