Re: Reducing vdpa migration downtime because of memory pin / maps

[Eugenio Perez Martin]

On Wed, Apr 12, 2023 at 8:19 AM Jason Wang <jasowang@redhat.com> wrote:
>
> On Wed, Apr 12, 2023 at 1:56 PM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Tue, Apr 11, 2023 at 8:34 PM Eugenio Perez Martin
> > <eperezma@redhat.com> wrote:
> > >
> > > On Wed, Apr 5, 2023 at 1:37 PM Eugenio Perez Martin <eperezma@redhat.com> 
> > > wrote:
> > > >
> > > > Hi!
> > > >
> > > > As mentioned in the last upstream virtio-networking meeting, one of
> > > > the factors that adds more downtime to migration is the handling of
> > > > the guest memory (pin, map, etc). At this moment this handling is
> > > > bound to the virtio life cycle (DRIVER_OK, RESET). In that sense, the
> > > > destination device waits until all the guest memory / state is
> > > > migrated to start pinning all the memory.
> > > >
> > > > The proposal is to bind it to the char device life cycle (open vs
> > > > close), so all the guest memory can be pinned for all the guest / qemu
> > > > lifecycle.
> > > >
> > > > This has two main problems:
> > > > * At this moment the reset semantics forces the vdpa device to unmap
> > > > all the memory. So this change needs a vhost vdpa feature flag.
> > > > * This may increase the initialization time. Maybe we can delay it if
> > > > qemu is not the destination of a LM. Anyway I think this should be
> > > > done as an optimization on top.
> > > >
> > >
> > > Expanding on this we could reduce the pinning even more now that vring
> > > supports VA [1] with the emulated CVQ.
> >
> > Note that VA for hardware means the device needs to support page fault
> > through either PRI or vendor specific interface.
> >
> > >
> > > Something like:
> > > - Add VHOST_VRING_GROUP_CAN_USE_VA ioctl to check if a given VQ group
> > > capability. Passthrough devices with emulated CVQ would return false
> > > for the dataplane and true for the control vq group.
>
> We don't even need this actually, since the pinning is not visible to
> the userspace. Userspace can only see the IOTLB abstraction actually.
>
> We can invent a group->use_va, then when we attach AS to a group that
> can use va, we can avoid the pinning.
>

That would solve one part for sure, but SVQ will keep translating HVA
to SVQ IOVA, and then the kernel needs to translate it back. With
VHOST_VRING_GROUP_CAN_USE_VA, the SVQ and the kernel skip all
translation.

Thanks!

> Thanks
>
> > > - If that is true, qemu does not need to map and translate addresses
> > > for CVQ but to directly provide VA for buffers. This avoids pinning,
> > > translations, etc in this case.
> >
> > For CVQ yes, but we only avoid the pinning for CVQ not others.
> >
> > Thanks
> >
> > >
> > > Thanks!
> > >
> > > [1] 
> > > https://lore.kernel.org/virtualization/20230404131326.44403-2-sgarzare@redhat.com/
> > >
>