[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH for 9.0 08/12] vdpa: add vhost_vdpa_load_setup
From: |
Jason Wang |
Subject: |
Re: [PATCH for 9.0 08/12] vdpa: add vhost_vdpa_load_setup |
Date: |
Thu, 21 Dec 2023 10:17:41 +0800 |
On Wed, Dec 20, 2023 at 3:07 PM Eugenio Perez Martin
<eperezma@redhat.com> wrote:
>
> On Wed, Dec 20, 2023 at 6:22 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Sat, Dec 16, 2023 at 1:28 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > >
> > > Callers can use this function to setup the incoming migration thread.
> > >
> > > This thread is able to map the guest memory while the migration is
> > > ongoing, without blocking QMP or other important tasks. While this
> > > allows the destination QEMU not to block, it expands the mapping time
> > > during migration instead of making it pre-migration.
> >
> > If it's just QMP, can we simply use bh with a quota here?
> >
>
> Because QEMU cannot guarantee the quota at write(fd,
> VHOST_IOTLB_UPDATE, ...).
So you mean the delay may be caused by a single syscall?
> Also, synchronization with
> vhost_vdpa_dev_start would complicate as it would need to be
> re-scheduled too.
Just a flush of the bh, or not?
But another question. How to synchronize with the memory API in this
case. Currently the updating (without vIOMMU) is done under the
listener callback.
Usually after the commit, Qemu may think the memory topology has been
updated. If it is done asynchronously, would we have any problem?
>
> As a half-baked idea, we can split the mapping chunks in manageable
> sizes, but I don't like that idea a lot.
>
> > Btw, have you measured the hotspot that causes such slowness? Is it
> > pinning or vendor specific mapping that slows down the progress? Or if
> > VFIO has a similar issue?
> >
>
> Si-Wei did the actual profiling as he is the one with the 128G guests,
> but most of the time was spent in the memory pinning. Si-Wei, please
> correct me if I'm wrong.
>
> I didn't check VFIO, but I think it just maps at realize phase with
> vfio_realize -> vfio_attach_device -> vfio_connect_container(). In
> previous testings, this delayed the VM initialization by a lot, as
> we're moving that 20s of blocking to every VM start.
>
> Investigating a way to do it only in the case of being the destination
> of a live migration, I think the right place is .load_setup migration
> handler. But I'm ok to move it for sure.
Adding Peter for more ideas.
>
> > >
> > > This thread joins at vdpa backend device start, so it could happen that
> > > the guest memory is so large that we still have guest memory to map
> > > before this time.
> >
> > So we would still hit the QMP stall in this case?
> >
>
> This paragraph is kind of outdated, sorry. I can only cause this if I
> don't enable switchover_ack migration capability and if I artificially
> make memory pinning in the kernel artificially slow. But I didn't
> check QMP to be honest, so I can try to test it, yes.
>
> If QMP is not responsive, that means QMP is not responsive in QEMU
> master in that period actually. So we're only improving anyway.
>
> Thanks!
>
Thanks
[PATCH for 9.0 09/12] vdpa: approve switchover after memory map in the migration destination, Eugenio Pérez, 2023/12/15
[PATCH for 9.0 03/12] vdpa: merge _begin_batch into _batch_begin_once, Eugenio Pérez, 2023/12/15
[PATCH for 9.0 10/12] vdpa: add vhost_vdpa_net_load_setup NetClient callback, Eugenio Pérez, 2023/12/15
[PATCH for 9.0 11/12] vdpa: add vhost_vdpa_net_switchover_ack_needed, Eugenio Pérez, 2023/12/15
[PATCH for 9.0 12/12] virtio_net: register incremental migration handlers, Eugenio Pérez, 2023/12/15
Re: [PATCH for 9.0 00/12] Map memory at destination .load_setup in vDPA-net migration, Lei Yang, 2023/12/24
Re: [PATCH for 9.0 00/12] Map memory at destination .load_setup in vDPA-net migration, Michael S. Tsirkin, 2023/12/25