Re: [Qemu-devel] [RFC] Split migration bitmaps by ramblock

[Dr. David Alan Gilbert]

* Yang Hongyang (address@hidden) wrote:
> 
> 
> On 2017/3/24 16:34, Juan Quintela wrote:
> > Yang Hongyang <address@hidden> wrote:
> >> Hi Juan,
> >>
> >>   First of all, I like the refactor patchset about RAMState, it makes
> >> things clean, great!
> > 
> > Thanks.
> > 
> > The whole idea of the series was to make testing changes easier.
> > 
> >> On 2017/3/24 5:01, Juan Quintela wrote:
> >>> Hi
> >>>
> >>> This series split the migration and unsent bitmaps by ramblock.  This
> >>> makes it easier to synchronize in small bits.  This is on top of the
> >>> RAMState and not-hotplug series.
> >>>
> >>> Why?
> >>>
> >>> reason 1:
> >>>
> >>> People have complained that by the time that we detect that a page is
> >>> sent, it has already been marked dirty "again" inside kvm, so we are
> >>> going to send it again.  On top of this patch, my idea is, for words
> >>> of the bitmap that have any bit set, just synchonize the bitmap before
> >>> sending the pages.  I have not looking into performance numbers yet,
> >>> jsut asking for comments about how it is done.
> >>
> >> Here you said 'synchonize the bitmap before sending the pages', do you
> >> mean synchronize the bitmap from kvm? If so, I doubt the performance...
> >> because every synchronization will require a ioctl(). If not, the
> >> synchronization of per block is useless.
> >>
> >> Currently, migration thread will synchronize the bitmap from kvm every
> >> iter(migration_bitmap_sync()). The RAMBlock already has kind of per block
> >> bitmap for this kind of sync. And the migration bitmap is used to put all
> >> those per block bitmap together for data sending use.
> > 
> > Hi
> > For huge memory machines, we are doing it always in one go.
> > 
> > 
> > bitmap_sync(1TB RAM)
> > walk bitmap for 512MB of RAM, at that point, it is very probable that
> > this page is again dirty in the KVM bitmap, so, we send it, but as it is
> > dirty again, we would have to send it in the next pass.  This sent is
> > completely useless.
> 
> I got your point, the problem is KVM do not have the ability to sync in
> small chunks currently, even it has, it will generate lots of ioctls:
> KVM bitmap sync is per Memory Region IIRC, think we have a 1T mem
> guest, 16 MRs for example, currently, every iter we need to do 16 ioctls.
> But if we sync in small chunks, 64M for example, we might need to do
> 16384 ioctls in the worst case. eg:mem press(dirty rate) is very high.

Yes but I think people  have suggested it should have that ability for a long
time; so hopefully we can add it.

I'd assumed the sizes of these chunks would be a bit bigger, a few GB each maybe
and also assumed we'd have a separate thread that was doing the syncing from
a different thread to the one doing the writing, trying to keep ahead of the
write pointer.

Even with the structure Juan has here, we could get to syncing each NUMA
node separately like that which would kind of makes some sense.

Dave

> 
> > 
> > So my idea is to split things in smaller chunks.  As we have to do an
> > ioctl, we wouldn't want to synchronize page by page, but perhaps 16MB at
> > a time, 64MB, anything less than the whole amount of memory.
> > 
> > Later, Juan.
> > 
> 
> -- 
> Thanks,
> Yang
> 
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK