Re: [Qemu-devel] [RFC] Split migration bitmaps by ramblock

[Yang Hongyang]

On 2017/3/24 16:34, Juan Quintela wrote:
> Yang Hongyang <address@hidden> wrote:
>> Hi Juan,
>>
>>   First of all, I like the refactor patchset about RAMState, it makes
>> things clean, great!
> 
> Thanks.
> 
> The whole idea of the series was to make testing changes easier.
> 
>> On 2017/3/24 5:01, Juan Quintela wrote:
>>> Hi
>>>
>>> This series split the migration and unsent bitmaps by ramblock.  This
>>> makes it easier to synchronize in small bits.  This is on top of the
>>> RAMState and not-hotplug series.
>>>
>>> Why?
>>>
>>> reason 1:
>>>
>>> People have complained that by the time that we detect that a page is
>>> sent, it has already been marked dirty "again" inside kvm, so we are
>>> going to send it again.  On top of this patch, my idea is, for words
>>> of the bitmap that have any bit set, just synchonize the bitmap before
>>> sending the pages.  I have not looking into performance numbers yet,
>>> jsut asking for comments about how it is done.
>>
>> Here you said 'synchonize the bitmap before sending the pages', do you
>> mean synchronize the bitmap from kvm? If so, I doubt the performance...
>> because every synchronization will require a ioctl(). If not, the
>> synchronization of per block is useless.
>>
>> Currently, migration thread will synchronize the bitmap from kvm every
>> iter(migration_bitmap_sync()). The RAMBlock already has kind of per block
>> bitmap for this kind of sync. And the migration bitmap is used to put all
>> those per block bitmap together for data sending use.
> 
> Hi
> For huge memory machines, we are doing it always in one go.
> 
> 
> bitmap_sync(1TB RAM)
> walk bitmap for 512MB of RAM, at that point, it is very probable that
> this page is again dirty in the KVM bitmap, so, we send it, but as it is
> dirty again, we would have to send it in the next pass.  This sent is
> completely useless.

I got your point, the problem is KVM do not have the ability to sync in
small chunks currently, even it has, it will generate lots of ioctls:
KVM bitmap sync is per Memory Region IIRC, think we have a 1T mem
guest, 16 MRs for example, currently, every iter we need to do 16 ioctls.
But if we sync in small chunks, 64M for example, we might need to do
16384 ioctls in the worst case. eg:mem press(dirty rate) is very high.


> 
> So my idea is to split things in smaller chunks.  As we have to do an
> ioctl, we wouldn't want to synchronize page by page, but perhaps 16MB at
> a time, 64MB, anything less than the whole amount of memory.
> 
> Later, Juan.
> 

-- 
Thanks,
Yang