Re: [PATCH] migration: Move bitmap_mutex out of migration_bitmap_clear_dirty()

[David Hildenbrand]

On 02.07.21 04:48, Wang, Wei W wrote:
> On Thursday, July 1, 2021 10:22 PM, David Hildenbrand wrote:
> > On 01.07.21 14:51, Peter Xu wrote:
> > Spoiler alert: the introduction of clean bitmaps partially broke free page 
> > hinting
> > already (as clearing happens deferred -- and might never happen if we don't
> > migrate *any* page within a clean bitmap chunk, so pages actually remain
> > dirty ...). "broke" here means that pages still get migrated even though they
> > were reported by the guest. We'd actually not want to use clean bmaps with free
> > page hinting ... long story short, free page hinting is a very fragile beast 
> > already
> > and some of the hints are basically ignored and pure overhead ...
>
> Not really. Both clear_bmap and free page hint are to "clear" the bitmap.
> No matter who comes first to clear it, it won’t cause more (unexpected) pages 
> to be sent.

I was able to reproduce something like that before my vacation by 
starting a 8GB VM, wait until Linux is up, migrate it.

With clear_bmap:

QEMU 6.0.50 monitor - type 'help' for more information
(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: completed
total time: 4532 ms
downtime: 4 ms
setup: 3 ms
transferred ram: 578480 kbytes
throughput: 1047.05 mbps
remaining ram: 0 kbytes
total ram: 8389384 kbytes
duplicate: 1080349 pages
skipped: 0 pages
normal: 141969 pages
normal bytes: 567876 kbytes
dirty sync count: 4
page size: 4 kbytes
multifd bytes: 0 kbytes
pages-per-second: 4102921
(qemu)

Without clear_bmap:

(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: completed
total time: 4891 ms
downtime: 61 ms
setup: 29 ms
transferred ram: 666400 kbytes
throughput: 1123.47 mbps
remaining ram: 0 kbytes
total ram: 8389384 kbytes
duplicate: 33427 pages
skipped: 0 pages
normal: 166202 pages
normal bytes: 664808 kbytes
dirty sync count: 4
page size: 4 kbytes
multifd bytes: 0 kbytes
pages-per-second: 32386
(qemu)

Without free page hinting:

QEMU 6.0.50 monitor - type 'help' for more information
(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: completed
total time: 4975 ms
downtime: 48 ms
setup: 3 ms
transferred ram: 639982 kbytes
throughput: 1055.09 mbps
remaining ram: 0 kbytes
total ram: 8389384 kbytes
duplicate: 1942431 pages
skipped: 0 pages
normal: 155424 pages
normal bytes: 621696 kbytes
dirty sync count: 3
page size: 4 kbytes
multifd bytes: 0 kbytes
pages-per-second: 32386
(qemu)


Take a look at "duplicate". We seem to end up reading+migrating a lot 
more pages (all filled with zero, as the guest never touched them) with 
clear_bmap, but not as much as with free page hinting disabled.


Repeating the same experiment (not performing the "no free page hinting" 
step) by running "memhog 7g" in the guest before migrating, such that we 
don't have all-zero pages:

With clear_bmap:

QEMU 6.0.50 monitor - type 'help' for more information
(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: completed
total time: 28574 ms
downtime: 73 ms
setup: 4 ms
transferred ram: 3715000 kbytes
throughput: 1065.33 mbps
remaining ram: 0 kbytes
total ram: 8389384 kbytes
duplicate: 21472 pages
skipped: 0 pages
normal: 926892 pages
normal bytes: 3707568 kbytes
dirty sync count: 4
page size: 4 kbytes
multifd bytes: 0 kbytes
pages-per-second: 32710
(qemu)

Without clear_bmap:

QEMU 6.0.50 monitor - type 'help' for more information
(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: completed
total time: 5818 ms
downtime: 24 ms
setup: 29 ms
transferred ram: 672486 kbytes
throughput: 952.18 mbps
remaining ram: 0 kbytes
total ram: 8389384 kbytes
duplicate: 19449 pages
skipped: 0 pages
normal: 167751 pages
normal bytes: 671004 kbytes
dirty sync count: 4
page size: 4 kbytes
multifd bytes: 0 kbytes
pages-per-second: 32710
(qemu)



I think that clearly shows the issue.

My theory I did not verify yet: Assume we have 1GB chunks in the clear 
bmap. Assume the VM reports all pages within a 1GB chunk as free (easy 
with a fresh VM). While processing hints, we will clear the bits from 
the dirty bmap in the RAMBlock. As we will never migrate any page of 
that 1GB chunk, we will not actually clear the dirty bitmap of the 
memory region. When re-syncing, we will set all bits bits in the dirty 
bmap again from the dirty bitmap in the memory region. Thus, many of our 
hints end up being mostly ignored. The smaller the clear bmap chunk, the 
more extreme the issue.

--
Thanks,

David / dhildenb