Re: [PATCH v5 00/10] numa/exec/migration: Fix resizing RAM blocks while migrating

[Dr. David Alan Gilbert]

* David Hildenbrand (david@redhat.com) wrote:
> v4 has been floating around for a while. Let's see if we can find someone
> to merge this; or at least give some more feedback ... all patches have
> at least one RB.
> 
> 
> I realized that resizing RAM blocks while the guest is being migrated
> (precopy: resize while still running on the source, postcopy: resize
>  while already running on the target) is buggy. In case of precopy, we
> can simply cancel migration. Postcopy handling is more involved. Resizing
> can currently happen during a guest reboot, triggered by ACPI rebuilds.
> 
> Along with the fixes, some cleanups.

Queued

> 
> --------------------------------------------------------------------------
> 
> Example to highlight one part of the problem:
> 
> 1. Start a paused VM (where a ramblock resize will trigger when booting):
>   sudo build/qemu-system-x86_64 \
>        --enable-kvm \
>        -S \
>        -machine q35,nvdimm=on \
>        -smp 1 \
>        -cpu host \
>        -m size=20G,slots=8,maxmem=22G \
>        -object memory-backend-file,id=mem0,mem-path=/tmp/nvdimm,size=256M \
>        -device nvdimm,label-size=131072,memdev=mem0,id=nvdimm0,slot=1 \
>        -nodefaults \
>        -chardev stdio,nosignal,id=serial \
>        -device isa-serial,chardev=serial \
>        -chardev socket,id=monitor,path=/var/tmp/monitor,server,nowait \
>        -mon chardev=monitor,mode=readline \
>        -device vmgenid \
>        -device intel-iommu \
>        -nographic
> 
> 2. Starting precopy and then starting the VM to trigger resizing during
>    precopy:
>   QEMU 5.2.95 monitor - type 'help' for more information
>   (qemu) migrate -d "exec:gzip -c > STATEFILE.gz"
>   QEMU 5.2.95 monitor - type 'help' for more information
>   (qemu) cont
> 
> 3a. Before this series, migration never completes:
>   QEMU 5.2.95 monitor - type 'help' for more information
>   (qemu) info migrate
>   globals:
>   store-global-state: on
>   only-migratable: off
>   send-configuration: on
>   send-section-footer: on
>   decompress-error-check: on
>   clear-bitmap-shift: 18
>   Migration status: active
>   total time: 43826 ms
>   expected downtime: 300 ms
>   setup: 5 ms
>   transferred ram: 65981 kbytes
>   throughput: 8.27 mbps
>   remaining ram: 18446744073709551612 kbytes
>   total ram: 21234188 kbytes
>   duplicate: 5308454 pages
>   skipped: 0 pages
>   normal: 93 pages
>   normal bytes: 372 kbytes
>   dirty sync count: 1
>   page size: 4 kbytes
>   multifd bytes: 0 kbytes
>   pages-per-second: 0
> 
> 4. With this change, migration is properly aborted:
>   (qemu) info migrate
>   globals:
>   store-global-state: on
>   only-migratable: off
>   send-configuration: on
>   send-section-footer: on
>   decompress-error-check: on
>   clear-bitmap-shift: 18
>   Migration status: cancelled
>   total time: 0 ms
> 
> --------------------------------------------------------------------------
> 
> Cc: Eduardo Habkost <ehabkost@redhat.com>
> Cc: Marcel Apfelbaum <marcel.apfelbaum@gmail.com>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Juan Quintela <quintela@redhat.com>
> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> Cc: Peter Xu <peterx@redhat.com>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> 
> v4 -> v5:
> - Rephrased some patch descriptions
> - Dropped some patches to reduce the footprint
> -- "stubs/ram-block: Remove stubs that are no longer needed"
> -- "migration/ram: Tolerate partially changed mappings in postcopy code"
> - Removed as already upstream now
> -- "migration/ram: Consolidate variable reset after placement in
>     ram_load_postcopy()"
> 
> v3 -> v4:
> - Rebased and retested
> - Added RBs
> 
> v2 -> v3:
> - Rebased on current master
> - Added RBs
> - "migration/ram: Tolerate partially changed mappings in postcopy code"
> -- Extended the comment for the uffdio unregister part.
> 
> v1 -> v2:
> - "util: vfio-helpers: Factor out and fix processing of existing ram
>    blocks"
> -- Stringify error
> - "migraton/ram: Handle RAM block resizes during precopy"
> -- Simplified check if we're migrating on the source
> - "exec: Relax range check in ram_block_discard_range()"
> -- Added to make discard during resizes actually work
> - "migration/ram: Discard new RAM when growing RAM blocks after
>    ram_postcopy_incoming_init()"
> -- Better checks if in the right postcopy mode.
> -- Better patch subject/description/comments
> - "migration/ram: Handle RAM block resizes during postcopy"
> -- Better comments
> -- Adapt to changed postcopy checks
> - "migrate/ram: Get rid of "place_source" in ram_load_postcopy()"
> -- Dropped, as broken
> - "migration/ram: Tolerate partially changed mappings in postcopy code"
> -- Better comment / description. Clarify that no implicit wakeup will
>    happen
> -- Warn on EINVAL (older kernels)
> -- Wake up any waiter explicitly
> 
> David Hildenbrand (10):
>   util: vfio-helpers: Factor out and fix processing of existing ram
>     blocks
>   numa: Teach ram block notifiers about resizeable ram blocks
>   numa: Make all callbacks of ram block notifiers optional
>   migration/ram: Handle RAM block resizes during precopy
>   exec: Relax range check in ram_block_discard_range()
>   migration/ram: Discard RAM when growing RAM blocks after
>     ram_postcopy_incoming_init()
>   migration/ram: Simplify host page handling in ram_load_postcopy()
>   migration/ram: Handle RAM block resizes during postcopy
>   migration/multifd: Print used_length of memory block
>   migration/ram: Use offset_in_ramblock() in range checks
> 
>  hw/core/numa.c             |  41 +++++++++--
>  hw/i386/xen/xen-mapcache.c |   7 +-
>  include/exec/cpu-common.h  |   1 +
>  include/exec/memory.h      |  10 +--
>  include/exec/ramblock.h    |  10 +++
>  include/exec/ramlist.h     |  13 ++--
>  migration/migration.c      |   9 ++-
>  migration/migration.h      |   1 +
>  migration/multifd.c        |   2 +-
>  migration/postcopy-ram.c   |  15 ++++-
>  migration/ram.c            | 135 +++++++++++++++++++++++++++++--------
>  softmmu/physmem.c          |  26 +++++--
>  target/i386/hax/hax-mem.c  |   5 +-
>  target/i386/sev.c          |  18 ++---
>  util/vfio-helpers.c        |  41 ++++-------
>  15 files changed, 241 insertions(+), 93 deletions(-)
> 
> -- 
> 2.30.2
> 
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK