Re: [PATCH v1 0/6] hw/arm/virt: Support dirty ring

[Zhenyu Zhang]

[PATCH v1 0/6] hw/arm/virt: Support dirty ring

The patches work well on my arm Ampere host.
The test results are as expected.

Testing
=======
(1) kvm-unit-tests/its-pending-migration,
kvm-unit-tests/its-migrate-unmapped-collection
    and kvm-unit-tests/its-migration with dirty ring or normal dirty
page tracking
    mechanism. All test cases passed.

    QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64 ACCEL=kvm \
    PROCESSOR=host ./its-migration

    QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64 ACCEL=kvm \
    PROCESSOR=host ./its-migrate-unmapped-collection

    QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64 ACCEL=kvm \
    PROCESSOR=host ./its-pending-migration

    QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64
ACCEL=kvm,dirty-ring-size=65536 \
    PROCESSOR=host ./its-migration

    QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64
ACCEL=kvm,dirty-ring-size=65536 \
    PROCESSOR=host ./its-migrate-unmapped-collection

    QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64
ACCEL=kvm,dirty-ring-size=65536 \
    PROCESSOR=host ./its-pending-migration

(2) Combinations of migration, post-copy migration, e1000e and virtio-net
    devices. All test cases passed.

    -device '{"driver": "virtio-net-pci", "mac": "9a:97:8f:c7:cc:a6",
"rombar": 0, "netdev": "idDGdh30", "bus": "pcie-root-port-4", "addr":
"0x0"}'  \
    -netdev tap,id=idDGdh30,vhost=on

    -device '{"driver": "e1000e", "mac": "9a:fd:93:f1:97:b1",
"netdev": "idXDOtMA", "bus": "pcie-root-port-4", "addr": "0x0"}'  \
    -netdev tap,id=idXDOtMA,vhost=on  \

(3) Simulate heavy memory pressure scenarios and compare the migration
    performance difference between dirty ring and dirty logging.

    I gave with a 200G memory guest, 40 vcpus, using 10g NIC as migration
    channel.  When idle or dirty workload small, I don't observe major
    difference on total migration time.  When with higher random dirty
    workload (Anonymous mapping 180G memory, 256MB/s dirty rate upon).
    Total migration time is (in seconds):


    |-------------------------+|-------------------------|
    | dirty ring (4k entries) | dirty logging       |
    |-------------------------+|-------------------------|
    |                           67 |                         74 |
    |                           67 |                         74 |
    |                           66 |                         76 |
    |                           66 |                         73 |
    |                           67 |                         76 |
    |                           67 |                         76 |
    |                           66 |                         73 |
    |                           67 |                         74 |
    |-------------------------+|-------------------------|

    Summary:

    dirty ring average:    67s
    dirty logging average: 75s

Tested-by: Zhenyu Zhang <zhenzha@redhat.com>


On Mon, Feb 13, 2023 at 8:39 AM Gavin Shan <gshan@redhat.com> wrote:
>
> This series intends to support dirty ring for live migration. The dirty
> ring use discrete buffer to track dirty pages. For ARM64, the speciality
> is to use backup bitmap to track dirty pages when there is no-running-vcpu
> context. It's known that the backup bitmap needs to be synchronized when
> KVM device "kvm-arm-gicv3" or "arm-its-kvm" has been enabled. The backup
> bitmap is collected in the last stage of migration.
>
> PATCH[1]    Synchronize linux-headers for dirty ring
> PATCH[2]    Introduce indicator of the last stage migration and pass it
>             all the way down
> PATCH[3]    Synchronize the backup bitmap in the last stage of live migration
> PATCH[4]    Introduce helper kvm_dirty_ring_init() to enable the dirty ring
> PATCH[5-6]  Enable dirty ring for hw/arm/virt
>
> RFCv1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00171.html
>
> Testing
> =======
> (1) kvm-unit-tests/its-pending-migration and kvm-unit-tests/its-migration with
>     dirty ring or normal dirty page tracking mechanism. All test cases passed.
>
>     QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
>     ./its-pending-migration
>
>     QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
>     ./its-migration
>
>     QEMU=./qemu.main/build/qemu-system-aarch64 
> ACCEL=kvm,dirty-ring-size=65536 \
>     ./its-pending-migration
>
>     QEMU=./qemu.main/build/qemu-system-aarch64 
> ACCEL=kvm,dirty-ring-size=65536 \
>     ./its-migration
>
> (2) Combinations of migration, post-copy migration, e1000e and virtio-net
>     devices. All test cases passed.
>
>     -netdev tap,id=net0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown  \
>     -device e1000e,bus=pcie.5,netdev=net0,mac=52:54:00:f1:26:a0
>
>     -netdev tap,id=vnet0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
>     -device virtio-net-pci,bus=pcie.6,netdev=vnet0,mac=52:54:00:f1:26:b0
>
> Changelog
> =========
> v1:
>   * Combine two patches into one PATCH[v1 2/6] for the last stage indicator   
>  (Peter)
>   * Drop the secondary bitmap and use the original one directly               
>  (Juan)
>   * Avoid "goto out" in helper kvm_dirty_ring_init()                          
>  (Juan)
>
>
> Gavin Shan (6):
>   linux-headers: Update for dirty ring
>   migration: Add last stage indicator to global dirty log
>     synchronization
>   kvm: Synchronize the backup bitmap in the last stage
>   kvm: Add helper kvm_dirty_ring_init()
>   hw/arm/virt: Enable backup bitmap for dirty ring
>   kvm: Enable dirty ring for arm64
>
>  accel/kvm/kvm-all.c           | 95 ++++++++++++++++++++++++-----------
>  hw/arm/virt.c                 | 26 ++++++++++
>  include/exec/memory.h         |  5 +-
>  include/sysemu/kvm_int.h      |  1 +
>  linux-headers/asm-arm64/kvm.h |  1 +
>  linux-headers/linux/kvm.h     |  2 +
>  migration/dirtyrate.c         |  4 +-
>  migration/ram.c               | 20 ++++----
>  softmmu/memory.c              | 10 ++--
>  target/arm/kvm64.c            | 25 +++++++++
>  target/arm/kvm_arm.h          | 12 +++++
>  11 files changed, 152 insertions(+), 49 deletions(-)
>
> --
> 2.23.0
>