[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [PATCH v7 0/5] virtio-balloon: free page hint reporting

From: Hailiang Zhang
Subject: Re: [Qemu-devel] [PATCH v7 0/5] virtio-balloon: free page hint reporting support
Date: Tue, 29 May 2018 23:00:21 +0800
User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0

On 2018/4/24 14:13, Wei Wang wrote:
This is the deivce part implementation to add a new feature,
VIRTIO_BALLOON_F_FREE_PAGE_HINT to the virtio-balloon device. The device
receives the guest free page hints from the driver and clears the
corresponding bits in the dirty bitmap, so that those free pages are
not transferred by the migration thread to the destination.

- Test Environment
     Host: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
     Guest: 8G RAM, 4 vCPU
     Migration setup: migrate_set_speed 100G, migrate_set_downtime 2 second

- Test Results
     - Idle Guest Live Migration Time (results are averaged over 10 runs):
         - Optimization v.s. Legacy = 271ms vs 1769ms --> ~86% reduction
     - Guest with Linux Compilation Workload (make bzImage -j4):
         - Live Migration Time (average)
           Optimization v.s. Legacy = 1265ms v.s. 2634ms --> ~51% reduction
         - Linux Compilation Time
           Optimization v.s. Legacy = 4min56s v.s. 5min3s
           --> no obvious difference

- Source Code
     - QEMU:  https://github.com/wei-w-wang/qemu-free-page-lm.git
     - Linux: https://github.com/wei-w-wang/linux-free-page-lm.git

           - add virtio_notify() at the end to notify the driver that
             the optimization is done, which indicates that the entries
             have all been put back to the vq and ready to detach them.
       virtio-balloon: use iothread to get free page hint
     1) migration:
         - bitmap_clear_dirty: update the dirty bitmap and dirty page
           count under the bitmap mutex as what other functions are doing;
         - qemu_guest_free_page_hint:
             - add comments for this function;
             - check the !block case;
             - check "offset > block->used_length" before proceed;
             - assign used_len inside the for{} body;
             - update the dirty bitmap and dirty page counter under the
               bitmap mutex;
         - ram_state_reset:
             - rs->free_page_support: && with use "migrate_postcopy"
               instead of migration_in_postcopy;
             - clear the ram_bulk_stage flag if free_page_support is true;
     2) balloon:
          - add the usage documentation of balloon_free_page_start and
            balloon_free_page_stop in code;
          - the optimization thread is named "balloon_fpo" to meet the
            requirement of "less than 14 characters";
          - virtio_balloon_poll_free_page_hints:
              - run on condition when runstate_is_running() is true;
              - add a qemu spin lock to synchronize accesses to the free
                page reporting related fields shared among the migration
                thread and the optimization thread;
           - virtio_balloon_free_page_start: just return if
             runstate_is_running is false;
           - virtio_balloon_free_page_stop: access to the free page
             reporting related fields under a qemu spin lock;
           - virtio_balloon_device_unrealize/reset: call
             virtio_balloon_free_page_stop is the free page hint feature is
           - virtio_balloon_set_status: call irtio_balloon_free_page_stop
             in case the guest is stopped by qmp when the optimization is
     1) bitmap: add a new API to count 1s starting from an offset of a
     2) migration:
         - qemu_guest_free_page_hint: calculate
           ram_state->migration_dirty_pages by counting how many bits of
           free pages are truely cleared. If some of the bits were
           already 0, they shouldn't be deducted by
           ram_state->migration_dirty_pages. This wasn't needed for
           previous versions since we optimized bulk stage only,
           where all bits are guaranteed to be set. It's needed now
           because we extened the usage of this optimizaton to all stages
           except the last stop&copy stage. From 2nd stage onward, there
           are possibilities that some bits of free pages are already 0.
      3) virtio-balloon:
          - virtio_balloon_free_page_report_status: introduce a new status,
            FREE_PAGE_REPORT_S_EXIT. This status indicates that the
            optimization thread has exited. FREE_PAGE_REPORT_S_STOP means
            the reporting is stopped, but the optimization thread still needs
            to be joined by the migration thread.
     1) virtio-balloon
         - virtio_balloon_free_page_start: poll the hints using a new
         - use cmd id between [0x80000000, UINT_MAX];
         - virtio_balloon_poll_free_page_hints:
             - stop the optimization only when it has started;
             - don't skip free pages when !poison_val;
         - add poison_val to vmsd to migrate;
         - virtio_balloon_get_features: add the F_PAGE_POISON feature when
           host has F_FREE_PAGE_HINT;
         - remove the timer patch which is not needed now.
     2) migration
        - new api, qemu_guest_free_page_hint;
        - rs->free_page_support set only in the precopy case;
        - use the new balloon APIs.
     1) virtio-balloon
         - use subsections to save free_page_report_cmd_id;
         - poll the free page vq after sending a cmd id to the driver;
         - change the free page vq size to VIRTQUEUE_MAX_SIZE;
         - virtio_balloon_poll_free_page_hints: handle the corner case
           that the free page block reported from the driver may cross
           the RAMBlock boundary.
     2) migration/ram.c
         - use balloon_free_page_poll to start the optimization

Wei Wang (5):
   bitmap: bitmap_count_one_with_offset
   migration: use bitmap_mutex in migration_bitmap_clear_dirty
   migration: API to clear bits of guest free pages from the dirty bitmap
   migration: use the free page hint feature from balloon

  balloon.c                                       |  58 +++++-
  hw/virtio/virtio-balloon.c                      | 241 ++++++++++++++++++++++--
  include/hw/virtio/virtio-balloon.h              |  27 ++-
  include/migration/misc.h                        |   2 +
  include/qemu/bitmap.h                           |  13 ++
  include/standard-headers/linux/virtio_balloon.h |   7 +
  include/sysemu/balloon.h                        |  15 +-
  migration/ram.c                                 |  73 ++++++-
  8 files changed, 406 insertions(+), 30 deletions(-)

Nice optimization, for the first stage of  current migration method, we need to 
migrate all the pages of
VM to destination,  with this capability, we can reduce lots of unnecessary 
pages migrating.

Just a small piece of advice, it is better to split the fourth patch into small 
ones, to make it more easy
for reviewing. Besides, should we make this capability an optional one, just 
like other migration capabilities do ?

reply via email to

[Prev in Thread] Current Thread [Next in Thread]