[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon)
From: |
Igor Mammedov |
Subject: |
Re: [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon) |
Date: |
Mon, 21 Sep 2020 11:00:23 +0200 |
On Sun, 20 Sep 2020 15:25:19 +0200
"Maciej S. Szmigiero" <mail@maciej.szmigiero.name> wrote:
> From: "Maciej S. Szmigiero" <maciej.szmigiero@oracle.com>
>From description it sounds like an alternative of virtio-mem,
CCing David.
> This series adds a Hyper-V Dynamic Memory Protocol driver (hv-balloon)
> and its protocol definitions.
> Also included is a driver providing backing devices for memory hot-add
> protocols ("haprots").
>
> A haprot device works like a virtual DIMM stick: it allows inserting
> extra RAM into the guest at run time.
>
> The main differences from the ACPI-based PC DIMM hotplug are:
> * Notifying the guest about the new memory range is not done via ACPI but
> via a protocol handler that registers with the haprot framework.
> This means that the ACPI DIMM slot limit does not apply.
>
> * A protocol handler can prevent removal of a haprot device when it is
> still in use by setting its "busy" field.
>
> * A protocol handler can also register an "unplug" callback so it gets
> notified when an user decides to remove the haprot device.
> This way the protocol handler can inform the guest about this fact and / or
> do its own cleanup.
>
> The hv-balloon driver is like virtio-balloon on steroids: it allows both
> changing the guest memory allocation via ballooning and inserting extra
> RAM into it by adding haprot virtual DIMM sticks.
> One of advantages of these over ACPI-based PC DIMM hotplug is that such
> memory can be hotplugged in much smaller granularity because the ACPI DIMM
> slot limit does not apply.
>
> In contrast with ACPI DIMM hotplug where one can only request to unplug a
> whole DIMM stick this driver allows removing memory from guest in single
> page (4k) units via ballooning.
> Then, once the guest has released the whole memory backed by a haprot
> virtual DIMM stick such device is marked "unused" and can be removed from
> the VM, if one wants so.
> A "HV_BALLOON_HAPROT_UNUSED" QMP event is emitted in this case so the
> software controlling QEMU knows that this operation is now possible.
>
> The haprot devices are also marked unused after a VM reboot (with a
> corresponding "HV_BALLOON_HAPROT_UNUSED" QMP event).
> They are automatically reinserted (if still present) after the guest
> reconnects to this protocol (a "HV_BALLOON_HAPROT_INUSE" QMP event is then
> emitted).
>
> For performance reasons, the guest-released memory is tracked in few range
> trees, as a series of (start, count) ranges.
> Each time a new page range is inserted into such tree its neighbors are
> checked as candidates for possible merging with it.
>
> Besides performance reasons, the Dynamic Memory protocol itself uses page
> ranges as the data structure in its messages, so relevant pages need to be
> merged into such ranges anyway.
>
> One has to be careful when tracking the guest-released pages, since the
> guest can maliciously report returning pages outside its current address
> space, which later clash with the address range of newly added memory.
> Similarly, the guest can report freeing the same page twice.
>
> The above design results in much better ballooning performance than when
> using virtio-balloon with the same guest: 230 GB / minute with this driver
> versus 70 GB / minute with virtio-balloon.
>
> During a ballooning operation most of time is spent waiting for the guest
> to come up with newly freed page ranges, processing the received ranges on
> the host side (in QEMU / KVM) is nearly instantaneous.
>
> The unballoon operation is also pretty much instantaneous:
> thanks to the merging of the ballooned out page ranges 200 GB of memory can
> be returned to the guest in about 1 second.
> With virtio-balloon this operation takes about 2.5 minutes.
>
> These tests were done against a Windows Server 2019 guest running on a
> Xeon E5-2699, after dirtying the whole memory inside guest before each
> balloon operation.
>
> Using a range tree instead of a bitmap to track the removed memory also
> means that the solution scales well with the guest size: even a 1 TB range
> takes just few bytes of memory.
>
> The required GTree operations are available at
> https://gitlab.gnome.org/maciejsszmigiero/glib/-/tree/gtree-add-iterators
> Since they are not yet in the upstream Glib a check for them was added to
> "configure" script, together with new "--enable-hv-balloon" and
> "--disable-hv-balloon" arguments.
> If these GTree operations are missing in the system Glib this driver will
> be skipped during QEMU build.
>
> An optional "status-report=on" device parameter requests memory status
> events from the guest (typically sent every second), which allow the host
> to learn both the guest memory available and the guest memory in use
> counts.
> They are emitted externally as "HV_BALLOON_STATUS_REPORT" QMP events.
>
> The driver is named hv-balloon since the Linux kernel client driver for
> the Dynamic Memory Protocol is named as such and to follow the naming
> pattern established by the virtio-balloon driver.
> The whole protocol runs over Hyper-V VMBus that has its implementation
> recently merged in.
>
> The driver was tested against Windows Server 2012 R2, Windows Server 2016
> and Windows Server 2016 guests and obeys the guest alignment requirements
> reported to the host via DM_CAPABILITIES_REPORT message.
> Extensive event tracing is available under 'hv_balloon_*' prefix.
>
> Example usage:
> * Add "-device vmbus-bridge,id=vmbus-bridge -device hv-balloon,id=hvb"
> to the QEMU command line and set "maxmem" value to something large,
> like 1T.
>
> * Use QEMU monitor commands to add a haprot virtual DIMM stick, together
> with its memory backend:
> object_add memory-backend-ram,id=mem1,size=200G
> device_add mem-haprot,id=ha1,memdev=mem1
> The first command is actually the same as for ACPI-based DIMM hotplug.
>
> * Use the ballooning interface monitor commands to force the guest to give
> out as much memory as possible:
> balloon 1
> The ballooning interface monitor commands can also be used to resize
> the guest up and down appropriately.
>
> * One can check the current guest size by issuing a "info balloon" command.
> This is useful to know what is happening, since large ballooning or
> unballooning operations take some time to complete.
>
> * Once the guest releases the whole memory backed by a haprot device
> (or is restarted) a "HV_BALLOON_HAPROT_UNUSED" QMP event will be
> generated.
> The haprot device then can be removed, together with its memory backend:
> device_del ha1
> object_del mem1
>
> Future directions:
> * Allow sharing the ballooning QEMU interface between hv-balloon and
> virtio-balloon drivers.
> Currently, only one of them can be added to the VM at the same time.
>
> * Allow new haport devices to reuse the same address range as the ones
> that were previously deleted via device_del monitor command without
> having to restart the VM.
>
> * Add vmstate / live migration support to the hv-balloon driver.
>
> * Use haprot device to also add memory via virtio interface (this requires
> defining a new operation in virtio-balloon protocol and appropriate
> support from the client virtio-balloon driver in the Linux kernel).
>
> Kconfig.host | 3 +
> configure | 35 +
> hw/hyperv/Kconfig | 5 +
> hw/hyperv/hv-balloon.c | 2172 ++++++++++++++++++++++++++++++
> hw/hyperv/meson.build | 1 +
> hw/hyperv/trace-events | 17 +
> hw/i386/Kconfig | 2 +
> hw/i386/pc.c | 18 +-
> hw/mem/Kconfig | 4 +
> hw/mem/haprot.c | 263 ++++
> hw/mem/meson.build | 1 +
> include/hw/hyperv/dynmem-proto.h | 425 ++++++
> include/hw/mem/haprot.h | 72 +
> meson.build | 2 +
> qapi/misc.json | 74 +
> 15 files changed, 3093 insertions(+), 1 deletion(-)
> create mode 100644 hw/hyperv/hv-balloon.c
> create mode 100644 hw/mem/haprot.c
> create mode 100644 include/hw/hyperv/dynmem-proto.h
> create mode 100644 include/hw/mem/haprot.h
>
>
- [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon), Maciej S. Szmigiero, 2020/09/20
- [PATCH 1/3] haprot: add a device for memory hot-add protocols, Maciej S. Szmigiero, 2020/09/20
- [PATCH 2/3] Add Hyper-V Dynamic Memory Protocol definitions, Maciej S. Szmigiero, 2020/09/20
- [PATCH 3/3] Add a Hyper-V Dynamic Memory Protocol driver (hv-balloon), Maciej S. Szmigiero, 2020/09/20
- Re: [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon), no-reply, 2020/09/20
- Re: [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon),
Igor Mammedov <=
- Re: [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon), David Hildenbrand, 2020/09/21
- Re: [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon), Maciej S. Szmigiero, 2020/09/21
- Re: [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon), David Hildenbrand, 2020/09/22
- Re: [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon), Maciej S. Szmigiero, 2020/09/22
- Re: [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon), David Hildenbrand, 2020/09/23
- Re: [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon), Maciej S. Szmigiero, 2020/09/24
- Re: [PATCH 0/3] Hyper-V Dynamic Memory Protocol driver (hv-balloon), David Hildenbrand, 2020/09/25