Re: [Qemu-devel] [RFC 00/13] Live memory snapshot based on userfaultfd

[Baptiste Reynal]

On Tue, Jul 5, 2016 at 3:49 AM, Hailiang Zhang
<address@hidden> wrote:
> On 2016/7/4 20:22, Baptiste Reynal wrote:
>>
>> On Thu, Jan 7, 2016 at 1:19 PM, zhanghailiang
>> <address@hidden> wrote:
>>>
>>> For now, we still didn't support live memory snapshot, we have discussed
>>> a scheme which based on userfaultfd long time ago.
>>> You can find the discussion by the follow link:
>>> https://lists.nongnu.org/archive/html/qemu-devel/2014-11/msg01779.html
>>>
>>> The scheme is based on userfaultfd's write-protect capability.
>>> The userfaultfd write protection feature is available here:
>>> http://www.spinics.net/lists/linux-mm/msg97422.html
>>>
>>> The process of this live memory scheme is like bellow:
>>> 1. Pause VM
>>> 2. Enable write-protect fault notification by using userfaultfd to
>>>     mark VM's memory to write-protect (readonly).
>>> 3. Save VM's static state (here is device state) to snapshot file
>>> 4. Resume VM, VM is going to run.
>>> 5. Snapshot thread begins to save VM's live state (here is RAM) into
>>>     snapshot file.
>>> 6. During this time, all the actions of writing VM's memory will be
>>> blocked
>>>    by kernel, and kernel will wakeup the fault treating thread in qemu to
>>>    process this write-protect fault. The fault treating thread will
>>> deliver this
>>>    page's address to snapshot thread.
>>> 7. snapshot thread gets this address, save this page into snasphot file,
>>>     and then remove the write-protect by using userfaultfd API, after
>>> that,
>>>     the actions of writing will be recovered.
>>> 8. Repeat step 5~7 until all VM's memory is saved to snapshot file
>>>
>>> Compared with the feature of 'migrate VM's state to file',
>>> the main difference for live memory snapshot is it has little time delay
>>> for
>>> catching VM's state. It just captures the VM's state while got users
>>> snapshot
>>> command, just like take a photo of VM's state.
>>>
>>> For now, we only support tcg accelerator, since userfaultfd is not
>>> supporting
>>> tracking write faults for KVM.
>>>
>>> Usage:
>>> 1. Take a snapshot
>>> #x86_64-softmmu/qemu-system-x86_64 -machine
>>> pc-i440fx-2.5,accel=tcg,usb=off -drive
>>> file=/mnt/windows/win7_install.qcow2.bak,if=none,id=drive-ide0-0-1,format=qcow2,cache=none
>>> -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1  -vnc :7 -m
>>> 8192 -smp 1 -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0
>>> --monitor stdio
>>> Issue snapshot command:
>>> (qemu)migrate -d file:/home/Snapshot
>>> 2. Revert to the snapshot
>>> #x86_64-softmmu/qemu-system-x86_64 -machine
>>> pc-i440fx-2.5,accel=tcg,usb=off -drive
>>> file=/mnt/windows/win7_install.qcow2.bak,if=none,id=drive-ide0-0-1,format=qcow2,cache=none
>>> -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1  -vnc :7 -m
>>> 8192 -smp 1 -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0
>>> --monitor stdio -incoming file:/home/Snapshot
>>>
>>> NOTE:
>>> The userfaultfd write protection feature does not support THP for now,
>>> Before taking snapshot, please disable THP by:
>>> echo never > /sys/kernel/mm/transparent_hugepage/enabled
>>>
>>> TODO:
>>> - Reduce the influence for VM while taking snapshot
>>>
>>> zhanghailiang (13):
>>>    postcopy/migration: Split fault related state into struct
>>>      UserfaultState
>>>    migration: Allow the migrate command to work on file: urls
>>>    migration: Allow -incoming to work on file: urls
>>>    migration: Create a snapshot thread to realize saving memory snapshot
>>>    migration: implement initialization work for snapshot
>>>    QEMUSizedBuffer: Introduce two help functions for qsb
>>>    savevm: Split qemu_savevm_state_complete_precopy() into two helper
>>>      functions
>>>    snapshot: Save VM's device state into snapshot file
>>>    migration/postcopy-ram: fix some helper functions to support
>>>      userfaultfd write-protect
>>>    snapshot: Enable the write-protect notification capability for VM's
>>>      RAM
>>>    snapshot/migration: Save VM's RAM into snapshot file
>>>    migration/ram: Fix some helper functions' parameter to use
>>>      PageSearchStatus
>>>    snapshot: Remove page's write-protect and copy the content during
>>>      setup stage
>>>
>>>   include/migration/migration.h     |  41 +++++--
>>>   include/migration/postcopy-ram.h  |   9 +-
>>>   include/migration/qemu-file.h     |   3 +-
>>>   include/qemu/typedefs.h           |   1 +
>>>   include/sysemu/sysemu.h           |   3 +
>>>   linux-headers/linux/userfaultfd.h |  21 +++-
>>>   migration/fd.c                    |  51 ++++++++-
>>>   migration/migration.c             | 101 ++++++++++++++++-
>>>   migration/postcopy-ram.c          | 229
>>> ++++++++++++++++++++++++++++----------
>>>   migration/qemu-file-buf.c         |  61 ++++++++++
>>>   migration/ram.c                   | 104 ++++++++++++-----
>>>   migration/savevm.c                |  90 ++++++++++++---
>>>   trace-events                      |   1 +
>>>   13 files changed, 587 insertions(+), 128 deletions(-)
>>>
>>> --
>>> 1.8.3.1
>>>
>>>
>>>
>>
>
> Hi,
>
>> Hi Hailiang,
>>
>> Can I get the status of this patch series ? I cannot find a v2.
>
>
> Yes, I haven't updated it for long time, it is based on userfault-wp API
> in kernel, and Andrea didn't update the related patches until recent days.
> I will update this series in the next one or two weeks. But it will only
> support TCG until userfault-wp API supports KVM.
>

May I have a pointer to those patches ? The last I found is
http://thread.gmane.org/gmane.linux.kernel.mm/141647 and it seems
pretty old.

>> About TCG limitation, is KVM support on a TODO list or is there a
>> strong technical barrier ?
>>
>
> I don't think there are any technical hurdles, I would like to
> have a try on realizing the KVM part to support userfault-wp,
> But I'm a little busy with other things now.
> Andrea may has a plan to achieve it.
>
> To: Andrea Arcangeli <address@hidden>
>
> Thanks,
> Hailiang
>

Ok, if it is not on Andrea schedule I am willing to take the action,
at least for ARM/ARM64 support.

Regards,
Baptiste

>> Thanks,
>> Baptiste
>>
>> .
>>
>