[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[PATCH v3 00/17] migration/snapshot: External snapshot utility

From: nikita . lapshin
Subject: [PATCH v3 00/17] migration/snapshot: External snapshot utility
Date: Thu, 16 Jun 2022 13:27:54 +0300

From: Nikita Lapshin <nikita.lapshin@openvz.org>

Changes v2 -> v3
 * Refactored tool code to decrease duplications of migration code.
 * Used sequential migration for saving which means that vmstate
   will be send first and only after that ram part will be sent.
   For this purpose stream-content-list paramter was used.
 * Removed aio work with block driver. Should be replaced with
   existing in qcow2 format in next versions.
 * Removed postcopy percent. Should be added in next versions.

Changes v1 -> v2:
 * Fixed CI checks

Changes v0 -> v1:
 * Changed command-line format, now use blockdev specification to
   define vmstate image.
 * Don't deal with image creation in the tool, create externally.
 * Better block layer AIO handling in the load path.
 * Reduced fragmentation of the image backing file by using 'writtent-slice'
   bitmaps in RAM blocks. Zero block write is issued to a never written slice
   before the actual memory page write takes place.
 * Improved load performance in postcopy by using 'loaded-slice' bitmaps
   in RAM blocks.
 * Refactored error handling/messages.
 * Refactored namings.

This series is a kind of PoC for asynchronous snapshot reverting. It is
about external snapshots only and doesn't involve block devices. Thus, it's
mainly intended to be used with the new 'background-snapshot' migration
capability and otherwise standard QEMU migration mechanism.

The major ideas behind this version were:
  * Make it compatible with 'exec:'-style migration - options can be create
    some separate tool or integrate into qemu-system.
  * Support asynchronous revert stage by using unaltered postcopy logic
    at destination. To do this, we should be capable of saving RAM pages
    so that any particular page can be directly addressed by it's block ID
    and page offset. Possible solutions here seem to be:
      use separate index (and storing it somewhere)
      create sparse file on host FS and address pages with file offset
      use QCOW2 (or other) image container with inherent sparsity support
  * Make image file dense on the host FS so we don't depend on
    copy/backup tools and how they deal with sparse files. Off course,
    there's some performance cost for this choice.
  * Try to keep page save latencies small while not degrading migration
    bandwidth too much.

This version of snapshot-tool is the first step to integrate tool into
main QEMU. Now tool replace ram hanlers so it can call existing functions
in migration/* part to parse migration stream.

For the storage format, QCOW2 as a container and large (1MB) cluster size seem
to be an optimal choice. Larger cluster is beneficial for performance 
in the case when image preallocation is disabled. Such cluster size does not 
in too high internal fragmentation level (~10% of space waste in most cases) yet
allows to reduce significantly the number of expensive cluster allocations.

There was no strict guarantee that there is no sections in ram part
rather than ram. So to solve this problem we decided to implement
parameters stream-content-list to provide such guarantee strictly.
This decision also helps with reusing of existed migration code.
You can see it in tool load part where tool disables all handlers except
ram using this parameter. If you have already seen it in previous patches you
can skip first 8 commits.

"sequential migration"
One problem remains unsolved. We need to run two migrations first to
save vmstate and second to save ram. We cannot run migration if VM is in
postmigrate state. But if we want to make snapshot this prohibition is
unnecessary so I changed some parts of migration and softmmu so
sequential migration become permitted. But that is not a solution. May
be new capability should be implementedi for that purpose.

Some of the upgrades were removed for now. This happened because of refactoring
and should be implemented in next versions.

How to use:

* > qemu-img create -f qcow2 -o size=<2_x_ram_size>,cluster_size=1M,
           preallocation=off,refcount_bits=8 <image-filename>
* qemu> migrate_set_capability background-snapshot on
* #set SCL to "vmstate" only
* qemu> migrate "exec:qemu-snapshot --save-vmstate
* #set SCL to "ram" only
* qemu> migrate "exec:qemu-snapshot

* Use 'qemu-system-* -incoming defer'
* qemu> migrate_incoming "exec:qemu-snapshot --revert

**Load with postcopy:**
* Use 'qemu-system-* -incoming defer'
* qemu> migrate_set_capability postcopy-ram on
* qemu> migrate_incoming "exec:qemu-snapshot --revert --postcopy

Nikita Lapshin (17):
  migration: Implemented new parameter stream_content
  migration: should_skip() implemented
  migration: Add vmstate part of migration stream
  igration: Add dirty-bitmaps part of migration stream
  Add block part of migration stream
  migration: Add RAM part of migration stream
  migration: analyze-migration script changed
  migration: Test for RAM and vmstate parts
  migration/snapshot: Introduce qemu-snapshot tool
  migration/snapshot: Build changes for qemu-snapshot-tool
  migration/qemu-file: Fix qemu_ftell() for non-writable file
  migration/snapshot: Move RAM_SAVE_FLAG_xxx defines to migration/ram.h
  migration/snapshot: Block layer support in qemu-snapshot
  migration/snpashot: Implement API for RAMBlock
  migration/snapshot: Save part implement
  migration/snapshot: Precopy load implemented
  migration/snapshot: Postcopy load implemented

 include/qemu-snapshot.h                       |   94 ++
 meson.build                                   |   18 +
 migration/meson.build                         |    4 +-
 migration/migration.c                         |  199 ++-
 migration/migration.h                         |    4 +
 migration/qemu-file.c                         |    3 +-
 migration/qemu-snapshot-io.c                  |  112 ++
 migration/qemu-snapshot.c                     | 1126 +++++++++++++++++
 migration/ram.c                               |   22 +-
 migration/ram.h                               |   16 +
 migration/savevm.c                            |  116 +-
 migration/savevm.h                            |    8 +
 qapi/migration.json                           |   21 +-
 qemu-snapshot.c                               |  540 ++++++++
 scripts/analyze-migration.py                  |   19 +-
 .../tests/migrate-ram-stream-content-test     |   96 ++
 .../tests/migrate-ram-stream-content-test.out |    5 +
 17 files changed, 2342 insertions(+), 61 deletions(-)
 create mode 100644 include/qemu-snapshot.h
 create mode 100644 migration/qemu-snapshot-io.c
 create mode 100644 migration/qemu-snapshot.c
 create mode 100644 qemu-snapshot.c
 create mode 100755 tests/qemu-iotests/tests/migrate-ram-stream-content-test
 create mode 100644 tests/qemu-iotests/tests/migrate-ram-stream-content-test.out


reply via email to

[Prev in Thread] Current Thread [Next in Thread]