[Qemu-devel] [PATCH v10 00/14, resend] rdma: migration support

[mrhines]

From: "Michael R. Hines" <address@hidden>

Please pull.

Changes since v9: patch #14, #4, and #10 :
    Insted of timing just the pinning, we now time the entire setup
    phase of migration. This requires postponing MIG_STATE_ACTIVE
    initialization to the migration_thread instead of the QMP command.

    Also importantly: We are maintaining a patch to libvirt which
    knows how to deal with this, because the 'ACTIVE' state may 
    happen later than expected, since libvirt was previous not handling
    the 'setup' state - this was never exposed over QMP until now.
    As soon as RDMA is merged, we will immediately send a patch to
    libvir-list development list to deal with this.

    For very large virtual machines, pinning can take a long time.
    While this does not affect the migration's *actual* time itself,
    it is still important for the user to know what's going on and
    to know what component of the total time is actual taken up by
    pinning.

    For example, using a 14GB virtual machine, pinning can take as
    long as 5 seconds, for which the user would not otherwise know
    what was happening.

    Also: Fix patch #4 & #10: "(since 1.5)" => "(since 1.6) 

Reviewed-by: Paolo Bonzini <address@hidden>
Reviewed-by: Chegu Vinod <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Tested-by: Chegu Vinod <address@hidden>
Tested-by: Michael R. Hines <address@hidden>
Signed-off-by: Michael R. Hines <address@hidden>

Wiki: http://wiki.qemu.org/Features/RDMALiveMigration
Github: address@hidden:hinesmr/qemu.git

Here is a brief summary of total migration time and downtime using RDMA:

Using a 40gbps infiniband link performing a worst-case stress test,
using an 8GB RAM virtual machine:
Using the following command:

$ apt-get install stress
$ stress --vm-bytes 7500M --vm 1 --vm-keep

RESULTS:

1. Migration throughput: 26 gigabits/second.
2. Downtime (stop time) varies between 15 and 100 milliseconds.

EFFECTS of memory registration on bulk phase round:

For example, in the same 8GB RAM example with all 8GB of memory in 
active use and the VM itself is completely idle using the same 40 gbps 
infiniband link:

1. x-rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
2. x-rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps

These numbers would of course scale up to whatever size virtual machine
you have to migrate using RDMA.

Enabling this feature does *not* have any measurable affect on 
migration *downtime*. This is because, without this feature, all of the 
memory will have already been registered already in advance during
the bulk round and does not need to be re-registered during the successive
iteration rounds.

The following changes since commit f3aa844bbb2922a5b8393d17620eca7d7e921ab3:

  build: include config-{, all-}devices.mak after defining CONFIG_SOFTMMU and 
CONFIG_USER_ONLY (2013-04-24 12:18:41 -0500)

are available in the git repository at:

  address@hidden:hinesmr/qemu.git rdma_patch_v10

for you to fetch changes up to 75e6fac1f642885b93cefe6e1874d648e9850f8f:

  rdma: send pc.ram (2013-04-24 14:55:01 -0400)

----------------------------------------------------------------
Michael R. Hines (14):
  rdma: add documentation
  rdma: introduce qemu_update_position()
  rdma: export yield_until_fd_readable()
  rdma: export throughput w/ MigrationStats QMP
  rdma: introduce qemu_file_mode_is_not_valid()
  rdma: export qemu_fflush()
  rdma: introduce ram_handle_compressed()
  rdma: introduce qemu_ram_foreach_block()
  rdma: new QEMUFileOps hooks
  rdma: introduce capability x-rdma-pin-all
  rdma: core logic
  rdma: send pc.ram
  rdma: fix mlock() freezes and accounting
  rdma: add setup time accounting to QMP statistics

 Makefile.objs                 |    1 +
 arch_init.c                   |   69 +-
 configure                     |   29 +
 docs/rdma.txt                 |  415 ++++++
 exec.c                        |    9 +
 hmp.c                         |   12 +-
 include/block/coroutine.h     |    6 +
 include/exec/cpu-common.h     |    5 +
 include/migration/migration.h |   32 +
 include/migration/qemu-file.h |   32 +
 migration-rdma.c              | 2819 +++++++++++++++++++++++++++++++++++++++++
 migration.c                   |   64 +-
 qapi-schema.json              |   21 +-
 qemu-coroutine-io.c           |   23 +
 savevm.c                      |  114 +-
 15 files changed, 3589 insertions(+), 62 deletions(-)
 create mode 100644 docs/rdma.txt
 create mode 100644 migration-rdma.c

-- 
1.7.10.4