[Qemu-devel] [PATCH 00/46] Postcopy implementation

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" <address@hidden>

Hi,
  This is the 1st cut of my version of postcopy; it is designed for use with
the Linux kernel additions recently posted by Andrea Arcangeli here:

   http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg00525.html

The current status is:
  1) It works - I've done testing on large/busy VMs using
     google-stress app test, and it completes cleanly even over a 1Gbps
     link.

  2) It's still rather rough around the corner cases and exit code,
     so it's probably not ready for review yet, but I thought I'd put
     it out for people working in the area and for people to see the
     general shape.

I've taken some ideas from Isaku Yamahata's work, but the code is
structured a bit differently, however I'm greatful for the understanding
I got from Isaku's code of the problems postcopy has to deal with.

Note, the last patch in the series is a chunk of documentation,
which it's probably useful to read for detail.

Some points:
   a) I've tried to keep the pieces I've added general rather than postcopy
      specific.

   b) Precopy is done using the main precopy code (with very few modifications)
      and it automatically switches to postcopy after a timeout.

   c) A 'command' section type is added that controls the progress of postcopy
      and can be used for anything else people need to control that's not 
actually
      guest state.

   d) There is a 'return path' mechanism for sending data from the destination
      back to the source, postcopy uses this for the page requests, but I think
      it's also useful for things like the failover implementations, and can
      be used to signal failure from the destination to the source.

   e) I've added a 'migration_set_parameter' command as somewhere to put integer
      parameters associated with migration.
      e.1) And I use that initially for the length of precopy to try.

   f) I've added a 'MigrationIncomingState' where postcopy keeps it's data,
      it seems a good idea to move other incoming state there as well.

c & d together provide a 'reqack' that's basically a ping, quite useful
for debugging

Current TODO:
   1) It's not bisectable yet
   2) There are no testsuite additions (although I have a virt-test modification
      I've been using).
   3) End-of-migration cleanup is unfinished, as is some of the error handling.
   4) Not all the code is there for systems with hostpagesize!=qemupagesize
   5) xbzrle needs disabling once in postcopy
   6) RDMA needs some rework
   7) I moved QEMUFile into a public header, it's a bad idea and I'll undo that.
   8) My latency measurements on page requests are generally pretty good, but 
I'm
      seeing a few high spikes, I need to investigate.
   9) Andrea has suggestions on ways to avoid some of the huge-page splitting
      that occurs during the discard phase after precopy.
  10) I'd like to format the data on the return path in a more structured way
      (i.e. maybe using stuff from my BER world).

I'm hoping to get that lot cleared up by 2.2

How to use:
   migrate_set_capability x-postcopy-ram on
   migrate_set_parameter x-postcopy-start-time 500            <--- that's in ms
   migrate -d tcp:whereever:port

Other ideas:
  I think the returnpath+command could be used to send qmp commands to
  the destination to allow the src/destination to coordinate a hotplug
  or to set up the destination state.

Dave
Dr. David Alan Gilbert (46):
  qemu_ram_foreach_block: pass up error value, and down the ramblock
    name
  Move QEMUFile structure to qemu-file.h
  QEMUSizedBuffer/QEMUFile
  improve DPRINTF macros, add to savevm
  Add qemu_get_counted_string to read a string prefixed by a count byte
  Create MigrationIncomingState
  Return path: Open a return path on QEMUFile for sockets
  Return path: socket_writev_buffer: Block even on non-blocking fd's
  Migration commands
  Return path: Control commands
  Return path: Send responses from destination to source
  Return path: Source handling of return path
  qemu_loadvm debug
  ram_debug_dump_bitmap: Dump a migration bitmap as text
  Rework loadvm path for subloops
  Add migration-capability boolean for postcopy-ram.
  Add wrappers and handlers for sending/receiving the postcopy-ram
    migration messages.
  QEMU_VM_CMD_PACKAGED: Send a packaged chunk of migration stream
  migrate_init: Call from savevm
  Allow savevm handlers to state whether they could go into postcopy
  postcopy: OS support test
  Migration parameters: Add qmp/hmp commands for setting/viewing
  MIG_STATE_POSTCOPY_ACTIVE: Add new migration state
  qemu_savevm_state_complete: Postcopy changes
  Postcopy: Maintain sentmap during postcopy pre phase
  Postcopy page-map-incoming (PMI) structure
  postcopy: Add incoming_init/cleanup functions
  postcopy: Incoming initialisation
  postcopy: ram_enable_notify to switch on userfault
  Postcopy: postcopy_start
  Postcopy: Rework migration thread for postcopy mode
  mig fd_connect: open return path
  Postcopy: Create a fault handler thread before marking the ram as
    userfault
  Page request:  Add MIG_RPCOMM_REQPAGES reverse command
  Page request: Process incoming page request
  Page request: Consume pages off the post-copy queue
  Add assertion to check migration_dirty_pages doesn't go -ve; have seen
    it happen once but not sure why
  postcopy_ram.c: place_page and helpers
  Postcopy: Use helpers to map pages during migration
  qemu_ram_block_from_host
  Handle userfault requests (although userfaultfd not done yet)
  Start up a postcopy/listener thread ready for incoming page data
  postcopy: Wire up loadvm_postcopy_ram_handle_{run,end} commands
  postcopy: Use userfaultfd
  End of migration for postcopy
  Start documenting how postcopy works.

 Makefile.objs                    |   2 +-
 arch_init.c                      | 477 ++++++++++++++++++++--
 docs/migration.txt               | 148 +++++++
 exec.c                           |  60 ++-
 hmp-commands.hx                  |  17 +
 hmp.c                            |  54 +++
 hmp.h                            |   4 +
 include/exec/cpu-common.h        |   7 +-
 include/migration/migration.h    | 115 ++++++
 include/migration/postcopy-ram.h |  83 ++++
 include/migration/qemu-file.h    |  62 +++
 include/migration/vmstate.h      |   2 +-
 include/qemu/typedefs.h          |   8 +-
 include/sysemu/sysemu.h          |  41 +-
 migration-rdma.c                 |   4 +-
 migration.c                      | 695 ++++++++++++++++++++++++++++++--
 monitor.c                        |  25 ++
 postcopy-ram.c                   | 833 +++++++++++++++++++++++++++++++++++++++
 qapi-schema.json                 |  56 ++-
 qemu-file.c                      | 578 +++++++++++++++++++++++++--
 qmp-commands.hx                  |  23 ++
 savevm.c                         | 822 +++++++++++++++++++++++++++++++++++---
 22 files changed, 3933 insertions(+), 183 deletions(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 postcopy-ram.c

-- 
1.9.3