[Qemu-commits] [qemu/qemu] 817c60: migration: Improve QMP documentation

[GitHub]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 817c60457f41e8643b612d451b3737433e9c7e0a
      
https://github.com/qemu/qemu/commit/817c60457f41e8643b612d451b3737433e9c7e0a
  Author: Juan Quintela <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M qmp-commands.hx

  Log Message:
  -----------
  migration: Improve QMP documentation

Signed-off-by: Juan Quintela <address@hidden>
Reviewed-by: Eric Blake <address@hidden>


  Commit: e344b8a16de429ada3d9126f26e2a96d71348356
      
https://github.com/qemu/qemu/commit/e344b8a16de429ada3d9126f26e2a96d71348356
  Author: David Gibson <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M include/migration/vmstate.h
    M savevm.c

  Log Message:
  -----------
  savevm: Add VMSTATE_UINT64_EQUAL helpers

The savevm code already includes a number of *_EQUAL helpers which act as
sanity checks verifying that the configuration of the saved state matches
that of the machine we're loading into to work.  Variants already exist
for 8 bit 16 bit and 32 bit integers, but not 64 bit integers.  This patch
fills that hole, adding a UINT64 version.

Signed-off-by: David Gibson <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: d58f5598342ffebe6c6278d8b90792060fca4792
      
https://github.com/qemu/qemu/commit/d58f5598342ffebe6c6278d8b90792060fca4792
  Author: David Gibson <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M hw/hw.h
    M include/migration/vmstate.h

  Log Message:
  -----------
  savevm: Add VMSTATE_UINTTL_EQUAL helper

This adds an _EQUAL VMSTATE helper for target_ulongs, defined in terms of
VMSTATE_UINT32_EQUAL or VMSTATE_UINT64_EQUAL as appropriate.

Signed-off-by: David Gibson <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 213945e4d753b5f214468ff746d65fa76e21dbd1
      
https://github.com/qemu/qemu/commit/213945e4d753b5f214468ff746d65fa76e21dbd1
  Author: David Gibson <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M include/migration/vmstate.h
    M savevm.c

  Log Message:
  -----------
  savevm: Add VMSTATE_FLOAT64 helpers

The current savevm code includes VMSTATE helpers for a number of commonly
used data types, but not for the float64 type used by the internal floating
point emulation code.  This patch fixes the deficiency.

Signed-off-by: David Gibson <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 8474a9dd6757be064bf4b35f422b4640d1cca0a5
      
https://github.com/qemu/qemu/commit/8474a9dd6757be064bf4b35f422b4640d1cca0a5
  Author: David Gibson <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M include/migration/vmstate.h

  Log Message:
  -----------
  savevm: Add VMSTATE_STRUCT_VARRAY_POINTER_UINT32

Currently the savevm code contains a VMSTATE_STRUCT_VARRAY_POINTER_INT32
helper (a variably sized array with the number of elements in an int32_t),
but not VMSTATE_STRUCT_VARRAY_POINTER_UINT32 (... with the number of
elements in a uint32_t).  This patch (trivially) fixes the deficiency.

Signed-off-by: David Gibson <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 377e2cb96b76c2b0023c1acc7230bf3a9e9f9f40
      
https://github.com/qemu/qemu/commit/377e2cb96b76c2b0023c1acc7230bf3a9e9f9f40
  Author: David Gibson <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M include/migration/vmstate.h

  Log Message:
  -----------
  savevm: Fix bugs in the VMSTATE_VBUFFER_MULTIPLY definition

The VMSTATE_BUFFER_MULTIPLY macro is misnamed - it actually specifies
a variably sized buffer with VMS_VBUFFER, so should be named
VMSTATE_VBUFFER_MULTIPLY.  This patch fixes this (the macro had no current
users under either name).

In addition, unlike the other VMSTATE_VBUFFER variants, this macro did not
specify VMS_POINTER.  This patch fixes this bug as well.

Signed-off-by: David Gibson <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: c61ca00ada744eb24825be2ba4d6ba8fe3a870a4
      
https://github.com/qemu/qemu/commit/c61ca00ada744eb24825be2ba4d6ba8fe3a870a4
  Author: Peter Lieven <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M arch_init.c
    M include/qemu-common.h

  Log Message:
  -----------
  move vector definitions to qemu-common.h

vector optimizations will now be used at various places
not just in is_dup_page() in arch_init.c

Signed-off-by: Peter Lieven <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 41a259bd2b1796ddabdae600ee539269a7ddb6a5
      
https://github.com/qemu/qemu/commit/41a259bd2b1796ddabdae600ee539269a7ddb6a5
  Author: Peter Lieven <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M include/qemu-common.h
    M util/cutils.c

  Log Message:
  -----------
  cutils: add a function to find non-zero content in a buffer

this adds buffer_find_nonzero_offset() which is a SSE2/Altivec
optimized function that searches for non-zero content in a
buffer.

the function starts full unrolling only after the first few chunks have
been checked one by one. analyzing real memory page data has revealed
that non-zero pages are non-zero within the first 256-512 bits in
most cases. as this function is also heavily used to check for zero memory
pages this tweak has been made to avoid the high setup costs of the fully
unrolled check for non-zero pages.

due to the optimizations used in the function there are restrictions
on buffer address and search length. the function
can_use_buffer_find_nonzero_content() can be used to check if
the function can be used safely.

Signed-off-by: Peter Lieven <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 56ded708ec38e4cb75a7c7357480ca34c0dc6875
      
https://github.com/qemu/qemu/commit/56ded708ec38e4cb75a7c7357480ca34c0dc6875
  Author: Peter Lieven <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M util/cutils.c

  Log Message:
  -----------
  buffer_is_zero: use vector optimizations if possible

performance gain on SSE2 is approx. 20-25%. altivec
is not tested. performance for unsigned long arithmetic
is unchanged.

Signed-off-by: Peter Lieven <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Orit Wasserman <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 49f676a00ab540fac1d2008be26434cf85607722
      
https://github.com/qemu/qemu/commit/49f676a00ab540fac1d2008be26434cf85607722
  Author: Peter Lieven <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M util/bitops.c

  Log Message:
  -----------
  bitops: unroll while loop in find_next_bit()

this patch adopts the loop unrolling idea of bitmap_is_zero() to
speed up the skipping of large areas with zeros in find_next_bit().

this routine is extensively used to find dirty pages in
live migration.

testing only the find_next_bit performance on a zeroed bitfield
the loop onrolling decreased executing time by approx. 50% on x86_64.

Signed-off-by: Peter Lieven <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 3edcd7e6ebae3ef0ac178eed5f4225803159562d
      
https://github.com/qemu/qemu/commit/3edcd7e6ebae3ef0ac178eed5f4225803159562d
  Author: Peter Lieven <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M arch_init.c

  Log Message:
  -----------
  migration: search for zero instead of dup pages

virtually all dup pages are zero pages. remove
the special is_dup_page() function and use the
optimized buffer_find_nonzero_offset() function
instead.

here buffer_find_nonzero_offset() is used directly
to avoid the unnecssary additional checks in
buffer_is_zero().

raw performace gain checking 1 GByte zeroed memory
over is_dup_page() is approx. 10-12% with SSE2
and 8-10% with unsigned long arithmedtic.

Signed-off-by: Peter Lieven <address@hidden>
Reviewed-by: Orit Wasserman <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 78d07ae7ac74bcc7f79aeefbaff17fb142f44b4d
      
https://github.com/qemu/qemu/commit/78d07ae7ac74bcc7f79aeefbaff17fb142f44b4d
  Author: Peter Lieven <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M arch_init.c

  Log Message:
  -----------
  migration: add an indicator for bulk state of ram migration

the first round of ram transfer is special since all pages
are dirty and thus all memory pages are transferred to
the target. this patch adds a boolean variable to track
this stage.

Signed-off-by: Peter Lieven <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Orit Wasserman <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: f1c72795af573b24a7da5eb52375c9aba8a37972
      
https://github.com/qemu/qemu/commit/f1c72795af573b24a7da5eb52375c9aba8a37972
  Author: Peter Lieven <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M arch_init.c
    M hmp.c
    M include/migration/migration.h
    M migration.c
    M qapi-schema.json
    M qmp-commands.hx

  Log Message:
  -----------
  migration: do not sent zero pages in bulk stage

during bulk stage of ram migration if a page is a
zero page do not send it at all.
the memory at the destination reads as zero anyway.

even if there is an madvise with QEMU_MADV_DONTNEED
at the target upon receipt of a zero page I have observed
that the target starts swapping if the memory is overcommitted.
it seems that the pages are dropped asynchronously.

this patch also updates QMP to return the number of
skipped pages in MigrationStats.

Signed-off-by: Peter Lieven <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 70c8652bf3c1fea79b7b68864e86926715c49261
      
https://github.com/qemu/qemu/commit/70c8652bf3c1fea79b7b68864e86926715c49261
  Author: Peter Lieven <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M arch_init.c

  Log Message:
  -----------
  migration: do not search dirty pages in bulk stage

avoid searching for dirty pages just increment the
page offset. all pages are dirty anyway.

Signed-off-by: Peter Lieven <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Orit Wasserman <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 5cc11c46cf187c7d5306b68e730ec0d372cd7ef0
      
https://github.com/qemu/qemu/commit/5cc11c46cf187c7d5306b68e730ec0d372cd7ef0
  Author: Peter Lieven <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M arch_init.c

  Log Message:
  -----------
  migration: use XBZRLE only after bulk stage

at the beginning of migration all pages are marked dirty and
in the first round a bulk migration of all pages is performed.

currently all these pages are copied to the page cache regardless
of whether they are frequently updated or not. this doesn't make sense
since most of these pages are never transferred again.

this patch changes the XBZRLE transfer to only be used after
the bulk stage has been completed. that means a page is added
to the page cache the second time it is transferred and XBZRLE
can benefit from the third time of transfer.

since the page cache is likely smaller than the number of pages
it's also likely that in the second round the page is missing in the
cache due to collisions in the bulk phase.

on the other hand a lot of unnecessary mallocs, memdups and frees
are saved.

the following results have been taken earlier while executing
the test program from docs/xbzrle.txt. (+) with the patch and (-)
without. (thanks to Eric Blake for reformatting and comments)

+ total time: 22185 milliseconds
- total time: 22410 milliseconds

Shaved 0.3 seconds, better than 1%!

+ downtime: 29 milliseconds
- downtime: 21 milliseconds

Not sure why downtime seemed worse, but probably not the end of the world.

+ transferred ram: 706034 kbytes
- transferred ram: 721318 kbytes

Fewer bytes sent - good.

+ remaining ram: 0 kbytes
- remaining ram: 0 kbytes
+ total ram: 1057216 kbytes
- total ram: 1057216 kbytes
+ duplicate: 108556 pages
- duplicate: 105553 pages
+ normal: 175146 pages
- normal: 179589 pages
+ normal bytes: 700584 kbytes
- normal bytes: 718356 kbytes

Fewer normal bytes...

+ cache size: 67108864 bytes
- cache size: 67108864 bytes
+ xbzrle transferred: 3127 kbytes
- xbzrle transferred: 630 kbytes

...and more compressed pages sent - good.

+ xbzrle pages: 117811 pages
- xbzrle pages: 21527 pages
+ xbzrle cache miss: 18750
- xbzrle cache miss: 179589

And very good improvement on the cache miss rate.

+ xbzrle overflow : 0
- xbzrle overflow : 0

Signed-off-by: Peter Lieven <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Orit Wasserman <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: d913829f0fd8451abcb1fd9d6dfce5586d9d7e10
      
https://github.com/qemu/qemu/commit/d913829f0fd8451abcb1fd9d6dfce5586d9d7e10
  Author: Orit Wasserman <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M include/migration/qemu-file.h

  Log Message:
  -----------
  Add QemuFileWritevBuffer QemuFileOps

This will allow us to write an iovec

Signed-off-by: Orit Wasserman <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 28085f7b4d06970efa004257fcef013caf495a08
      
https://github.com/qemu/qemu/commit/28085f7b4d06970efa004257fcef013caf495a08
  Author: Orit Wasserman <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M savevm.c

  Log Message:
  -----------
  Add socket_writev_buffer function

Signed-off-by: Orit Wasserman <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 7d8a30bb98e89c203b3d2289ab0638c38bbeb7c1
      
https://github.com/qemu/qemu/commit/7d8a30bb98e89c203b3d2289ab0638c38bbeb7c1
  Author: Orit Wasserman <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M savevm.c

  Log Message:
  -----------
  Update bytes_xfer in qemu_put_byte

Signed-off-by: Orit Wasserman <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: b3ea2bdb792f6d961ba3adf45cf1f0c63c61e09d
      
https://github.com/qemu/qemu/commit/b3ea2bdb792f6d961ba3adf45cf1f0c63c61e09d
  Author: Orit Wasserman <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M savevm.c

  Log Message:
  -----------
  Store the data to send also in iovec

All data is still copied into the static buffer.
Adjacent iovecs are coalesced so we send one big buffer
instead of many small buffers.

Signed-off-by: Orit Wasserman <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: cb88aa88d7e96cd12328915b33bf4a1bc054aa3f
      
https://github.com/qemu/qemu/commit/cb88aa88d7e96cd12328915b33bf4a1bc054aa3f
  Author: Orit Wasserman <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M savevm.c

  Log Message:
  -----------
  Use writev ops if available

Update qemu_fflush and stdio_close to use writev ops if they are available
Use the buffers stored in the iovec.

Signed-off-by: Orit Wasserman <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 6181ec245529e0d40ac669fe3044eef3a9e19610
      
https://github.com/qemu/qemu/commit/6181ec245529e0d40ac669fe3044eef3a9e19610
  Author: Orit Wasserman <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M include/migration/qemu-file.h
    M savevm.c

  Log Message:
  -----------
  Add qemu_put_buffer_async

This allows us to add a buffer to the iovec to send without copying it
into the static buffer, the buffer will be sent later when qemu_fflush is 
called.

Signed-off-by: Orit Wasserman <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: 500f0061d628b52220038939728f0d7aee634468
      
https://github.com/qemu/qemu/commit/500f0061d628b52220038939728f0d7aee634468
  Author: Orit Wasserman <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M arch_init.c

  Log Message:
  -----------
  Use qemu_put_buffer_async for guest memory pages

This will remove an unneeded copy of guest memory pages.
For the page header and device state we still copy the data to the
static buffer the other option is to allocate the memory on demand
which is more expensive.

Signed-off-by: Orit Wasserman <address@hidden>
Signed-off-by: Juan Quintela <address@hidden>


  Commit: cd18720a294bd7244ffda719677dd9c737317b67
      
https://github.com/qemu/qemu/commit/cd18720a294bd7244ffda719677dd9c737317b67
  Author: Anthony Liguori <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M include/char/char.h
    M qemu-char.c

  Log Message:
  -----------
  char: introduce a blocking version of qemu_chr_fe_write

Signed-off-by: Anthony Liguori <address@hidden>


  Commit: 0fbf01fe911e84c588be41636570f46f597113d6
      
https://github.com/qemu/qemu/commit/0fbf01fe911e84c588be41636570f46f597113d6
  Author: Anthony Liguori <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M qtest.c

  Log Message:
  -----------
  qtest: use synchronous I/O for char device

Peter reported that rtc-test would periodically hang.  It turns out
this was due to an EAGAIN occurring on qemu_chr_fe_write.

Instead of heavily refactoring qtest, just use a synchronous version
of the write operation for qemu_chr_fe_write to address this problem.

Reported-by: Peter Maydell <address@hidden>
Signed-off-by: Anthony Liguori <address@hidden>


  Commit: b57ed9bf075e33cdd2f9eb545ff555301dd57221
      
https://github.com/qemu/qemu/commit/b57ed9bf075e33cdd2f9eb545ff555301dd57221
  Author: Cornelia Huck <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M hw/s390x/s390-virtio-ccw.c

  Log Message:
  -----------
  virtio-ccw: Queue sanity check for notify hypercall.

Verify that the virtio-ccw notify hypercall passed a reasonable
value for queue.

Cc: address@hidden
Reported-by: Alexander Graf <address@hidden>
Signed-off-by: Cornelia Huck <address@hidden>


  Commit: fad5593ca646010d3cb83f7926d78dd30c20c610
      
https://github.com/qemu/qemu/commit/fad5593ca646010d3cb83f7926d78dd30c20c610
  Author: Anthony Liguori <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M hw/s390x/s390-virtio-ccw.c

  Log Message:
  -----------
  Merge remote-tracking branch 'cohuck/virtio-ccw-upstr' into staging

# By Cornelia Huck
# Via Cornelia Huck
* cohuck/virtio-ccw-upstr:
  virtio-ccw: Queue sanity check for notify hypercall.


  Commit: 18501ae6e825d8da72369fd091018ef71071bd87
      
https://github.com/qemu/qemu/commit/18501ae6e825d8da72369fd091018ef71071bd87
  Author: Anthony Liguori <address@hidden>
  Date:   2013-03-26 (Tue, 26 Mar 2013)

  Changed paths:
    M arch_init.c
    M hmp.c
    M hw/hw.h
    M include/migration/migration.h
    M include/migration/qemu-file.h
    M include/migration/vmstate.h
    M include/qemu-common.h
    M migration.c
    M qapi-schema.json
    M qmp-commands.hx
    M savevm.c
    M util/bitops.c
    M util/cutils.c

  Log Message:
  -----------
  Merge remote-tracking branch 'quintela/migration.next' into staging

# By Peter Lieven (9) and others
# Via Juan Quintela
* quintela/migration.next: (22 commits)
  Use qemu_put_buffer_async for guest memory pages
  Add qemu_put_buffer_async
  Use writev ops if available
  Store the data to send also in iovec
  Update bytes_xfer in qemu_put_byte
  Add socket_writev_buffer function
  Add QemuFileWritevBuffer QemuFileOps
  migration: use XBZRLE only after bulk stage
  migration: do not search dirty pages in bulk stage
  migration: do not sent zero pages in bulk stage
  migration: add an indicator for bulk state of ram migration
  migration: search for zero instead of dup pages
  bitops: unroll while loop in find_next_bit()
  buffer_is_zero: use vector optimizations if possible
  cutils: add a function to find non-zero content in a buffer
  move vector definitions to qemu-common.h
  savevm: Fix bugs in the VMSTATE_VBUFFER_MULTIPLY definition
  savevm: Add VMSTATE_STRUCT_VARRAY_POINTER_UINT32
  savevm: Add VMSTATE_FLOAT64 helpers
  savevm: Add VMSTATE_UINTTL_EQUAL helper
  ...


Compare: https://github.com/qemu/qemu/compare/e769bdc26ded...18501ae6e825