[Qemu-commits] [qemu/qemu] eed1cc: migration: delay postcopy paused state

[GitHub]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: eed1cc7866ae35bd28926d75447dd6076bd6bb96
      
https://github.com/qemu/qemu/commit/eed1cc7866ae35bd28926d75447dd6076bd6bb96
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M migration/savevm.c

  Log Message:
  -----------
  migration: delay postcopy paused state

Before this patch we firstly setup the postcopy-paused state then we
clean up the QEMUFile handles.  That can be racy if there is a very fast
"migrate-recover" command running in parallel.  Fix that up.

Reported-by: Peter Maydell <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 81e620531fa842f760086964ca1b8657ae6c07ba
      
https://github.com/qemu/qemu/commit/81e620531fa842f760086964ca1b8657ae6c07ba
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M migration/migration.c
    M migration/ram.c
    M migration/ram.h

  Log Message:
  -----------
  migration: move income process out of multifd

Move the call to migration_incoming_process() out of multifd code.  It's
a bit strange that we can migration generic calls in multifd code.
Instead, let multifd_recv_new_channel() return a boolean showing whether
it's ready to continue the incoming migration.

Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 884835fa1e38066e2596224375bb35ac6686be4d
      
https://github.com/qemu/qemu/commit/884835fa1e38066e2596224375bb35ac6686be4d
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M migration/migration.c

  Log Message:
  -----------
  migration: unbreak postcopy recovery

The whole postcopy recovery logic was accidentally broken.  We need to
fix it in two steps.

This is the first step that we should do the recovery when needed.  It
was bypassed before after commit 36c2f8be2c.

Introduce postcopy_try_recovery() helper for the postcopy recovery
logic.  Call it both in migration_fd_process_incoming() and
migration_ioc_process_incoming().

Fixes: 36c2f8be2c ("migration: Delay start of migration main routines")
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: a429e7f4887313370ed2d0d3607b7e6bdcfb0e1b
      
https://github.com/qemu/qemu/commit/a429e7f4887313370ed2d0d3607b7e6bdcfb0e1b
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M migration/exec.c
    M migration/fd.c
    M migration/migration.c
    M migration/socket.c

  Log Message:
  -----------
  migration: unify incoming processing

This is the 2nd patch to unbreak postcopy recovery.

Let's unify the migration_incoming_process() call at a single place
rather than calling it in connection setup codes.  This fixes a problem
that we will go into incoming migration procedure even if we are trying
to recovery from a paused postcopy migration.

Fixes: 36c2f8be2c ("migration: Delay start of migration main routines")
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 1aa8367861645c3c0917f585fe14b1b6b23b0f66
      
https://github.com/qemu/qemu/commit/1aa8367861645c3c0917f585fe14b1b6b23b0f66
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M migration/ram.c

  Log Message:
  -----------
  migration: simplify check to use qemu file buffer

Firstly, renaming the old matching_page_sizes variable to
matches_target_page_size, which suites more to what it did (it only
checks against target page size rather than multiple page sizes).
Meanwhile, simplify the check logic a bit, and enhance the comments.
Should have no functional change.

Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: fd037a656aca23dc3c47aa8c9de261bec6b8aad0
      
https://github.com/qemu/qemu/commit/fd037a656aca23dc3c47aa8c9de261bec6b8aad0
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M migration/savevm.c

  Log Message:
  -----------
  migration: loosen recovery check when load vm

We were checking against -EIO, assuming that it will cover all IO
failures.  But actually it is not.  One example is that in
qemu_loadvm_section_start_full() we can have tons of places that will
return -EINVAL even if the error is caused by IO failures on the
network.

Let's loosen the recovery check logic here to cover all the error cases
happened by removing the explicit check against -EIO.  After all we
won't lose anything here if any other failure happened.

Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: a725ef9fe36424351faf51696c3fc441ded13f35
      
https://github.com/qemu/qemu/commit/a725ef9fe36424351faf51696c3fc441ded13f35
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M migration/ram.c

  Log Message:
  -----------
  migration: fix incorrect bitmap size calculation

The calculation on size of received bitmap is incorrect for postcopy
recovery.  Here we wanted to let the size to cover all the valid bits in
the bitmap, we should use DIV_ROUND_UP() instead of a division.

For example, a RAMBlock with size=4K (which contains only one single 4K
page) will have nbits=1, then nbits/8=0, then the real bitmap won't be
sent to source at all.

Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 3c9928d9f9b6b5717fa8e53e9441c6b041d6554a
      
https://github.com/qemu/qemu/commit/3c9928d9f9b6b5717fa8e53e9441c6b041d6554a
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M migration/migration.c

  Log Message:
  -----------
  migration: show pause/recover state on dst host

These two states will be missing when doing "query-migrate" on
destination VM.  Add these states so that we can get the query results
as expected.

Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: d131662a1afd44ffd0ced3334fa0cf1f30f87196
      
https://github.com/qemu/qemu/commit/d131662a1afd44ffd0ced3334fa0cf1f30f87196
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M tests/migration-test.c

  Log Message:
  -----------
  tests: introduce migrate_postcopy_* helpers

Separate the old postcopy UNIX socket test into three steps, provide a
helper for each step.  With these helpers, we can do more compliated
tests like postcopy recovery, while keep the codes shared.

Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Reviewed-by: Balamuruhan S <address@hidden>
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>
  Fix up merge with 2e295789 / Skip tests for ppc tcg


  Commit: 7e1d74271c36d12a90608db73ea72aab352b5bb0
      
https://github.com/qemu/qemu/commit/7e1d74271c36d12a90608db73ea72aab352b5bb0
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M tests/migration-test.c

  Log Message:
  -----------
  tests: allow migrate() to take extra flags

For example, we can pass in '"resume": true' to resume a migration.

Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Reviewed-by: Balamuruhan S <address@hidden>
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 2f7074c6fdec5015086f5328f8427097bee86e13
      
https://github.com/qemu/qemu/commit/2f7074c6fdec5015086f5328f8427097bee86e13
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M tests/migration-test.c

  Log Message:
  -----------
  tests: introduce migrate_query*() helpers

Introduce helpers to query migration states and use it.

Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Reviewed-by: Balamuruhan S <address@hidden>
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 2f6d313836591659375db97a2c1de4f28413f12c
      
https://github.com/qemu/qemu/commit/2f6d313836591659375db97a2c1de4f28413f12c
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M tests/migration-test.c

  Log Message:
  -----------
  tests: introduce wait_for_migration_status()

It's generalized from wait_for_migration_complete() to allow us to wait
for any migration status besides failure.

Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Reviewed-by: Balamuruhan S <address@hidden>
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: d5f496407db4444758e3374b95109a5d18eba70b
      
https://github.com/qemu/qemu/commit/d5f496407db4444758e3374b95109a5d18eba70b
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M tests/migration-test.c

  Log Message:
  -----------
  tests: add postcopy recovery test

Test the postcopy recovery procedure by emulating a network failure
using migrate-pause command.

Tested-by: Balamuruhan S <address@hidden>
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 3e81f73c7a1286e251180c19f62829fe5c045e39
      
https://github.com/qemu/qemu/commit/3e81f73c7a1286e251180c19f62829fe5c045e39
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M tests/migration-test.c

  Log Message:
  -----------
  tests: hide stderr for postcopy recovery test

We dumped something when network failure happens.  We should avoid those
messages to be dumped when running the tests:

  $ ./tests/migration-test -p /x86_64/migration/postcopy/recovery
  /x86_64/migration/postcopy/recovery: qemu-system-x86_64: 
check_section_footer: Read section footer failed: -5
  qemu-system-x86_64: Detected IO failure for postcopy. Migration paused.
  qemu-system-x86_64: Detected IO failure for postcopy. Migration paused.
  OK

After the patch:

  $ ./tests/migration-test -p /x86_64/migration/postcopy/recovery
  /x86_64/migration/postcopy/recovery: OK

Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 858b6d62249a9a9510fae6c808a3d2de80e689b5
      
https://github.com/qemu/qemu/commit/858b6d62249a9a9510fae6c808a3d2de80e689b5
  Author: Peter Xu <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M migration/savevm.c

  Log Message:
  -----------
  migration: reorder MIG_CMD_POSTCOPY_RESUME

It was accidently added before MIG_CMD_PACKAGED so it might break
command compatibility when we run postcopy migration between old/new
QEMUs.  Fix that up quickly before the QEMU 3.0 release.

Reported-by: Lukáš Doktor <address@hidden>
Suggested-by: Dr. David Alan Gilbert <address@hidden>
Signed-off-by: Peter Xu <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Juan Quintela <address@hidden>
Signed-off-by: Dr. David Alan Gilbert <address@hidden>


  Commit: 0956ee3e4027d9063b08477a1c4f2b719380aac4
      
https://github.com/qemu/qemu/commit/0956ee3e4027d9063b08477a1c4f2b719380aac4
  Author: Peter Maydell <address@hidden>
  Date:   2018-07-10 (Tue, 10 Jul 2018)

  Changed paths:
    M migration/exec.c
    M migration/fd.c
    M migration/migration.c
    M migration/ram.c
    M migration/ram.h
    M migration/savevm.c
    M migration/socket.c
    M tests/migration-test.c

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20180710a' 
into staging

Migration pull 2018-07-10 (for 3.0)

Migration fixes and migration test fixes, mostly
around postcopy and postcopy recovery

# gpg: Signature made Tue 10 Jul 2018 16:27:19 BST
# gpg:                using RSA key 0516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <address@hidden>"
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A  9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-migration-20180710a:
  migration: reorder MIG_CMD_POSTCOPY_RESUME
  tests: hide stderr for postcopy recovery test
  tests: add postcopy recovery test
  tests: introduce wait_for_migration_status()
  tests: introduce migrate_query*() helpers
  tests: allow migrate() to take extra flags
  tests: introduce migrate_postcopy_* helpers
  migration: show pause/recover state on dst host
  migration: fix incorrect bitmap size calculation
  migration: loosen recovery check when load vm
  migration: simplify check to use qemu file buffer
  migration: unify incoming processing
  migration: unbreak postcopy recovery
  migration: move income process out of multifd
  migration: delay postcopy paused state

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/2b83714d4ea6...0956ee3e4027
      **NOTE:** This service been marked for deprecation: 
https://developer.github.com/changes/2018-04-25-github-services-deprecation/

      Functionality will be removed from GitHub.com on January 31st, 2019.