[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-devel] [PATCH v2 00/27] Convert migration to QIOChannel & support
From: |
Daniel P. Berrange |
Subject: |
[Qemu-devel] [PATCH v2 00/27] Convert migration to QIOChannel & support TLS |
Date: |
Tue, 16 Feb 2016 16:14:51 +0000 |
This is an update of patches that were previously posted
FYI: https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg00829.html
v1: https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg01914.html
The primary goal of this series of patches is to support TLS on the
migration data channel. The bulk of the work in this series though,
is converting the various QEMUFile implementations to be based on the
new QIOChannel framework.
At the end of this current series there is just one remaining impl
of QEMUFileOps that is not based on QIOChannel - the one in savevm.c
that is using BlockDriverState. It would be possible to create a
QIOChannel wrapper around BlockDriverState too, at which point all
QEMUFile impls would be QIOChannel based. This would then let us
cut out the QEMUFileOps driver callbacks entirely and thus simply
code even more. This patch series is already too large, so I left
that for now.
The first 6 patchs are some basic clean ups to QEMUFile code
The 7th patch introduces the QIOChannel based QEMUFile impl
and the 8th adds helpers for using it to start migrations.
Patch 9 adds very long overdue support for reporting errors
during migration back to the management app, which is critical
for TLS otherwise it is impossible to debug any failures.
Patches 10-17 convert the various migration protocols to use
the QIOChannel based QEMUFile impl. In this refactoring the
TCP and UNIX implementations were able to be merged into a
generic sockets impl.
Patches 18-21 remove the now unused QEMUFile impls that do
not use QIOChanel
Patches 22 & 23 do some more cleanup
Patch 24 defines some new migration parameters which are used
to enable use of TLS
Patch 25 actually implements support for TLS with migration,
working with tcp, unix, fd and exec migration backend protocols.
Only RDMA is unsupported with TLS. The commit message shows the
example usage via the HMP
Patches 26 & 27 do some final cleanup.
Overall we have a net win of deleting ~350 lines of code,
despite adding more features, which is always nice.
I have been testing the various migration protocols, including
RDMA and appear to be still functional.
In terms of performance, I have tested TCP with TLS migration
enabled over a 10 Gig-E network interface.
With plain TCP we were able to reach 8500mbs (according to
'info migrate' stats).
With TCP and TLS enabled, we are only able to reach 1800 mbs.
IOW, we can max out 1 Gig-E NICs with TLS, but not 10 Gig-E
where we only reach 21% of potential plain text throughput.
The source host migration thread is only hitting 60% CPU
utilization, but the target host incoming migration thread
is hitting 100% CPU.
The source migration thread is dominated solely by GNUTLS
AES encryption functions as would be expected.
The target migration thread is dominated by the same GNUTLS
AES encryption functions, but also memcpy(). IIUC, the memcpy
is QEMU generic migration code copying RAM pages into place.
In talking with Dave Gilbert we thought it might be possible
to use two threads for incoming migration on the target host.
The first would be responsible for doing network I/O into
local buffers, including the TLS decryption. The second
would be responsible for processing the data. That way the
memcpy() of RAM would move into another thread, allowing the
first thread to spend 100% of its time doing TLS decryption.
If we assume the decryption + encryption take equal amounts
of time, then it ought to let us raise TLS throughput from
1800 mbs, to approx 3000 mbs. Still a good way off 8500mbs
from non-TLS migration, but a worth while improvement none
the less.
NB, these TLS migration results were on a CPU with native AES
instructionset support. CPUs with AES instructions would be
even worse performance.
Changed in v2:
- Switch to setting migration parameters for TLS instead
of adding to the URI syntax
- Support TLS over tcp, unix, fd, and socket protocols, not
just tcp
- Allow passing in a hostname override for x509 cert checks
- Enable error reporting for outgoing migration problems
- Fix inverted I/O direction in post-copy code
- Use uint8_t / size_t in post-copy conversion instead of
casting types
- Merge unix and tcp driver implementations
- Use tracepoints instead of DPRINTF
- Use error_report for incoming migration problems
- Fix broken logic in RDMA read conversion
- Add missing I/O callback & set_blocking API callbacks
for RMDA QIOChannel impl
- Moved socket vs file FD detection to QIOChannel common
code
Daniel P. Berrange (27):
s390: use FILE instead of QEMUFile for creating text file
migration: remove use of qemu_bufopen from vmstate tests
migration: ensure qemu_fflush() always writes full data amount
migration: split migration hooks out of QEMUFileOps
migration: introduce set_blocking function in QEMUFileOps
migration: force QEMUFile to blocking mode for outgoing migration
migration: introduce a new QEMUFile impl based on QIOChannel
migration: add helpers for creating QEMUFile from a QIOChannel
migration: add reporting of errors for outgoing migration
migration: convert post-copy to use QIOChannelBuffer
migration: convert unix socket protocol to use QIOChannel
migration: rename unix.c to socket.c
migration: convert tcp socket protocol to use QIOChannel
migration: convert fd socket protocol to use QIOChannel
migration: convert exec socket protocol to use QIOChannel
migration: convert RDMA to use QIOChannel interface
migration: convert savevm to use QIOChannel for writing to files
migration: delete QEMUFile buffer implementation
migration: delete QEMUSizedBuffer struct
migration: delete QEMUFile sockets implementation
migration: delete QEMUFile stdio implementation
migration: move definition of struct QEMUFile back into qemu-file.c
migration: don't use an array for storing migrate parameters
migration: define 'tls-creds' and 'tls-hostname' migration parameters
migration: add support for encrypting data with TLS
migration: remove support for non-iovec based write handlers
migration: remove qemu_get_fd method from QEMUFile
docs/migration.txt | 4 +-
hmp-commands.hx | 15 ++
hmp.c | 56 +++++
hmp.h | 1 +
hw/s390x/s390-skeys.c | 26 +--
include/migration/migration.h | 26 ++-
include/migration/qemu-file.h | 57 ++---
include/qapi/error.h | 2 +-
include/qemu/typedefs.h | 1 -
include/sysemu/sysemu.h | 2 +-
migration/Makefile.objs | 7 +-
migration/exec.c | 64 +++---
migration/fd.c | 76 +++----
migration/migration.c | 155 +++++++++-----
migration/qemu-file-buf.c | 464 -----------------------------------------
migration/qemu-file-channel.c | 202 ++++++++++++++++++
migration/qemu-file-internal.h | 53 -----
migration/qemu-file-stdio.c | 196 -----------------
migration/qemu-file-unix.c | 325 -----------------------------
migration/qemu-file.c | 110 +++++-----
migration/ram.c | 6 +-
migration/rdma.c | 380 ++++++++++++++++++++++++---------
migration/savevm.c | 63 ++----
migration/socket.c | 182 ++++++++++++++++
migration/tcp.c | 103 ---------
migration/tls.c | 160 ++++++++++++++
migration/unix.c | 103 ---------
qapi-schema.json | 63 +++++-
tests/Makefile | 6 +-
tests/test-vmstate.c | 55 ++---
trace-events | 25 ++-
util/error.c | 2 +-
32 files changed, 1317 insertions(+), 1673 deletions(-)
delete mode 100644 migration/qemu-file-buf.c
create mode 100644 migration/qemu-file-channel.c
delete mode 100644 migration/qemu-file-internal.h
delete mode 100644 migration/qemu-file-stdio.c
delete mode 100644 migration/qemu-file-unix.c
create mode 100644 migration/socket.c
delete mode 100644 migration/tcp.c
create mode 100644 migration/tls.c
delete mode 100644 migration/unix.c
--
2.5.0
- [Qemu-devel] [PATCH v2 00/27] Convert migration to QIOChannel & support TLS,
Daniel P. Berrange <=
- [Qemu-devel] [PATCH v2 02/27] migration: remove use of qemu_bufopen from vmstate tests, Daniel P. Berrange, 2016/02/16
- [Qemu-devel] [PATCH v2 01/27] s390: use FILE instead of QEMUFile for creating text file, Daniel P. Berrange, 2016/02/16
- [Qemu-devel] [PATCH v2 03/27] migration: ensure qemu_fflush() always writes full data amount, Daniel P. Berrange, 2016/02/16
- [Qemu-devel] [PATCH v2 05/27] migration: introduce set_blocking function in QEMUFileOps, Daniel P. Berrange, 2016/02/16
- [Qemu-devel] [PATCH v2 04/27] migration: split migration hooks out of QEMUFileOps, Daniel P. Berrange, 2016/02/16
- [Qemu-devel] [PATCH v2 07/27] migration: introduce a new QEMUFile impl based on QIOChannel, Daniel P. Berrange, 2016/02/16
- [Qemu-devel] [PATCH v2 06/27] migration: force QEMUFile to blocking mode for outgoing migration, Daniel P. Berrange, 2016/02/16
- [Qemu-devel] [PATCH v2 10/27] migration: convert post-copy to use QIOChannelBuffer, Daniel P. Berrange, 2016/02/16
- [Qemu-devel] [PATCH v2 11/27] migration: convert unix socket protocol to use QIOChannel, Daniel P. Berrange, 2016/02/16
- [Qemu-devel] [PATCH v2 13/27] migration: convert tcp socket protocol to use QIOChannel, Daniel P. Berrange, 2016/02/16