On Thu, Nov 03, 2022 at 11:06:23PM +0530, manish.mishra wrote:
On 03/11/22 10:57 pm, Daniel P. Berrangé wrote:
On Thu, Nov 03, 2022 at 10:04:54PM +0530, manish.mishra wrote:
On 03/11/22 2:59 pm, Daniel P. Berrangé wrote:
On Thu, Nov 03, 2022 at 02:50:25PM +0530, manish.mishra wrote:
On 01/11/22 9:15 pm, Daniel P. Berrangé wrote:
On Tue, Nov 01, 2022 at 09:10:14PM +0530, manish.mishra wrote:
On 01/11/22 8:21 pm, Daniel P. Berrangé wrote:
On Tue, Nov 01, 2022 at 02:30:29PM +0000, manish.mishra wrote:
diff --git a/migration/migration.c b/migration/migration.c
index 739bb683f3..f4b6f278a9 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -733,31 +733,40 @@ void migration_ioc_process_incoming(QIOChannel *ioc,
Error **errp)
{
MigrationIncomingState *mis = migration_incoming_get_current();
Error *local_err = NULL;
- bool start_migration;
QEMUFile *f;
+ bool default_channel = true;
+ uint32_t channel_magic = 0;
+ int ret = 0;
- if (!mis->from_src_file) {
- /* The first connection (multifd may have multiple) */
+ if (migrate_use_multifd() && !migration_in_postcopy()) {
+ ret = qio_channel_read_peek_all(ioc, (void *)&channel_magic,
+ sizeof(channel_magic), &local_err);
+
+ if (ret != 1) {
+ error_propagate(errp, local_err);
+ return;
+ }
....and thus this will fail for TLS channels AFAICT.
Yes, thanks for quick review Daniel. You pointed this earlier too, sorry missed
it, will put another check !migrate_use_tls() in V2.
But we need this problem fixed with TLS too, so just excluding it
isn't right. IMHO we need to modify the migration code so we can
read the magic earlier, instead of peeking.
With regards,
Daniel
Hi Daniel, I was trying tls migrations. What i see is that tls session
creation does handshake. So if we read ahead in ioc_process_incoming
for default channel. Because client sends magic only after multiFD
channels are setup, which too requires tls handshake.
By the time we get to migrate_ioc_process_incoming, the TLS handshake
has already been performed.
migration_channel_process_incoming
-> migration_ioc_process_incoming
vs
migration_channel_process_incoming
-> migration_tls_channel_process_incoming
-> migration_tls_incoming_handshake
-> migration_channel_process_incoming
-> migration_ioc_process_incoming
Yes sorry i thought we block on source side till handshake is done but that is
not true. I checked then why that deadlock is happening. So this where the
dealock is happening.
static int ram_save_setup(QEMUFile *f, void *opaque) {
+
+
ram_control_before_iterate(f, RAM_CONTROL_SETUP);
ram_control_after_iterate(f, RAM_CONTROL_SETUP);
ret = multifd_send_sync_main(f);
if (ret < 0) {
return ret;
}
qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
qemu_fflush(f);
return 0;
}
Now if we block in migration_ioc_process_incoming for reading magic
value from channel, which is actually sent by client when this
qemu_fflush is done. Before this qemu_fflush we wait for
multifd_send_sync_main which actually requires that tls handshake is
done for multiFD channels as it blocks on sem_sync which posted by
multifd_send_thread which is called after handshake||. But then on
destination side we are blocked in migration_ioc_process_incoming()
waiting to read something from default channel hence handshake for
multiFD channels can not happen. This to me looks unresolvable
whatever way we try to manipulate stream until we do some changes
on source side.
The TLS handshake is already complete when migration_ioc_process_incoming
is blocking on read.
Regardless of which channel we're talking about, thue TLS handshake is
always performed & finished before we try to either send or recv any
data.
Yes Daniel, agree on that, in this case tls handshake is done for
default channel so we went in migration_ioc_process_incoming for
default channel. But if we try to read some data there, it does not
get because data on default channel is not yet flushed by source.
From source side data in flushed in above function i pointed. Which
blocks for multiFD channels to be setup with handshake, before
flushing data. I mean data is sent only when buffer is full or
explicitly flushed, till then it is just in buffer. But multiFD
handshake can not complete until we return from
migration_ioc_process_incoming for default channel which infintely
waits because nothing is sent yet on channel.
On the source side, if we're in ram_save_setup then the TLS
handshake is already complete for the main channel. In fact
I think the TLS handshake should act as a serialization
point that prevents the entire bug. It should be guaranteed
that the main channel is fully received by the dest, and
transferring data, before we even get to establishing the
multifd channels.