Re: [PATCH v2 0/1] migration: multifd live migration improvement

[Li Zhang]

On 12/6/21 8:54 PM, Dr. David Alan Gilbert wrote:
> * Li Zhang (lizhang@suse.de) wrote:
> > When testing live migration with multifd channels (8, 16, or a bigger number)
> > and using qemu -incoming (without "defer"), if a network error occurs
> > (for example, triggering the kernel SYN flooding detection),
> > the migration fails and the guest hangs forever.
> >
> > The test environment and the command line is as the following:
> >
> > QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
> > Host OS: SLE 15  with kernel: 5.14.5-1-default
> > Network Card: mlx5 100Gbps
> > Network card: Intel Corporation I350 Gigabit (1Gbps)
> >
> > Source:
> > qemu-system-x86_64 -M q35 -smp 32 -nographic \
> >          -serial telnet:10.156.208.153:4321,server,nowait \
> >          -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
> >          -monitor stdio
> > Dest:
> > qemu-system-x86_64 -M q35 -smp 32 -nographic \
> >          -serial telnet:10.156.208.154:4321,server,nowait \
> >          -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
> >          -monitor stdio \
> >          -incoming tcp:1.0.8.154:4000
> >
> > (qemu) migrate_set_parameter max-bandwidth 100G
> > (qemu) migrate_set_capability multifd on
> > (qemu) migrate_set_parameter multifd-channels 16
> >
> > The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.
> >
> > If a network problem happens, TCP ACK is not received by destination
> > and the destination resets the connection with RST.
> >
> > No.     Time    Source  Destination     Protocol        Length  Info
> > 119     1.021169        1.0.8.153       1.0.8.154       TCP     1410    60166 → 
> > 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 
> > TSecr=1399531897
> > No.     Time    Source  Destination     Protocol        Length  Info
> > 125     1.021181        1.0.8.154       1.0.8.153       TCP     54      4000 → 
> > 60166 [RST] Seq=1 Win=0 Len=0
> >
> > kernel log:
> > [334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. 
> > Sending cookies.  Check SNMP counters.
> > [334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. 
> > Sending cookies.  Check SNMP counters.
> > [334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. 
> > Sending cookies.  Check SNMP counters.
> > [334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. 
> > Sending cookies.  Check SNMP counters.
> > [335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. 
> > Sending cookies.  Check SNMP counters.
> > [335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. 
> > Sending cookies.  Check SNMP counters.
> Should we document somewhere how to avoid that?  Is there something we
> should be doing in the connection code to avoid it?

We should use the command line -incoming defer in QEMU command line 
instead of -incoming ip:port.

And the backlog of the socket will be set as the same as  multifd 
channels,  this problem doesn't happen as far as I test.

If we use --incoming ip:port in the QEMU command line, the backlog of 
the socket is always 1, it will cause the SYN flooding.


> Dave
>
> > There are two problems here:
> > 1. On the send side, the main thread is blocked on qemu_thread_join and
> >     send threads are blocked on sendmsg
> > 2. On receive side, the receive threads are blocked on qemu_sem_wait to
> >     wait for a semaphore.
> >
> > The patch is to fix the first problem, and the guest doesn't hang any more.
> > But there is no better solution to fix the second problem yet.
> >
> > Li Zhang (1):
> >    multifd: Shut down the QIO channels to avoid blocking the send threads
> >      when they are terminated.
> >
> >   migration/multifd.c | 3 +++
> >   1 file changed, 3 insertions(+)
> >
> > --
> > 2.31.1