Re: [PATCH for-5.0?] nbd: Attempt reconnect after server error of ESHUTDOWN

[Eric Blake]

On 4/2/20 8:33 AM, Eric Blake wrote:

> > Then, what about "SHOULD wait until no inflight requests"? We don't do 
> > it either.. Should we?
>
> qemu as server doesn't send NBD_ESHUTDOWN.  It probably should (the way 
> nbdkit does), but that's orthogonal to qemu as client responding to 
> NBD_ESHUTDOWN.

Other things I want to document here based on an IRC conversation with Dave:

Our notion of reconnect-delay has a baked-in notion of timeout, but 
selecting the right timeout can be difficult (how do you know it is long 
enough to catch all the cases you care about where recovery will work, 
but not so long that waiting for an actual timeout is not going to be 
painful when recovery is not possible).  And the qemu block layer 
already has the notion of pausing the guest on certain errors (whether 
that be just on ENOSPC, or on all errors), to give the management all 
the time it needs to resolve the problem and then resume the guest.

There's also the issue of TCP timeouts - if the server manages to send 
shutdown(SHUT_WR) before the connection dies, the client gets an instant 
EOF and can be pretty responsive to the need to start the retry cycle. 
But if the connection dies without a clean shutdown, the client may be 
stuck waiting several seconds for a TCP timeout to occur before 
realizing that things are down (use of TCP keep-alive may or may not 
help here) - management apps may be able to figure out from other means 
when an NBD server is having issues long before qemu itself sees the TCP 
connection go down.  In that case, having a way for the client to 
trigger shutdown(SHUT_RD) in order to speed up disconnection, rather 
than waiting for a TCP timeout, can come in handy.

Or, if we have a multipath scenario, where we know that several IP 
addresses will serve the same underlying storage, we may just need a way 
to reopen an existing NBD blockdev but with a different IP address to 
the server.

All of this implies we may want to add a QMP command to force a given 
NBD blockdev to attempt a reconnect now, rather than having to wait for 
a TCP connection death to let us know that a reconnect is the only way 
forward, or even as a way to make sure that we can resume the guest 
after it was paused due to I/O error.  I don't know if the existing 
'x-blockdev-reopen' can be extended to cover our needs, or if we need 
something completely new.

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org