Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
> On 01/26/2018 10:44 PM, Stefan Hajnoczi wrote:
> > On Thu, Jan 25, 2018 at 06:19:13PM +0800, Wei Wang wrote:
> > > On 01/24/2018 07:40 PM, Stefan Hajnoczi wrote:
> > > > On Tue, Jan 23, 2018 at 09:06:49PM +0800, Wei Wang wrote:
> > > > > On 01/23/2018 07:12 PM, Stefan Hajnoczi wrote:
> > > > > > On Mon, Jan 22, 2018 at 07:09:06PM +0800, Wei Wang wrote:
> > > > > > > On 01/19/2018 09:06 PM, Stefan Hajnoczi wrote:
> > > > > > > 
> > > > > > > 
> > > > > > >     - Suppose in the future there is also a kernel 
> > > > > > > virtio-vhost-user driver as
> > > > > > > other PCI devices, can we unbind the kernel driver first, and 
> > > > > > > then bind the
> > > > > > > device to the dpdk driver? A normal PCI device should be able to 
> > > > > > > smoothly
> > > > > > > switch between the kernel driver and dpdk driver.
> > > > > > It depends what you mean by "smoothly switch".
> > > > > > 
> > > > > > If you mean whether it's possible to go from a kernel driver to
> > > > > > vfio-pci, then the answer is yes.
> > > > > > 
> > > > > > But if the kernel driver has an established vhost-user connection 
> > > > > > then
> > > > > > it will be closed.  This is the same as reconnecting with AF_UNIX
> > > > > > vhost-user.
> > > > > > 
> > > > > Actually not only the case of switching to testpmd after kernel 
> > > > > establishes
> > > > > the connection, but also for several runs of testpmd. That is, if we 
> > > > > run
> > > > > testpmd, then exit testpmd. I think the second run of testpmd won't 
> > > > > work.
> > > > The vhost-user master must reconnect and initialize again (SET_FEATURES,
> > > > SET_MEM_TABLE, etc).  Is your master reconnecting after the AF_UNIX
> > > > connection is closed?
> > > Is this an explicit qmp operation to make the master re-connect?
> > I haven't tested it myself but I'm aware of two modes of operation:
> > 
> > 1. -chardev socket,id=chardev0,...,server
> >     -netdev vhost-user,chardev=chardev0
> > 
> >     When the vhost-user socket is disconnected the peer needs to
> >     reconnect.  In this case no special commands are necessary.
> > 
> >     Here we're relying on DPDK librte_vhost's reconnection behavior.
> > 
> > Or
> > 
> > 2. -chardev socket,id=chardev0,...,reconnect=3
> >     -netdev vhost-user,chardev=chardev0
> > 
> >     When the vhost-user socket is disconnected a new connection attempt
> >     will be made after 3 seconds.
> > 
> > In both cases vhost-user negotiation will resume when the new connection
> > is established.
> > 
> > Stefan
> 
> I've been thinking about the issues, and it looks vhost-pci outperforms in
> this aspect.
> Vhost-pci is like using a mail box. messages are just dropped into the box,
> and whenever vhost-pci pmd gets booted, it can always get the messages from
> the box, the negotiation between vhost-pci pmd and virtio-net is
> asynchronous.
> Virtio-vhost-user is like a phone call, which is a synchronous
> communication. If one side is absent, then the other side will hang on
> without knowing when it could get connected or hang up with messages not
> passed (lost).
> 
> I also think the above solutions won't help. Please see below:
> 
> Background:
> The vhost-user negotiation is split into 2 phases currently. The 1st phase
> happens when the connection is established, and we can find what's done in
> the 1st phase in vhost_user_init(). The 2nd phase happens when the master
> driver is loaded (e.g. run of virtio-net pmd) and set status to the device,
> and we can find what's done in the 2nd phase in vhost_dev_start(), which
> includes sending the memory info and virtqueue info. The socket is
> connected, till one of the QEMU devices exits, so pmd exiting won't end the
> QEMU side socket connection.
>
> Issues:
> Suppose we have both the vhost and virtio-net set up, and vhost pmd <->
> virtio-net pmd communication works well. Now, vhost pmd exits (virtio-net
> pmd is still there). Some time later, we re-run vhost pmd, the vhost pmd
> doesn't know the virtqueue addresses of the virtio-net pmd, unless the
> virtio-net pmd reloads to start the 2nd phase of the vhost-user protocol. So
> the second run of the vhost pmd won't work.

This isn't a problem for virtio-vhost-user since the vhost-pmd resets
the virtio-vhost-user device when it restarts.  The vhost-user AF_UNIX
socket reconnects and negotiation restarts.

Stefan

signature.asc
Description: PGP signature