[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

From: Jason Wang
Subject: Re: [Qemu-devel] vhost-pci and virtio-vhost-user
Date: Fri, 12 Jan 2018 11:32:56 +0800
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0

On 2018年01月11日 23:23, Stefan Hajnoczi wrote:
On Thu, Jan 11, 2018 at 06:57:03PM +0800, Jason Wang wrote:

On 2018年01月11日 00:14, Stefan Hajnoczi wrote:
Hi Wei,
I wanted to summarize the differences between the vhost-pci and
virtio-vhost-user approaches because previous discussions may have been

vhost-pci defines a new virtio device type for each vhost device type
(net, scsi, blk).  It therefore requires a virtio device driver for each
device type inside the slave VM.

Adding a new device type requires:
1. Defining a new virtio device type in the VIRTIO specification.
3. Implementing a new QEMU device model.
2. Implementing a new virtio driver.

virtio-vhost-user is a single virtio device that acts as a vhost-user
protocol transport for any vhost device type.  It requires one virtio
driver inside the slave VM and device types are implemented using
existing vhost-user slave libraries (librte_vhost in DPDK and
libvhost-user in QEMU).

Adding a new device type to virtio-vhost-user involves:
1. Adding any new vhost-user protocol messages to the QEMU
     virtio-vhost-user device model.
2. Adding any new vhost-user protocol messages to the vhost-user slave
3. Implementing the new device slave.

The simplest case is when no new vhost-user protocol messages are
required for the new device.  Then all that's needed for
virtio-vhost-user is a device slave implementation (#3).  That slave
implementation will also work with AF_UNIX because the vhost-user slave
library hides the transport (AF_UNIX vs virtio-vhost-user).  Even
better, if another person has already implemented that device slave to
use with AF_UNIX then no new code is needed for virtio-vhost-user
support at all!

If you compare this to vhost-pci, it would be necessary to design a new
virtio device, implement it in QEMU, and implement the virtio driver.
Much of virtio driver is more or less the same thing as the vhost-user
device slave but it cannot be reused because the vhost-user protocol
isn't being used by the virtio device.  The result is a lot of
duplication in DPDK and other codebases that implement vhost-user

The way that vhost-pci is designed means that anyone wishing to support
a new device type has to become a virtio device designer.  They need to
map vhost-user protocol concepts to a new virtio device type.  This will
be time-consuming for everyone involved (e.g. the developer, the VIRTIO
community, etc).

The virtio-vhost-user approach stays at the vhost-user protocol level as
much as possible.  This way there are fewer concepts that need to be
mapped by people adding new device types.  As a result, it will allow
virtio-vhost-user to keep up with AF_UNIX vhost-user and grow because
it's easier to work with.

What do you think?

So a question is what's the motivation here?

Form what I'm understanding, vhost-pci tries to build a scalable V2V private
datapath. But according to what you describe here, virito-vhost-user tries
to make it possible to implement the device inside another VM. I understand
the goal of vhost-pci could be done on top, but it looks to me it would then
rather similar to the design of Xen driver domain. So I can not figure out
how it can be done in a high performance way.
vhost-pci and virtio-vhost-user both have the same goal.  They allow
a VM to implement a vhost device (net, scsi, blk, etc).

Looks not, if I read the code correctly, vhost-pci has a device implementation in qemu, and in slave VM it only have a vhost-pci-net driver.

This allows
software defined network or storage appliances running inside a VM to
provide I/O services to other VMs.

Well, I think we can do it even with the existed virtio or whatever other emulated device which should not be bounded to any specific kind of device.  And what's more important, according to the kvm 2016 slides of vhost-pci, the motivation of vhost-pci is not building SDN but a chain of VNFs. So bypassing the central vswitch through a private VM2VM path does make sense. (Though whether or not vhost-pci is the best choice is still questionable).

  To the other VMs the devices look
like regular virtio devices.

I'm not sure I understand your reference to the Xen driver domain or

So what proposed here is basically memory sharing and event notification through eventfd, this model have been used by Xen for many years through grant table and event channel. Xen use this to move the backend implementation from dom0 to a driver domain which has direct access to some hardwares. Consider the case of network, it can then implement xen netback inside driver domain which can access hardware NIC directly.

This makes sense for Xen and for performance since driver domain (backend) can access hardware directly and event was triggered through lower overhead hypercall (or it can do busypolling). But for virtio-vhost-user, unless you want SRIOV based solutions inside the slave VM, I believe we won't want to go back to Xen since the hardware virtualization can bring extra overheads.

  Both vhost-pci and virtio-vhost-user work using shared
memory access to the guest RAM of the other VM.  Therefore they can poll
virtqueues and avoid vmexit.  They do also support cross-VM interrupts,
thanks to QEMU setting up irqfd/ioeventfd appropriately on the host.


So in conclusion, consider the complexity, I would suggest to figure out whether or not this (either vhost-pci or virito-vhost-user) is really required before moving ahead. E.g, for VM2VM direct network path, this looks simply an issue of network topology instead of the problem of device, so there's a lot of trick, for vhost-user one can easily image to write an application (or use testpmd) to build a zerocopied VM2VM datapath, isn't this not sufficient for the case?


reply via email to

[Prev in Thread] Current Thread [Next in Thread]