Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators

[Liang, Cunming]

> -----Original Message-----
> From: Jason Wang [mailto:address@hidden
> Sent: Friday, January 5, 2018 4:39 PM
> To: Liang, Cunming <address@hidden>; Bie, Tiwei
> <address@hidden>
> Cc: Tan, Jianfeng <address@hidden>; address@hidden;
> address@hidden; address@hidden; address@hidden;
> Wang, Xiao W <address@hidden>; address@hidden; Wang,
> Zhihong <address@hidden>; address@hidden; Daly, Dan
> <address@hidden>
> Subject: Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support
> VFIO based accelerators
> 
> 
> 
> On 2018年01月05日 14:58, Liang, Cunming wrote:
> >> Thanks for the pointer. Looks rather interesting.
> >>
> >>> We're also working on it (including defining a standard device for
> >>> vhost data path acceleration based on mdev to hide vendor specific
> >>> details).
> >> This is exactly what I mean. Form my point of view, there's no need
> >> for any extension for vhost protocol, we just need to reuse qemu
> >> iothread to implement a userspace vhost dataplane and do the mdev
> inside that thread.
> > On functional perspective, it makes sense to have qemu native support of
> those certain usage. However, qemu doesn't have to take responsibility for
> dataplane. There're already huge amounts of codes for different devices
> emulation, leveraging external dataplane library is an effective way to
> introduce more.
> 
> This does not mean to drop external dataplane library. Actually, you can link
> dpdk to qemu directly.
It's not a bad idea, then the interface comes to be new API/ABI definition of 
external dataplane library instead of existing vhost protocol. dpdk as a 
library is not a big deal to link with, customized application is.
In addition, it will ask for qemu to provide flexible process model then. Lots 
of application level features (e.g. hot upgrade/fix) becomes burden.
I'm open to that option, keep eyes on any proposal there.

> 
> > The beauty of vhost_user is to open a door for variable userland
> workloads(e.g. vswitch). The dataplane connected with VM usually need to
> be close integrated with those userland workloads, a control place
> interface(vhost-user) is better than a datapath interface(e.g. provided by
> dataplace in qemu iothread).
> 
> Do we really need vswitch for vDPA?
Accelerators come into the picture of vswitch, which usually provides in-chip 
EMC for early classification. It gives a fast path for those throughput 
sensitive(SLA) VNF to bypass the further table lookup. It co-exists other VNF 
whose SLA level is best effort but requires more functions(e.g. stateful 
conntrack, security check, even higher layer WAF support) support, DPDK based 
datapath still boost the throughput there. It's not used to be a single choice 
of dedicated or shared datapath, usually they're co-exist. 

> 
> >   On workloads point of view, it's not excited to be part of qemu process.
> 
> Don't see why, qemu have dataplane for virtio-blk/scsi.
Qemu has vhost-user for scsi too. I'm not saying which one is bad, just point 
out sometime it's very workloads driven. Network is different with 
blk/scsi/crypto.

> 
> > That comes up with the idea of vhost-user extension. Userland workloads
> decides to enable accelerators or not, qemu provides the common control
> plane infrastructure.
> 
> It brings extra complexity: endless new types of messages and a huge brunch
> of bugs. And what's more important, the split model tends to be less efficient
> in some cases, e.g guest IOMMU integration. I'm pretty sure we will meet
> more in the future.
vIOMMU relevant message has been supported by vhost protocol. It's independent 
effort there.
I don't see this patch introduce endless new types. My taking of your 
fundamental concern is about continues adding new features on vhost-user.
Feel free to correct me if I misunderstood your point.

> 
> >>> And IMO it's also not a bad idea to extend vhost-user protocol to
> >>> support the accelerators if possible. And it could be more flexible
> >>> because it could support (for example) below things easily without
> >>> introducing any complex command line options or monitor commands to
> >>> QEMU:
> >> Maybe I was wrong but I don't think we care about the complexity of
> >> command line or monitor command in this case.
> >>
> >>> - the switching among different accelerators and software version
> >>>     can be done at runtime in vhost process;
> >>> - use different accelerators to accelerate different queue pairs
> >>>     or just accelerate some (instead of all) queue pairs;
> >> Well, technically, if we want, these could be implemented in qemu too.
> > You're right if just considering I/O. The ways to consume those I/O is
> another perspective.
> > Simply 1:1 associating guest virtio-net and accelerator w/ SW datapath
> fallback is not the whole picture.
> 
> Pay attention:
> 
> 1) What I mean is not a fallback here. You can still do a lot of tricks e.g
> offloading datapath to hardware or doorbell map.
> 2) Qemu supports (very old and inefficient) a split model of device emulation
> and network backend. This means we can switch between backends (though
> not implemented).
Accelerator won't be defined in the same device layout, it means there're 
different kinds of drivers.
Qemu definitely won't like to have HW relevant driver there, that's end up with 
another vhost-vfio in my slides. A mediated device can help to unify the device 
layout definition, and leave the driver part in its own place.
This approach is quite good when application doesn't need to put userland SW 
datapath and accelerator datapath in the same picture as which I 
mentioned(vswitch cases).

> 
> >   It's variable usages on workload side to abstract the device (e.g. port 
> > re-
> presenter for vswitch) and etc. I don't think qemu is interested for all bunch
> of things there.
> >
> 
> Again, you can link any dataplane to qemu directly instead of using vhost-
> user if vhost-user tends to be less useful in some cases (vDPA is one of the
> case I think).
See my previous words.

> 
> Thanks