Re: [Qemu-devel] [RfC PATCH] Add udmabuf misc device

[Oleksandr Andrushchenko]

On 04/06/2018 09:57 PM, Dongwon Kim wrote:
> On Fri, Apr 06, 2018 at 03:36:03PM +0300, Oleksandr Andrushchenko wrote:
> > On 04/06/2018 02:57 PM, Gerd Hoffmann wrote:
> > >    Hi,
> > >
> > > > > I fail to see any common ground for xen-zcopy and udmabuf ...
> > > > Does the above mean you can assume that xen-zcopy and udmabuf
> > > > can co-exist as two different solutions?
> > > Well, udmabuf route isn't fully clear yet, but yes.
> > >
> > > See also gvt (intel vgpu), where the hypervisor interface is abstracted
> > > away into a separate kernel modules even though most of the actual vgpu
> > > emulation code is common.
> > Thank you for your input, I'm just trying to figure out
> > which of the three z-copy solutions intersect and how much
> > > > And what about hyper-dmabuf?
> xen z-copy solution is pretty similar fundamentally to hyper_dmabuf
> in terms of these core sharing feature:
>
> 1. the sharing process - import prime/dmabuf from the producer -> extract
> underlying pages and get those shared -> return references for shared pages
>
> 2. the page sharing mechanism - it uses Xen-grant-table.
>
> And to give you a quick summary of differences as far as I understand
> between two implementations (please correct me if I am wrong, Oleksandr.)
>
> 1. xen-zcopy is DRM specific - can import only DRM prime buffer
> while hyper_dmabuf can export any dmabuf regardless of originator
Well, this is true. And at the same time this is just a matter
of extending the API: xen-zcopy is a helper driver designed for
xen-front/back use-case, so this is why it only has DRM PRIME API
> 2. xen-zcopy doesn't seem to have dma-buf synchronization between two VMs
> while (as danvet called it as remote dmabuf api sharing) hyper_dmabuf sends
> out synchronization message to the exporting VM for synchronization.
This is true. Again, this is because of the use-cases it covers.
But having synchronization for a generic solution seems to be a good idea.
> 3. 1-level references - when using grant-table for sharing pages, there will
> be same # of refs (each 8 byte)
To be precise, grant ref is 4 bytes
> as # of shared pages, which is passed to
> the userspace to be shared with importing VM in case of xen-zcopy.
The reason for that is that xen-zcopy is a helper driver, e.g.
the grant references come from the display backend [1], which implements
Xen display protocol [2]. So, effectively the backend extracts references
from frontend's requests and passes those to xen-zcopy as an array
of refs.
>   Compared
> to this, hyper_dmabuf does multiple level addressing to generate only one
> reference id that represents all shared pages.
In the protocol [2] only one reference to the gref directory is passed 
between VMs
(and the gref directory is a single-linked list of shared pages 
containing all
of the grefs of the buffer).

> 4. inter VM messaging (hype_dmabuf only) - hyper_dmabuf has inter-vm msg
> communication defined for dmabuf synchronization and private data (meta
> info that Matt Roper mentioned) exchange.
This is true, xen-zcopy has no means for inter VM sync and meta-data,
simply because it doesn't have any code for inter VM exchange in it,
e.g. the inter VM protocol is handled by the backend [1].
> 5. driver-to-driver notification (hyper_dmabuf only) - importing VM gets
> notified when newdmabuf is exported from other VM - uevent can be optionally
> generated when this happens.
>
> 6. structure - hyper_dmabuf is targetting to provide a generic solution for
> inter-domain dmabuf sharing for most hypervisors, which is why it has two
> layers as mattrope mentioned, front-end that contains standard API and backend
> that is specific to hypervisor.
Again, xen-zcopy is decoupled from inter VM communication
> > > No idea, didn't look at it in detail.
> > >
> > > Looks pretty complex from a distant view.  Maybe because it tries to
> > > build a communication framework using dma-bufs instead of a simple
> > > dma-buf passing mechanism.
> we started with simple dma-buf sharing but realized there are many
> things we need to consider in real use-case, so we added communication
> , notification and dma-buf synchronization then re-structured it to
> front-end and back-end (this made things more compicated..) since Xen
> was not our only target. Also, we thought passing the reference for the
> buffer (hyper_dmabuf_id) is not secure so added uvent mechanism later.
>
> > Yes, I am looking at it now, trying to figure out the full story
> > and its implementation. BTW, Intel guys were about to share some
> > test application for hyper-dmabuf, maybe I have missed one.
> > It could probably better explain the use-cases and the complexity
> > they have in hyper-dmabuf.
> One example is actually in github. If you want take a look at it, please
> visit:
>
> https://github.com/downor/linux_hyper_dmabuf_test/tree/xen/simple_export
Thank you, I'll have a look
> > > Like xen-zcopy it seems to depend on the idea that the hypervisor
> > > manages all memory it is easy for guests to share pages with the help of
> > > the hypervisor.
> > So, for xen-zcopy we were not trying to make it generic,
> > it just solves display (dumb) zero-copying use-cases for Xen.
> > We implemented it as a DRM helper driver because we can't see any
> > other use-cases as of now.
> > For example, we also have Xen para-virtualized sound driver, but
> > its buffer memory usage is not comparable to what display wants
> > and it works somewhat differently (e.g. there is no "frame done"
> > event, so one can't tell when the sound buffer can be "flipped").
> > At the same time, we do not use virtio-gpu, so this could probably
> > be one more candidate for shared dma-bufs some day.
> > >    Which simply isn't the case on kvm.
> > >
> > > hyper-dmabuf and xen-zcopy could maybe share code, or hyper-dmabuf build
> > > on top of xen-zcopy.
> > Hm, I can imagine that: xen-zcopy could be a library code for hyper-dmabuf
> > in terms of implementing all that page sharing fun in multiple directions,
> > e.g. Host->Guest, Guest->Host, Guest<->Guest.
> > But I'll let Matt and Dongwon to comment on that.
> I think we can definitely collaborate. Especially, maybe we are using some
> outdated sharing mechanism/grant-table mechanism in our Xen backend (thanks
> for bringing that up Oleksandr). However, the question is once we collaborate
> somehow, can xen-zcopy's usecase use the standard API that hyper_dmabuf
> provides? I don't think we need different IOCTLs that do the same in the final
> solution.
If you think of xen-zcopy as a library (which implements Xen
grant references mangling) and DRM PRIME wrapper on top of that
library, we can probably define proper API for that library,
so both xen-zcopy and hyper-dmabuf can use it. What is more, I am
about to start upstreaming Xen para-virtualized sound device driver soon,
which also uses similar code and gref passing mechanism [3].
(Actually, I was about to upstream drm/xen-front, drm/xen-zcopy and
snd/xen-front and then propose a Xen helper library for sharing big buffers,
so common code of the above drivers can use the same code w/o code 
duplication)

Thank you,
Oleksandr

P.S. All, is it a good idea to move this out of udmabuf thread into a 
dedicated one?
> > > cheers,
> > >    Gerd
> > Thank you,
> > Oleksandr
> >
> > P.S. Sorry for making your original mail thread to discuss things much
> > broader than your RFC...
[1] https://github.com/xen-troops/displ_be
[2] 
https://elixir.bootlin.com/linux/v4.16-rc7/source/include/xen/interface/io/displif.h#L484
[3] 
https://elixir.bootlin.com/linux/v4.16-rc7/source/include/xen/interface/io/sndif.h