RE: RFC: use VFIO over a UNIX domain socket to implement device offloading

[Thanos Makatos]

> > > I've just shared with you the Google doc we've working on with John
> > where we've
> > > been drafting the protocol specification, we think it's time for some 
> > > first
> > > comments. Please feel free to comment/edit and suggest more people
> to
> > be on the
> > > reviewers list.
> > >
> > > You can also find the Google doc here:
> > >
> > >
> > https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__docs.google.com_document_d_1FspkL0hVEnZqHbdoqGLUpyC38rSk-
> 5F&d=DwIFAw&c=s883GpUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw6ogtti46a
> tk736SI4vgsJiUKIyDE&m=lJC7YeMMsAaVsr99tmTYncQdjEfOXiJQkRkJW7NMg
> Rg&s=RyyhgVrLX2bBTqpXZnBmllqkCg_wyalxwZKkfcYt50c&e=
> > 7HhY471TsVwyK8/edit?usp=sharing
> > >
> > > If a Google doc doesn't work for you we're open to suggestions.
> >
> > I can't add comments to the document so I've inlined them here:
> >
> > The spec assumes the reader is already familiar with VFIO and does not
> > explain concepts like the device lifecycle, regions, interrupts, etc.
> > We don't need to duplicate detailed VFIO information, but I think the
> > device model should be explained so that anyone can start from the
> > VFIO-user spec and begin working on an implementation.  Right now I
> > think they would have to do some serious investigation of VFIO first in
> > order to be able to write code.
> 
> I've added a high-level overview of how VFIO is used in this context.
> 
> > "only the source header files are used"
> > I notice the current <linux/vfio.h> header is licensed "GPL-2.0 WITH
> > Linux-syscall-note".  I'm not a lawyer but I guess this means there are
> > some restrictions on using this header file.  The <linux/virtio*.h>
> > header files were explicitly licensed under the BSD license to make it
> > easy to use the non __KERNEL__ parts.
> 
> My impression is that this note actually relaxes the licensing requirements, 
> so
> that proprietary software can use the system call headers and run on Linux
> without being considered derived work. In any case I'll double check with our
> legal team.
> 
> > VFIO-user Command Types: please indicate for each request type whether
> > it is client->server, server->client, or both.  Also is it a "command"
> > or "request"?
> 
> Will do. It's a command.
> 
> 
> > vfio_user_req_type <-- is this an extension on top of <linux/vfio.h>?
> > Please make it clear what is part of the base <linux/vfio.h> protocol
> > and what is specific to vfio-user.
> 
> Correct, it's an extension over <linux/vfio.h>. I've clarified the two symbol
> namespaces.
> 
> 
> > VFIO_USER_READ/WRITE serve completely different purposes depending
> on
> > whether they are sent client->server or server->client.  I suggest
> > defining separate request type constants instead of overloading them.
> 
> Fixed.
> 
> > What is the difference between VFIO_USER_MAP_DMA and
> > VFIO_USER_REG_MEM?
> > They both seem to be client->server messages for setting up memory but
> > I'm not sure why two request types are needed.
> 
> John will provide more information on this.
> 
> > struct vfio_user_req->data.  Is this really a union so that every
> > message has the same size, regardless of how many parameters are
> passed
> > in the data field?
> 
> Correct, it's a union so that every message has the same length.
> 
> > "a framebuffer where the guest does multiple stores to the virtual
> > device."  Do you mean in SMP guests?  Or even in a single CPU guest?
> 
> @John
> 
> > Also, is there any concurrency requirement on the client and server
> > side?  Can I implement a client/server that processes requests
> > sequentially and completes them before moving on to the next request or
> > would that deadlock for certain message types?
> 
> I believe that this might also depend on the device semantics, will need to
> think about it in greater detail.

I've looked at this but can't provide a definitive answer yet. I believe the
safest thing to do is for the server to process requests in order.

> More importantly, considering:
> a) Marc-André's comments about data alignment etc., and
> b) the possibility to run the server on another guest or host,
> we won't be able to use native VFIO types. If we do want to support that
> then
> we'll have to redefine all data formats, similar to
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_qemu_qemu_blob_master_docs_interop_vhost-
> 2Duser.rst&d=DwIFAw&c=s883GpUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw6
> ogtti46atk736SI4vgsJiUKIyDE&m=lJC7YeMMsAaVsr99tmTYncQdjEfOXiJQkRkJ
> W7NMgRg&s=1d_kB7VWQ-8d4t6Ikga5KSVwws4vwiVMvTyWVaS6PRU&e= .
> 
> So the protocol will be more like an enhanced version of the Vhost-user
> protocol
> than VFIO. I'm fine with either direction (VFIO vs. enhanced Vhost-user),
> so we need to decide before proceeding as the request format is
> substantially
> different.

Regarding the ability to use the protocol on non-AF_UNIX sockets, we can 
support this future use case without unnecessarily complicating the protocol by
defining the C structs and stating that data alignment and endianness for the 
non AF_UNIX case must be the one used by GCC on a x86_64 bit machine, or can 
be overridden as required.

We've polished the document a bit more with the help of Marc-André,
Raphael, and Swapnil.

The major outstanding issue is agreeing on the having a pair of commands for
registering/unregistering guest memory and another pair of commands to map/unmap
DMA regions, similar to QEMU.