Re: RFC: use VFIO over a UNIX domain socket to implement device offloading

[Stefan Hajnoczi]

On Thu, Mar 26, 2020 at 09:47:38AM +0000, Thanos Makatos wrote:
> Build MUSER with vfio-over-socket:
> 
>         git clone --single-branch --branch vfio-over-socket 
> address@hidden:tmakatos/muser.git
>         cd muser/
>         git submodule update --init
>         make
> 
> Run device emulation, e.g.
> 
>         ./build/dbg/samples/gpio-pci-idio-16 -s <N>
> 
> Where <N> is an available IOMMU group, essentially the device ID, which must 
> not
> previously exist in /dev/vfio/.
> 
> Run QEMU using the vfio wrapper library and specifying the MUSER device:
> 
>         LD_PRELOAD=muser/build/dbg/libvfio/libvfio.so qemu-system-x86_64 \
>                 ... \
>                 -device vfio-pci,sysfsdev=/dev/vfio/<N> \
>                 -object 
> memory-backend-file,id=ram-node0,prealloc=yes,mem-path=mem,share=yes,size=1073741824
>  \
>                 -numa node,nodeid=0,cpus=0,memdev=ram-node0
> 
> Bear in mind that since this is just a PoC lots of things can break, e.g. some
> system call not intercepted etc.

Cool, I had a quick look at libvfio and how the transport integrates
into libmuser.  The integration on the libmuser side is nice and small.

It seems likely that there will be several different implementations of
the vfio-over-socket device side (server):
1. libmuser
2. A Rust equivalent to libmuser
3. Maybe a native QEMU implementation for multi-process QEMU (I think JJ
   has been investigating this?)

In order to interoperate we'll need to maintain a protocol
specification.  Mayb You and JJ could put that together and CC the vfio,
rust-vmm, and QEMU communities for discussion?

It should cover the UNIX domain socket connection semantics (does a
listen socket only accept 1 connection at a time?  What happens when the
client disconnects?  What happens when the server disconnects?), how
VFIO structs are exchanged, any vfio-over-socket specific protocol
messages, etc.  Basically everything needed to write an implementation
(although it's not necessary to copy the VFIO struct definitions from
the kernel headers into the spec or even document their semantics if
they are identical to kernel VFIO).

The next step beyond the LD_PRELOAD library is a native vfio-over-socket
client implementation in QEMU.  There is a prototype here:
https://github.com/elmarco/qemu/blob/wip/vfio-user/hw/vfio/libvfio-user.c

If there are any volunteers for working on that then this would be a
good time to discuss it.

Finally, has anyone looked at CrosVM's out-of-process device model?  I
wonder if it has any features we should consider...

Looks like a great start to vfio-over-socket!

Stefan

signature.asc
Description: PGP signature