Re: [Qemu-devel] Guest IOMMU and Cisco usnic

[Benoît Canet]

The Wednesday 12 Feb 2014 à 12:34:25 (-0700), Alex Williamson wrote :
> On Wed, 2014-02-12 at 19:10 +0100, Benoît Canet wrote:
> > Hi Alex,
> > 
> > After the IRC conversation we had a few days ago I understood that guest 
> > IOMMU
> > was not implemented.
> > 
> > I have a real use case for it:
> > 
> > Cisco usnic allow to write MPI applications while driving the network card 
> > in
> > userspace in order to optimize the latency. It's made for compute clusters.
> > 
> > The typical cloud provider don't provide bare metal access but only vms on 
> > top
> > of Cisco's hardware hence VFIO is using the IOMMU to passthrough the NIC to 
> > the
> > guest and no IOMMU is present in the guest.
> > 
> > questions: Would writing a performing guest IOMMU implementation be 
> > possible ?
> >            How complex this project looks for someone knowing IOMMUs issues 
> > ?
> > 
> > The ideal implementation would forward the IOMMU work to the host hardware 
> > for
> > speed.
> > 
> > I can devote time writing the feature if it's doable.
> 
> Hi Benoît,
> 
> I imagine it's doable, but it's certainly not trivial, beyond that I
> haven't put much thought into it.

Thanks for the anwser.
I am afraid when an expert of the field says "not trivial" :)

Best regards

Benoît

> 
> VFIO running in a guest would need an IOMMU that implements both the
> IOMMU API and IOMMU groups.  Whether that comes from an emulated
> physical IOMMU (like VT-d) or from a new paravirt IOMMU would be for you
> to decide.  VT-d would imply using a PCIe chipset like Q35 and trying to
> bandage on VT-d or updating Q35 to something that natively supports
> VT-d.  Getting a sufficiently similar PCIe hierarchy between host an
> guest would also be required.
> 
> The current model of putting all guest devices in a single IOMMU domain
> on the host is likely not what you would want and might imply a new VFIO
> IOMMU backend that is better tuned for separate domains, sparse
> mappings, and low-latency.  VFIO has a modular IOMMU design, so this
> isn't architecturally a problem.  The VFIO user (QEMU) is able to select
> which backend to use and the code is written with supporting multiple
> backends in mind.
> 
> A complication you'll have is that the granularity of IOMMU operations
> through VFIO is at the IOMMU group level, so the guest would not be able
> to easily split devices grouped together on the host between separate
> users in the guest.  That could be modeled as a conventional PCI bridge
> masking the requester ID of devices in the guest such that host groups
> are mirrored as guest groups.
> 
> There might also be more simple "punch-through" ways to do it, for
> instance what if instead of trying to make it work like it does on the
> host we invented a paravirt VFIO interface and the vfio-pv driver in the
> guest populated /dev/vfio as slightly modified passthroughs to the host
> fds.  The guest OS may not even really need to be aware of the device.
> 
> It's an interesting project and certainly a valid use case.  I'd also
> like to see things like Intel's DPDK move to using VFIO, but the current
> UIO DPDK is often used in guests.  Thanks,
> 
> Alex
> 
>