Re: [Qemu-devel] [0/8] RFC: VFIO and guest side IOMMUs, revisited

[David Gibson]

On Mon, May 13, 2013 at 03:30:26PM +0200, Paolo Bonzini wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Il 13/05/2013 15:13, David Gibson ha scritto:
> > On Mon, May 13, 2013 at 02:23:30PM +0200, Paolo Bonzini wrote:
> >> Il 13/05/2013 12:54, David Gibson ha scritto:
> >>> Specifically the way the iommu is determined from a callback
> >>> in the PCIBus means that it won't be assigned for devices under
> >>> a PCI-PCI bridge.
> >> 
> >> Right.  I saw the report from Alexey, but I am a bit wary of 
> >> touching it because it's not a regression.  In fact there is
> >> even a FIXME for it:
> >> 
> >> /* FIXME: inherit memory region from bus creator */
> > 
> > Uh.. sort of.
> > 
> >> Perhaps we can make pci_iommu_as a Bus method, where the default
> >>  implementation looks up along the chain, and the end of the 
> >> recursion is in SysBus or in PCI buses that have set the 
> >> callback.
> > 
> > So, this is complicated by the fact that there are two cases, and 
> > they can both be found in existing hardware.
> > 
> > 1) One is where devices behind the bridge are not visible / 
> > differentiable to the IOMMU, and so effectively all their DMAs 
> > originate from the bridge device itself.  In this case the correct
> >  thing is to give all devices under the bridge the same DMA 
> > AddressSpace as the bridge device, as suggested by the FIXME. This
> > will be typical behaviour for PCI-E to PCI bridges.
> > 
> > 2) The other case is where the bridge passes through RIDs, so that 
> > the IOMMU can still differentiate devices behind it.  For this 
> > case, we really want the hook to be in the host bridge / root bus, 
> > and it can make a decision based on the full bus/dev/fn 
> > information.  This will be typical for PCI-E to PCI-E bridges (or 
> > switches or nexuses or whatever they're usually called for PCI-E). 
> > This case will be very important as we start to model newer PCI-E 
> > based machines by default, where typically *all* devices are
> > behind a logical p2p bridge inside the root complex (but are still 
> > differentiable by the Intel IOMMU amongst others).
> > 
> > I'm not sure at this stage how to properly handle both cases.
> 
> Suppose you have a host bridge pci_bus0 and a PCIE->PCIE bridge
> pci_bus1.  pci_bus1 does not define a IOMMU callback, pci_bus0 does.
> 
> Would it work to use the PCIBus callback provided by pci_bus0, but
> invoke it as
> 
>     pci_bus0->iommu_fn(pci_bus1, pci_bus0->iommu_opaque, devfn)

Hrm.  I'm a bit nervous about that, because I think when writing an
iommu_fn it would be very easy to forget that it could be called with
a bus other than the one the hook is attached to - and e.g. assuming
they can use bus->qbus.parent_dev to get to the host bridge.

Tbh, my first inclination would just be to move the hook to the host
bridge class (that will make more sense on top of my other pending PCI
patches), passing the bus through.  The host bridge hook would need to
handle any intervening p2p bridges.  That's a bit nasty in theory, but
for the cases I know about it shouldn't be hard because either 1) the
IOMMU exists on PCI-E based systems, so the RIDs are passed through,
or 2) there's only one IOMMU address space per host bridge anyway.

The other approach would be to have both a bus and a host bridge
hook.  The bus hook would be the one used directly.  The default bus
hook would find the host bridge and invoke its hook.  Bridges that
lose RID information, though, can invoke the parent bus hook with the
bridge's details.  The tricky bit is to work out which is the correct
bridge type for each platform/configuration.

-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: Digital signature