Re: [Qemu-devel] About virtio device hotplug in Q35! 【外域邮件.谨慎查阅】

[Alex Williamson]

On Wed, 26 Jul 2017 09:21:38 +0300
Marcel Apfelbaum <address@hidden> wrote:

> On 25/07/2017 11:53, 陈博 wrote:
> > To accelerate data traversing between devices under the same PCIE Root 
> > Port or Switch.
> > 
> > See https://lists.nongnu.org/archive/html/qemu-devel/2017-07/msg07209.html
> >   
> 
> Hi,
> 
> It may be possible, but maybe PCIe Switch assignment is not
> the only way to go.
> 
> Adding Alex and Michael for their input on this matter.
> More info at:
> https://lists.nongnu.org/archive/html/qemu-devel/2017-07/msg07209.html

I think you need to look at where the IOMMU is in the topology and what
address space the devices are working in when assigned to a VM to
realize that it doesn't make any sense to assign switch ports to a VM.
GPUs cannot do switch level peer to peer when assigned because they are
operating in an I/O virtual address space.  This is why we configure
ACS on downstream ports to prevent peer to peer.  Peer to peer
transactions must be forwarded upstream by the switch ports in order to
reach the IOMMU for translation.  Note however that we do populate peer
to peer mappings within the IOMMU, so if the hardware supports it, the
IOMMU can reflect the transaction back out to the I/O bus to reach the
other device without CPU involvement.

Therefore I think the better solution, if it encourages the NVIDIA
driver to do the right thing, is to use emulated switches.  Assigning
the physical switch would really do nothing more than make the PCIe link
information more correct in the VM, everything else about the switch
would be emulated.  Even still, unless you have an I/O topology which
integrates the IOMMU into the switch itself, the data flow still needs
to go all the way to the root complex to hit the IOMMU before being
reflected to the other device.  Direct peer to peer between downstream
switch ports operates in the wrong address space.  Thanks,

Alex