Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Gonglei (Arei)]

> On Wed, 2014-08-20 at 06:39 +0200, Knut Omang wrote:
> > On Wed, 2014-08-20 at 02:16 +0000, Gonglei (Arei) wrote:
> > > > -----Original Message-----
> > > > From: Michael S. Tsirkin [mailto:address@hidden
> > > > Sent: Wednesday, August 20, 2014 5:19 AM
> > > > To: Gonglei (Arei)
> > > > Cc: Paolo Bonzini; Marcel Apfelbaum; address@hidden;
> > > > address@hidden; address@hidden; Huangweidong (C)
> > > > Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> > > >
> > > > On Tue, Aug 19, 2014 at 06:25:56AM +0000, Gonglei (Arei) wrote:
> > > > > > >> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35
> machine?
> > > > > > >>
> > > > > > >> On Sun, 2014-08-17 at 13:00 +0200, Michael S. Tsirkin wrote:
> > > > > > >>> On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
> > > > > > >>>> Hi,
> > > > > > >>>>
> > > > > > >>>> I noticed that the qemu-2.1 release change log says
> > > > > > >>>> " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
> > > > > > >>>> And then I made a testing for the hotplugging function of Q35.
> > > > > > >>>> But I'm failed, and I got the dmesg log in guest os as below:
> > > > > > >>>>
> > > > > > >>>> [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on
> Slot (0 -
> > > > 4)
> > > > > > >>>> [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot
> (0 - 4)
> > > > > > >>>> [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 -
> powering
> > > > on
> > > > > > due
> > > > > > >> to button press.
> > > > > > >>>> [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link
> status
> > > > > > >>>> [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on
> Slot (0
> > > > - 4)
> > > > > > >>>> [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on
> Slot (0
> > > > - 4)
> > > > > > >>>>
> > > > > > >>>> Steps of testing:
> > > > > > >>>>
> > > > > > >>>> #1. QEMU version:
> > > > > > >>>>
> > > > > > >>>>        The lateset master tree source.
> > > > > > >>>>
> > > > > > >>>> #2. Command line:
> > > > > > >>>>
> > > > > > >>>> ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35
> -device
> > > > > > >> ide-drive,bus=ide.2,drive=MacHDD \
> > > > > > >>>>  -drive
> > > > > > id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img
> > > > > > >> -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
> > > > > > >>>> QEMU 2.0.93 monitor - type 'help' for more information
> > > > > > >>>> (qemu) device_add
> > > > > > >>
> virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
> > > > > > >>>
> > > > > > >>> I don't think you can use any slot except slot 0 for pci 
> > > > > > >>> express.
> > > > > > >
> > > > > > > OK. Does the PCIe specification say that?
> > > > > > > I appreciate very much that you explain more.
> > > > > >
> > > > > > The closest I could find is in "7.3. Configuration Transaction
> > > > > > Rules"/"7.3.1. Device Number":
> > > > > >
> > > > > > With non-ARI Devices, PCI Express components are restricted to
> > > > > > implementing a single Device Number on their primary interface
> (Upstream
> > > > > > Port) [...] Downstream Ports that do not have ARI Forwarding enabled
> > > > > > must associate only Device 0 with the device attached to the Logical
> Bus
> > > > > > representing the Link from the Port. Configuration Requests
> > > > > > targeting the Bus Number associated with a Link specifying Device
> Number
> > > > > > 0 are delivered to the device attached to the Link; Configuration
> > > > > > Requests specifying all other Device Numbers (1-31)
> > > > > > must be terminated by the Switch Downstream Port or the Root Port
> with
> > > > > > an Unsupported Request Completion Status (equivalent to Master
> Abort in
> > > > > > PCI).
> > > > > >
> > > > > Thanks a lot, Paolo.
> > > > > And I found another issue when cold-plugging don't using slot 0, the
> PCIe
> > > > device also can't
> > > > > be searched in guest os.
> > > > >
> > > > > So, I have some questions and ideas:
> > > > >
> > > > > 1. Does qemu support ARI Forwarding for PCIe at present? If yes, how
> to
> > > > enable it ?
> > > >
> > > > What do you mean by forwarding?
> > >
> > > Just ARI supporting, came from the PCIe spec.
> > >
> > > > What would you like to do?
> > >
> > > I just want to add some check. Because it made me confused,
> > > and other people IMHO.
> > >
> > > > We do have code to emulate ARI, I don't think many people
> > > > tested it so it's likely incomplete.
> > > >
> > > Yes, you have said (pcie_ari_init).
> >
> > I have actually been playing around with this a bit in the context of
> > SR/IOV, and have some patches waiting to be posted. I haven't had any
> > success with the downstream switch - as you indicate, pieces may be
> > missing, but in my case, using the ioh3420 root port worked fine after
> > setting the ARI forwarding capability and a few other minor fixes that
> > may even be considered trivial patches. I will put my acts together and
> > rebase these and post in a separate mail.
> >
> > Basically I am able to run and hotplug an ARI capable device on any root
> > port. So instead of using the downstream switch, you could just add
> > another ioh3420 for your next device, if that suits your needs.
> >
> > Note that there are two places to add "ARI support", one is the
> > forwarding capability of bridges/switches, the other is the device's
> > capability to be an ARI device, represented by the ARI PCIe capability.
> >
Hi, you are welcome to post your patches.
At present, both root roots and downstream ports 
are not support a device with slot > 0.

> > > > > 2. If not, we should add some check for PCIe root ports and downstream
> > > > ports,
> > > > >  meanwhile add explaining document.
> > > >
> > > > You want an attempt to add a device at slot !=0 to report an error?
> > > > We can do this if device at slot 0 does not have ARI support.
> > > > Seems like a low priority issue I think.
> >
> > If I understand the spec right it is devices in slots > 0 that is
> 
> Sorry - just noticed that a confusing "not" was lost here - I meant:
> 
> "it is not devices in slots > 0 that are"
> 
> > limited, it is that each slot may only expose one device, even if that
> > device has more than 8 functions and supports ARI, in which case we
> > should create a master abort.

PCIe spec says that Downstream Ports that do not have ARI Forwarding 
enabled must associate only Device 0 with the device attached to the Logical
Bus representing the Link from the Port.        

> > But the better solution in my view would
> > be just to implement ARI forwarding..
> 
Yes.

Best regards,
-Gonglei