Re: [Qemu-devel] [RFC] SPAPR-PCI Hotplug Support in Qemu

[Benjamin Herrenschmidt]

On Thu, 2013-10-10 at 18:30 -0400, Mike Day wrote:
> Adding Anthony's corrected address.
> 
> 
> On Thu, Oct 10, 2013 at 6:25 PM, Mike Day <address@hidden> wrote:
> [RFC] SPAPR-PCI Hotplug Support in Qemu
> >
> > Background:
> > ppc64 has a unique bus structure for PCI slots: each slot is connected
> > to its PHB by a pci switch. This is true in some IBM hardware as well as
> > paravirtual hardware (PAPR).

Not *exactly* ... in PAPR, it's frequent that PCI and PCIe devices
appear to be direct children of the PHB. In fact that has been
problematic in the past because PCIe devices are supposed to always be
behind a virtual bridge, but that's not how pHyp exposes them so you end
up with a device with a PCIe capability without a parent with such a
capability. That has confused Linux more than once.

What is very common however is for us to just pop a new virtual PHB for
a new device (or group of devices). This will be especially true with
SR-IOV where we'll pretty much expect VFs to show up behind dedicated
virtual PHBs.

Additionally, with pass-through, there is the constraint that devices in
the same iommu group appear behind a common parent bridge (and nothing
else below that bridge is on a different group) since the way we expose
the iommu to the guest is via properties in the bridge (or PHB) node of
the device-tree.

> > SLOF firmware normally scans the hardware bus and creates the correct
> > slot/PCI switch -complex in the open firmware device tree. It also
> > configures the slot and PCI switch (BARs, etc.)
> >
> > For devices set up by platform firmware, each PCI device is attached to
> > its PHB and correctly configured.
> >
> > For Linux hot-plugged devices running under PowerVM today, each device
> > is created with a PCI switch hanging off the dev->subordinate
> > pointer. (PowerVM gets this info from the open firmware device tree in
> > rtas.)

That is not entirely correct, I have many cases where PowerVM does not
expose a PCI bridge/switch but shows the devices directly below the PHB.

Guests should cope either way.

The problem really lies in the fact that devices below a given bridge
(or PHB) are expected to share the iommu table.

> > Problem:
> >
> > The Qemu hot-plug path doesn't anticipate a PCI switch being attached
> > to every PHB slot.
> >
> > When hot-plugging a device, Qemu qdev creates the device, which allows
> > the device to initialize itself. Qemu then passes this initialized
> > device to the ppc PHB via the hot-plug path.[1]
> >
> > The current ppc hot-plug code then creates a device tree node for the
> > device [2], and allocates resources (BARs etc) for the new device. [3]
> >
> > The ppc64 kernel expects each hot-plugged PCI device structure to
> > point to a subordinate bus dev->subordinate. This assumption is held
> > throughout the ppc PCI code, and there are numerous opportunities for
> > panics when the device gets passed to a kernel routine with a
> > subordinate pointer. [4]
> >
> >
> > Proposed Solutions:
> >
> > (1) Create and hook an inert PCI switch to every hot-plugged PCI
> > device in Qemu.
> >
> > (a) After the device has initialized itself, at hot-plug time, create a
> > new PCI switch, configure the switch, allocate BARs, and attach the
> > switch to the hot-plugged devices (dev->subordinate).
> >
> > (b) create a new device tree node that begins with the PCI switch and
> > the parent of the hot-plugged device. Add the PCI switch/device
> > complex to the device tree under the PHB.
> >
> > (2) Add each hot-plugged PCI device to its own complex of PHB
> > (Processor Host Bus) and PCI switch.
> >
> > Simplify (1) by creating a new PHB for each hot-plugged device.
> >
> > (a) At PHB creation time, create a PCI switch device node for each PHB
> > slot.
> >
> > (b) At hot-plug time, create and configure a new PHB and add the
> > hot-plugged device to one of the slots. Configure and allocate
> > resources as normally.
> >
> > Comments:
> >
> > The current code has only one PHB. We know we need to support more
> > than one PHB ultimately. Solution #2 is consistent with this approach.
> >
> >
> > [1] https://github.com/mdroth/qemu/blob/spapr-pci-hotplug/hw/ppc/spapr_pci.c
> >
> > [2] ibm,rtas_configure_connector:
> > https://github.com/mdroth/qemu/blob/spapr-pci-hotplug/hw/ppc/spapr_pci.c#L575
> >
> > [3] spapr_phb_add_pci_dt
> > https://github.com/mdroth/qemu/blob/spapr-pci-hotplug/hw/ppc/spapr_pci.c#L900
> >
> > [4] dlpar_pci_add_bus
> > http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/drivers/pci/hotplug/rpadlpar_core.c?id=8bf3379a74bc9132751bfa685bad2da318fd59d7#n165
> >

Ben.