[Qemu-ppc] [RFC] SPAPR-PCI Hotplug Support in Qemu

[Mike Day]

[RFC] SPAPR-PCI Hotplug Support in Qemu

Background:
ppc64 has a unique bus structure for PCI slots: each slot is connected
to its PHB by a pci switch. This is true in some IBM hardware as well as
paravirtual hardware (PAPR).

SLOF firmware normally scans the hardware bus and creates the correct
slot/PCI switch -complex in the open firmware device tree. It also
configures the slot and PCI switch (BARs, etc.)

For devices set up by platform firmware, each PCI device is attached to
its PHB and correctly configured.

For Linux hot-plugged devices running under PowerVM today, each device
is created with a PCI switch hanging off the dev->subordinate
pointer. (PowerVM gets this info from the open firmware device tree in
rtas.)

Problem:

The Qemu hot-plug path doesn't anticipate a PCI switch being attached
to every PHB slot. 

When hot-plugging a device, Qemu qdev creates the device, which allows
the device to initialize itself. Qemu then passes this initialized
device to the ppc PHB via the hot-plug path.[1]

The current ppc hot-plug code then creates a device tree node for the
device [2], and allocates resources (BARs etc) for the new device. [3]

The ppc64 kernel expects each hot-plugged PCI device structure to
point to a subordinate bus dev->subordinate. This assumption is held
throughout the ppc PCI code, and there are numerous opportunities for
panics when the device gets passed to a kernel routine with a
subordinate pointer. [4]


Proposed Solutions:

(1) Create and hook an inert PCI switch to every hot-plugged PCI
device in Qemu. 

(a) After the device has initialized itself, at hot-plug time, create a
new PCI switch, configure the switch, allocate BARs, and attach the
switch to the hot-plugged devices (dev->subordinate).

(b) create a new device tree node that begins with the PCI switch and
the parent of the hot-plugged device. Add the PCI switch/device
complex to the device tree under the PHB.

(2) Add each hot-plugged PCI device to its own complex of PHB
(Processor Host Bus) and PCI switch.

Simplify (1) by creating a new PHB for each hot-plugged device. 

(a) At PHB creation time, create a PCI switch device node for each PHB
slot. 

(b) At hot-plug time, create and configure a new PHB and add the
hot-plugged device to one of the slots. Configure and allocate
resources as normally. 

Comments:

The current code has only one PHB. We know we need to support more
than one PHB ultimately. Solution #2 is consistent with this approach.


[1] https://github.com/mdroth/qemu/blob/spapr-pci-hotplug/hw/ppc/spapr_pci.c

[2] ibm,rtas_configure_connector:
https://github.com/mdroth/qemu/blob/spapr-pci-hotplug/hw/ppc/spapr_pci.c#L575

[3] spapr_phb_add_pci_dt
https://github.com/mdroth/qemu/blob/spapr-pci-hotplug/hw/ppc/spapr_pci.c#L900

[4] dlpar_pci_add_bus
http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/drivers/pci/hotplug/rpadlpar_core.c?id=8bf3379a74bc9132751bfa685bad2da318fd59d7#n165


-- 

Mike Day | + 1 919 371-8786 | address@hidden
"Endurance is a Virtue"