Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines

[Alex Williamson]

On Thu,  1 Sep 2016 16:22:07 +0300
Marcel Apfelbaum <address@hidden> wrote:

> Proposes best practices on how to use PCIe/PCI device
> in PCIe based machines and explain the reasoning behind them.
> 
> Signed-off-by: Marcel Apfelbaum <address@hidden>
> ---
> 
> Hi,
> 
> Please add your comments on what to add/remove/edit to make this doc usable.
> 
> Thanks,
> Marcel
> 
>  docs/pcie.txt | 145 
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 145 insertions(+)
>  create mode 100644 docs/pcie.txt
> 
> diff --git a/docs/pcie.txt b/docs/pcie.txt
> new file mode 100644
> index 0000000..52a8830
> --- /dev/null
> +++ b/docs/pcie.txt
> @@ -0,0 +1,145 @@
> +PCI EXPRESS GUIDELINES
> +======================
> +
> +1. Introduction
> +================
> +The doc proposes best practices on how to use PCIe/PCI device
> +in PCIe based machines and explains the reasoning behind them.
> +
> +
> +2. Device placement strategy
> +============================
> +QEMU does not have a clear socket-device matching mechanism
> +and allows any PCI/PCIe device to be plugged into any PCI/PCIe slot.
> +Plugging a PCI device into a PCIe device might not always work and
> +is weird anyway since it cannot be done for "bare metal".
> +Plugging a PCIe device into a PCI slot will hide the Extended
> +Configuration Space thus is also not recommended.
> +
> +The recommendation is to separate the PCIe and PCI hierarchies.
> +PCIe devices should be plugged only into PCIe Root Ports and
> +PCIe Downstream ports (let's call them PCIe ports).
> +
> +2.1 Root Bus (pcie.0)
> +=====================
> +Plug only legacy PCI devices as Root Complex Integrated Devices
> +even if the PCIe spec does not forbid PCIe devices. The existing

Surely we can have PCIe device on the root complex??

> +hardware uses mostly PCI devices as Integrated Endpoints. In this
> +way we may avoid some strange Guest OS-es behaviour.
> +Other than that plug only PCIe Root Ports, PCIe Switches (upstream ports)
> +or DMI-PCI bridges to start legacy PCI hierarchies.
> +
> +
> +   pcie.0 bus
> +   --------------------------------------------------------------------------
> +        |                |                    |                   |
> +   -----------   ------------------   ------------------  ------------------
> +   | PCI Dev |   | PCIe Root Port |   |  Upstream Port |  | DMI-PCI bridge |
> +   -----------   ------------------   ------------------  ------------------

Do you have a spec reference for plugging an upstream port directly
into the root complex?  IMHO this is invalid, an upstream port can only
be attached behind a downstream port, ie. a root port or downstream
switch port.

> +
> +2.2 PCIe only hierarchy
> +=======================
> +Always use PCIe Root ports to start a PCIe hierarchy. Use PCIe switches 
> (Upstream
> +Ports + several Downstream Ports) if out of PCIe Root Ports slots. PCIe 
> switches
> +can be nested until a depth of 6-7. Plug only PCIe devices into PCIe Ports.

This seems to contradict 2.1, but I agree more with this statement to
only start a PCIe sub-hierarchy with a root port, not an upstream port
connected to the root complex.  The 2nd sentence is confusing, I don't
know if you're referring to fan-out via PCIe switch downstream of a
root port or again suggesting to use upstream switch ports directly on
the root complex.

> +
> +
> +   pcie.0 bus
> +   ----------------------------------------------------
> +        |                |               |
> +   -------------   -------------   -------------
> +   | Root Port |   | Root Port |   | Root Port |
> +   ------------   --------------   -------------
> +         |                               |
> +    ------------                 -----------------
> +    | PCIe Dev |                 | Upstream Port |
> +    ------------                 -----------------
> +                                  |            |
> +                     -------------------    -------------------
> +                     | Downstream Port |    | Downstream Port |
> +                     -------------------    -------------------
> +                             |
> +                         ------------
> +                         | PCIe Dev |
> +                         ------------
> +
> +2.3 PCI only hierarchy
> +======================
> +Legacy PCI devices can be plugged into pcie.0 as Integrated Devices or
> +into DMI-PCI bridge. PCI-PCI bridges can be plugged into DMI-PCI bridges
> +and can be nested until a depth of 6-7. DMI-BRIDGES should be plugged
> +only into pcie.0 bus.
> +
> +   pcie.0 bus
> +   ----------------------------------------------
> +        |                            |
> +   -----------               ------------------
> +   | PCI Dev |               | DMI-PCI BRIDGE |
> +   ----------                ------------------
> +                               |            |
> +                        -----------    ------------------
> +                        | PCI Dev |    | PCI-PCI Bridge |
> +                        -----------    ------------------
> +                                         |           |
> +                                  -----------     -----------
> +                                  | PCI Dev |     | PCI Dev |
> +                                  -----------     -----------
> +

I really wish we had generic PCIe-to-PCI bridges rather than this DMI
bridge thing...

> +
> +
> +3. IO space issues
> +===================
> +PCIe Ports are seen by Firmware/Guest OS as PCI bridges and

Yeah, I've lost the meaning of Ports here, this statement is true for
upstream ports as well.

> +as required by PCI spec will reserve a 4K IO range for each.
> +The firmware used by QEMU (SeaBIOS/OVMF) will further optimize
> +it by allocation the IO space only if there is at least a device
> +with IO BARs plugged into the bridge.
> +Behind a PCIe PORT only one device may be plugged, resulting in

Here I think you're trying to specify root/downstream ports, but
upstream ports have the same i/o port allocation problems and do not
have this one device limitation.

> +the allocation of a whole 4K range for each device.
> +The IO space is limited resulting in ~10 PCIe ports per system
> +if devices with IO BARs are plugged into IO ports.
> +
> +Using the proposed device placing strategy solves this issue
> +by using only PCIe devices with PCIe PORTS. The PCIe spec requires
> +PCIe devices to work without IO BARs.
> +The PCI hierarchy has no such limitations.

Actually it does, but it's mostly not an issue since we have 32 slots
available (minus QEMU/libvirt excluding 1 for no good reason)
downstream of each bridge.

> +
> +
> +4. Hot Plug
> +============
> +The root bus pcie.0 does not support hot-plug, so Integrated Devices,
> +DMI-PCI bridges and Root Ports can't be hot-plugged/hot-unplugged.
> +
> +PCI devices can be hot-plugged into PCI-PCI bridges. (There is a bug
> +in QEMU preventing it to work, but it would be solved soon).

Probably want to give some sort of date/commit references to these
current state of affairs facts, a reader is not likely to lookup the
git commit for this verbiage and extrapolate it to a QEMU version.

> +The PCI hotplug is ACPI based and can work side by side with the PCIe
> +native hotplug.
> +
> +PCIe devices can be natively hot-plugged/hot-unplugged into/from
> +PCIe Ports (Root Ports/Downstream Ports). Switches are hot-pluggable.

Why?  This seems like a QEMU bug.  Clearly we need the downstream ports
in place when the upstream switch is hot-added, but this should be
feasible.

> +Keep in mind you always need to have at least one PCIe Port available
> +for hotplug, the PCIe Ports themselves are not hot-pluggable.

If a user cares about hotplug...

> +
> +
> +5. Device assignment
> +====================
> +Host devices are mostly PCIe and should be plugged only into PCIe ports.
> +PCI-PCI bridge slots can be used for legacy PCI host devices.

I don't think we have any evidence to suggest this as a best practice.
We have a lot of experience placing PCIe host devices into a
conventional PCI topology on 440FX.  We don't have nearly as much
experience placing them into downstream PCIe ports.  This seems like
how we would like for things to behave to look like real hardware
platforms, but it's just navel gazing whether it's actually the right
thing to do.  Thanks,

Alex

> +
> +
> +6. Virtio devices
> +=================
> +Virtio devices plugged into the PCI hierarchy or as an Integrated Devices
> +will remain PCI and have transitional behaviour as default.
> +Virtio devices plugged into PCIe ports are Express devices and have
> +"1.0" behavior by default without IO support.
> +In both case disable-* properties can be used to override the behaviour.
> +
> +
> +7. Conclusion
> +==============
> +The proposal offers a usage model that is easy to understand and follow
> +and in the same time overcomes some PCIe limitations.
> +
> +
> +