Re: [RFC] hw/arm/virt: Provide DT binding generation for PCI eXpander Bridges

[Jonathan Cameron]

On Mon, 24 Apr 2023 10:28:30 +0100
Peter Maydell <peter.maydell@linaro.org> wrote:

> On Fri, 21 Apr 2023 at 17:50, Jonathan Cameron
> <Jonathan.Cameron@huawei.com> wrote:
> >
> > This patch and the problem it is trying to resolve will form part of a talk
> > I will be giving next week at Linaro Connect. https://sched.co/1K85p
> >
> > So in the spirit of giving people almost no warning... Plus being able to
> > say the discussion has kicked off here is the simplest solution I could
> > come up with. If we can agree on an approach to the sizing of memory
> > windows question by Thursday even better (I'll update the slides!) ;)
> >
> > This is a precursor for CXL on arm-virt (for which DT support was
> > requested). CXL QEMU emulation uses pxb-cxl which inherits from the
> > slightly simpler pxb-pcie. ACPI support for these additional host bridges
> > has been available for some time but relies an interesting dance with
> > EDK2:
> > * During initial board creation the PXB buses are largely ignored
> >   and ACPI tables + DT passed to EDK2 only expose the root bus on
> >   which the section of the PXB instance that exists as a normal PCI EP
> >   may be discovered.
> > * EDK2 then carries out full PCI bus enumeration, including the buses
> >   below the PXB host bridges.
> > * Finally EDK2 calls back into QEMU to rebuild the tables via
> >   virt_acpi_build_update(), at which point the correct DSDT for the
> >   PXB host bridges is generated and an adjust DSDT section is generated
> >   for the primary host bridge (holes are bunched in the _CRS).
> >
> > For device tree bindings there is no such dance with the firmware and as
> > such we need QEMU to directly present device tree entries for the primary
> > PCIe bus and the PXB instances.  
> 
> So, not knowing anything about CXL, my naive question is:
> this is PCI, why can't we handle it the way we handle everything
> else PCI, i.e. present the PCI controller in the DTB and it's
> the guest kernel's job to probe, enumerate, etc whatever it wants ?

Absolutely the kernel will still do the enumeration.  But there's a problem
- it won't always work, unless there is 'enough room'.

If the aim is to do it in a similar fashion to how PCI Expander Bridges (PXB)
are handled today with ACPI (we could look at doing something different of 
course)

We have one set of memory windows for assigning PCI bars into etc. In the model
used for PXB, that set of resources is shared between different host bridges
including the main one (each one has separate DT description).

So for virt, VIRT_PCIE_MMIO, VIRT_PCIE_IO, VIRT_PCIE_ECAM, VIRT_HIGH_PCIE_ECAM
and VIRT_HIGH_PCIE_MMIO are split up between the host bridges.
Each host bridge has it's own DT description. 

For ACPI, how to split up available memory windows up depends on the 
requirements
of the PCIe devices below the host bridges.  For ACPI we 'know' the answer
as to what is required at the point of creating the description because EDK2
did the work for us.  For DT we currently don't.  What to do about that is the
question this RFC tries to address.

In theory we could change the kernel to support enumerating PXB instances, but
that's a very different model from today where they are just 'standard'
host bridges, using the generic bindings for ACPI (and after this patch for DT).

I'll add an example in a reply (on wrong machine and stuck in a meeting today).

Jonathan

> 
> thanks
> -- PMM