[Qemu-devel] [RFC PATCH 0/7] qemu: CCIX pcie config space emulation

[Jonathan Cameron]

CCIX topologies are 'layered' on top of PCIe tree topologies.
This is done primarily by allowing a single CCIX device to appear as
multiple disjoint nodes in the PCIe tree.

This layering is described via extensive PCIe DVSEC extended
capabilities in PCIe config space across all the functions that
are present in the device (some placement rules apply).

The extremely flexible nature of allowed topologies makes the
development of generic firmware and OS software difficult if we rely
on actual hardware setups, due to the large test set that is necessary.

To enable the ongoing work on EDK2 and within the Linux kernel and
userspace, this series provides the bare bones of what is necessary
to be able to test 'configuration' of a CCIX topology.  Note
that no actual CCIX data flow is being emulated within this patchset,
merely a substantial subset of the interface that allows the topologies
to be configured.  Testing has to rely on verification based on
the result rather than true emulation of the coherency protocol.
(that is a very different form of emulation for which other tools
are perhaps better suited).

For information on how to do the coherency protocol configuration,
see the forthcoming CCIX SW guide, in conjunction with the CCIX 1.0
Base Specification.

An example of a 2x2 mesh with a spur to the host can be run with:

qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a53 \
...
 -device ioh3420,id=root_port1 \
 -device 
ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev1",bus=root_port1,addr=0.0,multifunction="on",id=up0,port_id=0
 \
 -device 
ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=0,chassis=2,id=bus_top,port_id=1
 \
 -device 
ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=1,chassis=2,id=bus_left,port_id=2
 \
 -device 
ccix-ep,primaryport=false,home_agents=1,request_agents=1,ccix_device="ccix_dev1",bus=root_port1,addr=0.1,multifunction="on"
 \
 -device 
ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev2",bus=bus_top,addr=0.0,multifunction="on",id=up1,port_id=0
 \
 -device 
ccix-downstream-port,ccix_device="ccix_dev2",bus=up1,slot=0,chassis=3,id=bus_right,port_id=1
 \
 -device 
ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev2",bus=bus_top,addr=0.1,multifunction="on"
 \
 -device 
ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev3",bus=bus_left,addr=0.0,multifunction="on",id=up2,port_id=0
 \
 -device 
ccix-downstream-port,ccix_device="ccix_dev3",bus=up2,slot=0,chassis=4,id=bus_bottom,port_id=1,multifunciton="on"
 \
 -device 
ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev3",bus=bus_left,addr=0.1,multifunction="on"
 \
 -device 
ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_right,addr=0.0,port_id=0
 \
 -device 
ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_bottom,addr=0.0,port_id=1


I'm not going to try drawing all the detail (it's bad enough
trying to draw these in inkscape, but in a very much simplifed
fashion, this generates.

-----------------
|     Host      |
|               | 
--- root_port1--
        |
        |
        v
-----------------          ---------------
|  ccix_dev1    | -------> |  ccix_dev2  |
-----------------          ---------------
        |                         |
        V                         V
-----------------          ---------------
|  ccix_dev3    | -------> |  ccix_dev4  |
-----------------          ---------------

$lspci -t
-[0000:00]-+-00.0
           
+-01.0-[01-08]--+-00.0-[02-08]--+-00.0-[03-05]--+-00.0-[04-05]----00.0-[05]----00.0
                           |               |               |               
\-00.1
                           |               |               
\-01.0-[06-08]--+-00.0-[07-08]----00.0-[08]----00.0
                           |               |                               
\-00.1
                           |               \-00.1
                                                       
RFC questions:

1.  The nature of CCIX devices is that we need to extend normal
    PCI devices, slots, and ports.  I could not find an elegant way of
    doing this without lots of code replication.  The current solution
    just exposes some internal functions from xio3130 port implementations.
    Is there a better way to do this?

2.  The association of the different PCIDevices to a given CCIX device is
    currently somewhat of a hack. Can anyone suggest a cleaner solution
    for this?  I can improved the current implementation, but don't really
    like that we basically search for all the parts whenever a cross
    device implementation is needed.

3.  Is emulation via a large number of PCIe devices like this a good
    approach or is there a nicer way to handle it?  Given we need to
    be able to 'spread' the CCIX device configuration across multiple
    PCIe functions anyway perhaps this is the most sensible approach.

4.  I've cut and paste a 100+ lines of code from each of the xio3130_*
    modules as we are also implemening PCIE up and downstream ports
    and as this is a emulation only device, we might as well use the
    same register set.  There are various possible ways to avoid this:
    * Add a library with the shared code in it.
    * Add an xio3130_upstream.h header and similar to allow the CCIX
      port modules to call those functions directly.
    * Don't worry about the replication in the interests of keeping
      the code structure simple.

5.  I'm not that familiar with qemu 'style' yet, so pointers on structures
    to use etc most welcome.

Note that not all elements of CCIX are supported by the current implementation,
for example slave agents and error records are missing.  These will follow
either in later revisions or as follow patches.  We also have no actual
accelerator functions in the current design and hence no mapping to RAs.

Only a subset of configuration constraints are currently implemented.
This will want tightenning up in future.

As we don't have any actual chunks of the spec in here so I haven't
added the statement from the trademark grant that follows.

Thanks,

Jonathan

This patch is being distributed by the CCIX Consortium, Inc. (CCIX) to
you and other parties that are paticipating (the "participants") in
qemu with the understanding that the participants will use CCIX's
name and trademark only when this patch is used in association with
qemu.

CCIX is also distributing this patch to these participants with the
understanding that if any portion of the CCIX specification will be
used or referenced in qemu, the participants will not modify the cited
portion of the CCIX specification and will give CCIX propery copyright
attribution by including the following copyright notice with
the cited part of the CCIX specification:
"© 2019 CCIX CONSORTIUM, INC. ALL RIGHTS RESERVED."

Jonathan Cameron (7):
  Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.
  pci: Add Huawei vendor ID and Huawei Emulated CCIX Device IDs.
  pci: CCIX config space emulation library.
  pci-bridge: CCIX capable PCIE/CCIX switch upstream port.
  pci-bridge: CCIX capable PCIE/CCIX switch downstream port
  misc: CCIX endpoint function
  Temp: Add to ARM64 makefiles for testing

 default-configs/arm-softmmu.mak           |    3 +-
 hw/misc/Kconfig                           |    5 +
 hw/misc/Makefile.objs                     |    1 +
 hw/misc/ccix-ep.c                         |  112 ++
 hw/pci-bridge/Kconfig                     |    5 +
 hw/pci-bridge/Makefile.objs               |    1 +
 hw/pci-bridge/ccix_downstream.c           |  222 ++++
 hw/pci-bridge/ccix_upstream.c             |  197 ++++
 hw/pci/Kconfig                            |    3 +
 hw/pci/Makefile.objs                      |    1 +
 hw/pci/ccix_lib.c                         | 1299 +++++++++++++++++++++
 include/hw/misc/ccix.h                    |   28 +
 include/hw/pci/pci_ids.h                  |    7 +
 include/standard-headers/linux/pci_regs.h |    3 +-
 14 files changed, 1885 insertions(+), 2 deletions(-)
 create mode 100644 hw/misc/ccix-ep.c
 create mode 100644 hw/pci-bridge/ccix_downstream.c
 create mode 100644 hw/pci-bridge/ccix_upstream.c
 create mode 100644 hw/pci/ccix_lib.c
 create mode 100644 include/hw/misc/ccix.h

-- 
2.20.1