Re: [Qemu-arm] [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Diana Madalina Craciun]

Hi Peter,

On 07/10/2017 08:10 PM, Peter Maydell wrote:

On 23 May 2017 at 12:12, Diana Craciun <address@hidden> wrote:

The NXP DPAA2 is a hardware architecture designed for high-speeed network
packet processing. The DPAA2 hardware components are managed by a hardware
component called the Management Complex (or MC) which provides an
object-base abstraction for software drivers to use the DPAA2 hardware.
For more details you can see:
https://emea01.safelinks.protection.outlook.com/?url="">

The interrupts generated by the DPAA2 hardware components are MSIs. We will add
support for direct assigning these DPAA2 components/objects to a virtual
machine. However, this will add the need to expand the MSI usage in QEMU.

Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the
GIC ITS is using a device ID for interrupt translation. Currently, for
PCI, the requester ID is used as device ID. This will not work when
we add another entity that needs also a device ID which is supposed to
be unique across the system.

My proposal is to add a static allocation in the virt machine. I considered
that this allocation is specific to each machine/platform. Currently only
virt machine has it, but other implementations may use the same mechanism
as well.
So, I used a static allocation with this formula:

DeviceID = zero_extend( RequesterID[15:0] ) + 0x10000 * Constant

This formula was taken from SBSA spec (Appendix I: DeviceID generation and
ITS groups). In case of QEMU the constant will be different for each entity.
In this way a unique DeviceID will be generated and the device ID will be
derived from a requesterID (in case of PCI) or other means in case of other
entities.

The implementation is generic as there might be in the future other non-pci devices
that are using MSIs or IOMMU. Any architecture can use it, though currently
only the ARM architecture is using the function that retrieves the stream ID. I
did not change all the replacements of the pci_requester_id (with pci_stream_id)
in the code (although if the constant is 0, the stream_id is equal with requester_id).
The other architectures (e.g. intel iommu code) assume that the ID is the
requester ID.

Tested on NXP LS2080 platform.

So I'm still a bit confused about what this is for and what the
relationship is with how the equivalent task would be performed
in real hardware. Can you explain how that works, please?
Would there be a system MMU involved somewhere that would be
allocating these IDs somehow?

Let's keep the discussion to MSIs for now (assuming that there is no SMMU as QEMU does not have currently an SMMU emulation integrated).
Every device that sends MSIs has a deviceID associated with it. The deviceID is used by the GIC-ITS hardware for interrupt translation. The deviceID is assigned by the "system designer", so there is not an hardware entity that generates/allocates the deviceIDs (the device ID can be programmed into the device by software).

For PCI these IDs are specified in the device tree:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/pci-msi.txt

(msi-map property).

The patchset also seems to be introducing a fair amount of
mechanism without also implementing the actual use for it -- yes,
if we had two PCI controllers in the virt board we would need to
sort something out I guess, but we don't, we only have one.
Evaluating whether the mechanism makes sense without its major
user is always trickier.

Right, we have only one PCI controller in the QEMU for now. We are in the process of implementing device pass-through for our non-PCI bus infrastructure which is using MSIs as well. As Mike pointed out, the mc-bus infrastructure is on the point of getting out of staging:

"
The fsl-mc is a hardware resource manager that manages specialized hardware objects used in network-oriented packet processing applications. After the fsl-mc block is enabled, pools of hardware resources are available, such as queues, buffer pools, I/O interfaces. These resources are building blocks that can be used to create functional hardware objects such as network interfaces, crypto accelerator instances, or L2 switches.

All the fsl-mc managed hardware resources/objects are represented in a physical grouping mechanism called a 'container' or DPRC (data path resource container).

>From the point of view of an OS, a DPRC functions similar to a plug and play bus. Using fsl-mc commands software can enumerate the contents of the DPRC discovering the hardware objects present and binding them to drivers. Hardware objects can be created and removed dynamically, providing hot pluggability of the hardware objects.
"

After adding pass-through support, we will have two entities that are using MSIs and thus that are needing unique deviceIDs.
PCI is making the assumption that the deviceID=requesterID, so, theoretically, we could allocate numbers bigger than the requesterID for our bus infrastructure. But I thought that this is more like an temporary workaround which will not work in the future when: another PCI controller is added or another platform device that is using MSIs is added.

Essentially, I know very little about this use case, but my
default position is that if in doubt we should aim to match
how real hardware handles something. Virtualization-specific
code, and especially something specific to a niche use case like
this, is something I'd really prefer to avoid. The ideal is to
have something like PCI device passthrough, where the guest can
automatically probe for things and QEMU doesn't need to get
involved.

It is not automatically probed for PCI, either. In the current implementation an assumption is made:

if the msi-map is missing from the device tree (property which specifies the translation between requesterID and deviceID) it is asummed that the deviceID is the same with the requesterID.

I was hoping Eric would review this, since he's more up to speed
on direct hardware passthrough than I am, but I think he's just
gone off on holiday...

thanks
-- PMM

Thanks,

Diana