Re: [Qemu-arm] [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Mike Caraman]

Peter, thanks for feedback. Diana will return from vacation next week,
I am trying clarify some of the questions.

>From: Peter Maydell <address@hidden>
>Sent: Monday, July 10, 2017 8:10 PM
>To: Diana Madalina Craciun
>Cc: QEMU Developers; Michael S. Tsirkin; Eric Auger; Mike Caraman; qemu-arm; Marcel Apfelbaum; Bharat Bhushan; Christoffer Dall; Laurentiu Tudor
>Subject: Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine
>
>On 23 May 2017 at 12:12, Diana Craciun <address@hidden> wrote:
>> The NXP DPAA2 is a hardware architecture designed for high-speeed network
>> packet processing. The DPAA2 hardware components are managed by a hardware
>> component called the Management Complex (or MC) which provides an
>> object-base abstraction for software drivers to use the DPAA2 hardware.
>> For more details you can see:
>> https://emea01.safelinks.protection.outlook.com/?url=""> >>
>> The interrupts generated by the DPAA2 hardware components are MSIs. We will add
>> support for direct assigning these DPAA2 components/objects to a virtual
>> machine. However, this will add the need to expand the MSI usage in QEMU.
>>
>> Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the
>> GIC ITS is using a device ID for interrupt translation. Currently, for
>> PCI, the requester ID is used as device ID. This will not work when
>> we add another entity that needs also a device ID which is supposed to
>> be unique across the system.
>>
>> My proposal is to add a static allocation in the virt machine. I considered
>> that this allocation is specific to each machine/platform. Currently only
>> virt machine has it, but other implementations may use the same mechanism
>> as well.
>> So, I used a static allocation with this formula:
>>
>> DeviceID = zero_extend( RequesterID[15:0] ) + 0x10000 * Constant
>>
>> This formula was taken from SBSA spec (Appendix I: DeviceID generation and
>> ITS groups). In case of QEMU the constant will be different for each entity.
>> In this way a unique DeviceID will be generated and the device ID will be
>> derived from a requesterID (in case of PCI) or other means in case of other
>> entities.
>>
>> The implementation is generic as there might be in the future other non-pci devices
>> that are using MSIs or IOMMU. Any architecture can use it, though currently
>> only the ARM architecture is using the function that retrieves the stream ID. I
>> did not change all the replacements of the pci_requester_id (with pci_stream_id)
>> in the code (although if the constant is 0, the stream_id is equal with requester_id).
>> The other architectures (e.g. intel iommu code) assume that the ID is the
>> requester ID.
>>
>> Tested on NXP LS2080 platform.
>
>So I'm still a bit confused about what this is for and what the
>relationship is with how the equivalent task would be performed
>in real hardware. Can you explain how that works, please?
>Would there be a system MMU involved somewhere that would be
>allocating these IDs somehow?

This is the first step to accommodate device pass-though with MSI support,
required for non-PCI/heterogeneous buses on ARM64 platforms with GIC-ITS.

We reached the point where our mc-bus infrastructure got the ack and
it is ready to get out of staging:
https://lkml.org/lkml/2017/7/18/708

The problem we want to address is related to guest DeviceIDs allocation
for GIC-ITS emulation. On real hardware the boot loader, in our case u-boot,
allocates the DeviceIDs and populates the PCI msi-map device tree binding.
We have a similar binding mechanism for MC devices:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/pci-msi.txt
http://git.denx.de/?p=u-boot.git;a=blob;f=drivers/pci/pcie_layerscape_fixup.c;h=9e6c2f5dfcf62e8b06935aa9ae3c40a734f9571d;hb=HEAD#l118

Qemu does not populate yet the msi-map and the DeviceIDs for PCI devices
defaults to bdf at run-time:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/of/irq.c#n599
http://elixir.free-electrons.com/linux/latest/source/drivers/pci/msi.c#L1488

The proposal is to allocate the DeviceIDs as described in SBSA and to populate the
device tree, as oppose of generating them at runtime, in order to stick to device
binding definitions and to avoid clashes.

>The patchset also seems to be introducing a fair amount of
>mechanism without also implementing the actual use for it -- yes,
>if we had two PCI controllers in the virt board we would need to
>sort something out I guess, but we don't, we only have one.
>Evaluating whether the mechanism makes sense without its major
>user is always trickier.
>
>Essentially, I know very little about this use case, but my
>default position is that if in doubt we should aim to match
>how real hardware handles something. Virtualization-specific
>code, and especially something specific to a niche use case like
>this, is something I'd really prefer to avoid. The ideal is to
>have something like PCI device passthrough, where the guest can
>automatically probe for things and QEMU doesn't need to get
>involved.
>
>I was hoping Eric would review this, since he's more up to speed
>on direct hardware passthrough than I am, but I think he's just
>gone off on holiday...
>
>thanks
>-- PMM
>

Regards,
Mike