Re: [RFC][PATCH] docs: note exception for PCIe IO port access

[Kevin Locke]

Thanks Laszlo, I really appreciate the detailed explanation and
feedback!

On Thu, 2022-06-09 at 09:45 +0200, Laszlo Ersek wrote:
> (a) your original (non-functional) use case:
> 
> qemu-system-x86_64 \
>     -no-user-config \
>     -nodefaults \
>     -machine q35,accel=kvm \
>     -m 1G \
>     -cdrom "$iso" \
>     -device pcie-root-port,id=pci.1,bus=pcie.0 \
>     -device VGA,bus=pci.1
> 
> violates the following part of "pcie.txt":
> 
>> Plugging a PCI device into a PCI Express slot might not always work and
>> is weird anyway since it cannot be done for "bare metal".
> 
> AIUI, what Gerd explains in that SeaBIOS thread is why and how exactly
> such an attempt breaks. The statement that it would break is already
> spelled out in "pcie.txt".
> 
> So I think that, given strictly your original report on the SeaBIOS
> list, no updates to "pcie.txt" are necessary.
> 
> Note that your original (functional) use case:
> 
> qemu-system-x86_64 \
>     -no-user-config \
>     -nodefaults \
>     -machine q35,accel=kvm \
>     -m 1G \
>     -cdrom "$iso" \
>     -device VGA
> 
> does not conflict with
> 
>> Place only the following kinds of devices directly on the Root Complex:
>>     (1) PCI Devices (e.g. network card, graphics card, IDE controller),
>>         not controllers. Place only legacy PCI devices on
>>         the Root Complex. These will be considered Integrated Endpoints.
>>         Note: Integrated Endpoints are not hot-pluggable.
>>
>>         Although the PCI Express spec does not forbid PCI Express devices as
>>         Integrated Endpoints, existing hardware mostly integrates legacy PCI
>>         devices with the Root Complex. Guest OSes are suspected to behave
>>         strangely when PCI Express devices are integrated
>>         with the Root Complex.
> 
> because "-device VGA" is a legacy PCI device, not a PCI Express device.
> So the second quoted paragraph does not apply to it at all, and the
> first paragraph is in sync with your functional use case.

Ah, right you are.  My mistake.  Sorry for the noise.

I had thought virtio-vga might be an exception, since there was
discussion about changing virtio-vga to PCIe when connected to the
Root Complex[4], but it looks like the change was not made due to
compatibility concerns[5].  As you note below, this is currently the
case for all devices with VGA compatibility, so this docs change is
mostly a non-issue as long as this continues to be the case.

On Thu, 2022-06-09 at 09:45 +0200, Laszlo Ersek wrote:
> While VGA is quirky ("there was only ever intended to be one device" --
> see
> <http://vfio.blogspot.com/2014/08/whats-deal-with-vga-arbitration.html>),
> I maintain that it's sufficiently covered already (although not
> specifically) by "pcie.txt"; see above. "-device VGA" is a legacy PCI
> device, you can't plug it in an Express slot (root complex or port
> alike).
> 
> In the blog post at
> <https://www.kraxel.org/blog/2019/09/display-devices-in-qemu/>, Gerd
> explains that (effectively) only "bochs-display" and "virtio-gpu-pci"
> (from QEMU's emulated devices) may be placed in PCI Express slots (root
> complex or separate port, alike).

Those posts are both enlightening, thanks!

On Thu, 2022-06-09 at 09:45 +0200, Laszlo Ersek wrote:
> On 06/09/22 04:00, Kevin Locke wrote:
>> and whether "Guest OSes are suspected to behave strangely when PCI
>> Express devices are integrated with the Root Complex" is still the case.
> 
> I vaguely recall that Alex didn't fully agree with this statement when
> the document was being reviewed.
> 
>> David Gibson had previously noted that graphics cards appear on the Root
>> Complex on real hardware.[3]
> 
> You forgot to provide the link for [3].

Whoops.  Links to my qemu-discuss post[2] and David Gibson's post[3]
mentioning "the GPU shows up as an integrated PCI Express endpoint" on
his laptop are included in this email below.

On Thu, 2022-06-09 at 09:45 +0200, Laszlo Ersek wrote:
> On 06/09/22 04:00, Kevin Locke wrote:
> Anyway, the full paragraph in the document goes like
> 
>>         Although the PCI Express spec does not forbid PCI Express devices as
>>         Integrated Endpoints, existing hardware mostly integrates legacy PCI
>>         devices with the Root Complex. Guest OSes are suspected to behave
>>         strangely when PCI Express devices are integrated
>>         with the Root Complex.
> 
> So the recommendation is based on motherboards seen in the wild, and the
> potential consequent assumptions in OSes. Whether that still holds up
> with today's motherboards (and OSes), I have no clue about.
> 
> [...]
> 
>> --- a/docs/pcie.txt
>> +++ b/docs/pcie.txt
>> @@ -48,6 +48,11 @@ Place only the following kinds of devices directly on the 
>> Root Complex:
>>          strangely when PCI Express devices are integrated
>>          with the Root Complex.
>>
>> +        An exception to this rule is PCI Express devices which will be
>> +        accessed using IO ports.
> 
> I find this too general; a PCI Express device is supposed to work
> without IO resources. Graphics cards with legacy VGA compatibility are
> the exception AIUI (see again Alex's blog about VGA arbitration), so we
> should spell that out.
> 
> Furthermore, from QEMU's emulated graphics cards, all device models that
> provide VGA compatibility are legacy PCI (not PCI Express) devices (see
> again Gerd's post: VGA, virtio-vga, qxl-vga, cirrus-vga, ati-vga), so
> I'd even restrict this docs update to assigned GPUs.
> 
> [...]
> 
> How about inserting a new paragraph here, instead:
> 
>     (2) Assigned PCI Express GPUs that offer legacy VGA compatibility,
>         and that such compatibility is expected of (due to booting with
>         SeaBIOS, or due to UEFI driver bugs or native OS driver bugs).

That sounds great to me.  Would you like me to send that as a v2
patch, or would you prefer to make the change?

Thanks again,
Kevin

[2]: https://lists.nongnu.org/archive/html/qemu-discuss/2022-05/msg00053.html
[3]: https://lore.kernel.org/qemu-devel/20170209041634.GC14524@umbus/
[4]: 
https://lore.kernel.org/qemu-devel/20170208061602.17666-1-david@gibson.dropbear.id.au/T/
[5]: 
https://lore.kernel.org/qemu-devel/5ea3785c-b979-8b8c-3ab0-243d69384697@redhat.com/