Re: [Qemu-devel] KVM call minutes 2013-01-29 - Port I/O

[Alex Williamson]

On Thu, 2013-01-31 at 12:49 +0200, Michael S. Tsirkin wrote:
> On Wed, Jan 30, 2013 at 04:28:30PM -0700, Alex Williamson wrote:
> > On Thu, 2013-01-31 at 10:02 +1100, Benjamin Herrenschmidt wrote:
> > > On Thu, 2013-01-31 at 00:49 +0200, Michael S. Tsirkin wrote:
> > > > > In practice they do (VGA at least)
> > > > > 
> > > > > >From a SW modelling standpoint, I don't think it's worth
> > > > differentiating
> > > > > PCI and PCIE.
> > > > > 
> > > > > Cheers,
> > > > > Ben.
> > > > 
> > > > Interesting.
> > > > Do you have such hardware? Could you please dump
> > > > the output of lspci -vv?
> > > 
> > > Any ATI or nVidia card still supports hard decoding of VGA regions for
> > > the sake of legacy operating systems and BIOSes :-) I don't know about
> > > Intel but I suppose it's the same.
> > 
> > For example:
> > 
> > -[0000:00]-+-00.0  Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI 
> > bridge (external gfx0 p
> >            +-04.0-[02]--+-00.0  Advanced Micro Devices [AMD] nee ATI Cedar 
> > PRO [Radeon HD 5450/6350]
> > 
> > 00:04.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI 
> > bridge (PCI express gpp port D) (prog-if 00 [Normal decode])
> >     Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> > Stepping- SERR- FastB2B- DisINTx-
> >     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
> > <MAbort- >SERR- <PERR- INTx-
> >     Latency: 0, Cache Line Size: 64 bytes
> >     Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
> >     I/O behind bridge: 0000c000-0000cfff
> >     Memory behind bridge: fd100000-fd1fffff
> >     Prefetchable memory behind bridge: 00000000d0000000-00000000dfffffff
> >     Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
> > <MAbort+ <SERR- <PERR-
> >     BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B-
> >                                       ^^^^
> > VGA+ (VGA Enable) indicates positive decode of 0x3b0 - 0x3bb, 0x3c0 -
> > 0x3df, and 0xa0000 - 0xbfff.  Device 2:00.0 of course doesn't report
> > these "ISA" ranges as they're implicit in the VGA class code.
> 
> OK but this appears behind a bridge.  So the bridge configuration tells
> the root complex where to send accesses to the VGA.
> 
> But qemu currently puts devices directly on root bus.
> 
> And as far as I can tell when we present devices directly on bus 0, we
> pretend these are integrated in the root complex. The spec seems to
> say explicitly that root complex integrated devices should not use legacy
> addresses or support hotplug. So I would be surprised if such one
> appears in real world.
> 
> Luckily guests do not seem to be worried as long as we use ACPI.

Yes, in fact I just figured out last night that Windows is unhappy with
assigned PCI devices on bus 0 that claim to be an endpoint in their PCIe
capability rather than an integrated endpoint.  We'll need to do extra
mangling of the PCIe capability to massage it into the guest visible
topology.

Section 1.3.2.3 of the 3.0 spec says integrated endpoints must not
require I/O resources claimed through BAR(s).  VGA skirts around this by
not having the legacy resources claimed by BARs, but instead being
implicit.  Are there other sections restricting legacy I/O?

It's common that a plugin VGA card sits behind a root port where the
bridge registers tell us about VGA routing, but integrated VGA devices
are often on bus 0 though, here's an example:

-[0000:00]-+-00.0  Intel Corporation 2nd Generation Core Processor Family DRAM 
Controller
           +-02.0  Intel Corporation 2nd Generation Core Processor Family 
Integrated Graphics Controller

Often these systems will disable the integrated graphics when a plugin
graphics is installed below a root port.  I'm not sure how the system
knows to route VGA to the integrated device vs the root port otherwise.

Here's a more interesting example:

-+-[0000:01]-+-00.0  NVIDIA Corporation GT218 [GeForce G210M]
 |           \-00.1  NVIDIA Corporation High Definition Audio Controller
 \-[0000:00]-+-00.0  Intel Corporation Mobile 4 Series Chipset Memory 
Controller Hub
             +-01.0  Intel Corporation Mobile 4 Series Chipset PCI Express 
Graphics Port

This system seems to have two host bridges with VGA behind each of them.
There's no bridge to control VGA routing, so I don't know how the
selection is done.  It's possible the g210m never sees legacy VGA
accesses in this mode.  This bios has another mode which makes the g210m
the primary graphics and hides the integrated graphics, essentially the
same as I mention above with hiding integrated endpoint graphics when
plugin graphics are used.  Thanks,

Alex