Re: [Qemu-devel] KVM call minutes 2013-01-29 - Port I/O

[Alex Williamson]

On Thu, 2013-01-31 at 23:11 +0200, Michael S. Tsirkin wrote:
> On Thu, Jan 31, 2013 at 09:34:03AM -0700, Alex Williamson wrote:
> > 
> > On Thu, 2013-01-31 at 12:49 +0200, Michael S. Tsirkin wrote:
> > > On Wed, Jan 30, 2013 at 04:28:30PM -0700, Alex Williamson wrote:
> > > > On Thu, 2013-01-31 at 10:02 +1100, Benjamin Herrenschmidt wrote:
> > > > > On Thu, 2013-01-31 at 00:49 +0200, Michael S. Tsirkin wrote:
> > > > > > > In practice they do (VGA at least)
> > > > > > > 
> > > > > > > >From a SW modelling standpoint, I don't think it's worth
> > > > > > differentiating
> > > > > > > PCI and PCIE.
> > > > > > > 
> > > > > > > Cheers,
> > > > > > > Ben.
> > > > > > 
> > > > > > Interesting.
> > > > > > Do you have such hardware? Could you please dump
> > > > > > the output of lspci -vv?
> > > > > 
> > > > > Any ATI or nVidia card still supports hard decoding of VGA regions for
> > > > > the sake of legacy operating systems and BIOSes :-) I don't know about
> > > > > Intel but I suppose it's the same.
> > > > 
> > > > For example:
> > > > 
> > > > -[0000:00]-+-00.0  Advanced Micro Devices [AMD] nee ATI RD890 PCI to 
> > > > PCI bridge (external gfx0 p
> > > >            +-04.0-[02]--+-00.0  Advanced Micro Devices [AMD] nee ATI 
> > > > Cedar PRO [Radeon HD 5450/6350]
> > > > 
> > > > 00:04.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to 
> > > > PCI bridge (PCI express gpp port D) (prog-if 00 [Normal decode])
> > > >         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
> > > > ParErr- Stepping- SERR- FastB2B- DisINTx-
> > > >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> > > > <TAbort- <MAbort- >SERR- <PERR- INTx-
> > > >         Latency: 0, Cache Line Size: 64 bytes
> > > >         Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
> > > >         I/O behind bridge: 0000c000-0000cfff
> > > >         Memory behind bridge: fd100000-fd1fffff
> > > >         Prefetchable memory behind bridge: 
> > > > 00000000d0000000-00000000dfffffff
> > > >         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> > > > <TAbort- <MAbort+ <SERR- <PERR-
> > > >         BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B-
> > > >                                       ^^^^
> > > > VGA+ (VGA Enable) indicates positive decode of 0x3b0 - 0x3bb, 0x3c0 -
> > > > 0x3df, and 0xa0000 - 0xbfff.  Device 2:00.0 of course doesn't report
> > > > these "ISA" ranges as they're implicit in the VGA class code.
> > > 
> > > OK but this appears behind a bridge.  So the bridge configuration tells
> > > the root complex where to send accesses to the VGA.
> > > 
> > > But qemu currently puts devices directly on root bus.
> > > 
> > > And as far as I can tell when we present devices directly on bus 0, we
> > > pretend these are integrated in the root complex. The spec seems to
> > > say explicitly that root complex integrated devices should not use legacy
> > > addresses or support hotplug. So I would be surprised if such one
> > > appears in real world.
> > > 
> > > Luckily guests do not seem to be worried as long as we use ACPI.
> > 
> > Yes, in fact I just figured out last night that Windows is unhappy with
> > assigned PCI devices on bus 0 that claim to be an endpoint in their PCIe
> > capability rather than an integrated endpoint.  We'll need to do extra
> > mangling of the PCIe capability to massage it into the guest visible
> > topology.
> 
> For now, just put you device behind an express bridge. This breaks acpi
> hotplug for now, but I'm looking into hotplug with bridges anyway.

We have the problem in both directions though, Endpoints that should be
Integrated Endpoints and Integrated Endpoints that should be Endpoints.
So I think we need to mangle the type.

> If you really need it I can give you a hack for hotplug too.
> 
> Of course express  does not allow hotplug of root complex parts
> but happens to work because we use ACPI.

That's a little odd.

> > Section 1.3.2.3 of the 3.0 spec says integrated endpoints must not
> > require I/O resources claimed through BAR(s).  VGA skirts around this by
> > not having the legacy resources claimed by BARs, but instead being
> > implicit.
> 
> Aha. I missed this point.
> 
> >  Are there other sections restricting legacy I/O?
> 
> One other interesting things is that VGA enable bit (for bridge control
> register) does not appear in express spec at all.

Yep, but it appears on hardware.

> > It's common that a plugin VGA card sits behind a root port where the
> > bridge registers tell us about VGA routing,
> > but integrated VGA devices
> > are often on bus 0 though, here's an example:
> > 
> > -[0000:00]-+-00.0  Intel Corporation 2nd Generation Core Processor Family 
> > DRAM Controller
> >            +-02.0  Intel Corporation 2nd Generation Core Processor Family 
> > Integrated Graphics Controller
> > 
> > Often these systems will disable the integrated graphics when a plugin
> > graphics is installed below a root port.  I'm not sure how the system
> > knows to route VGA to the integrated device vs the root port otherwise.
> 
> I am guessing it disables the integrated graphics?
> 
> > Here's a more interesting example:
> > 
> > -+-[0000:01]-+-00.0  NVIDIA Corporation GT218 [GeForce G210M]
> >  |           \-00.1  NVIDIA Corporation High Definition Audio Controller
> >  \-[0000:00]-+-00.0  Intel Corporation Mobile 4 Series Chipset Memory 
> > Controller Hub
> >              +-01.0  Intel Corporation Mobile 4 Series Chipset PCI Express 
> > Graphics Port
> > 
> > This system seems to have two host bridges with VGA behind each of them.
> > There's no bridge to control VGA routing, so I don't know how the
> > selection is done.
> 
> Is IO space disabled for the inactive card? Maybe that is how.

The card has BAR defined I/O space resources.  My guess is that VGA is
just statically routed to the integrated device and the secondary works
only in non-legacy mode until the BIOS switch is flipped, the integrated
device is hidden and VGA is switched to static routing for the nvidia
device.  I suppose that means I'll never be able to assign the nvidia to
a guest, at least not with any kind of legacy VGA support.  Thanks,

Alex