Re: [Qemu-devel] [PATCH] vfio-pci: Fix Nvidia MSI ACK through 0x88000 quirk

[Alex Williamson]

On Tue, 2013-11-12 at 07:55 +1000, Dave Airlie wrote:
> On Tue, Nov 12, 2013 at 7:43 AM, Alex Williamson
> <address@hidden> wrote:
> > When MSI is enabled on Nvidia GeForce cards the driver seems to
> > acknowledge the interrupt by writing a 0xff byte to the MSI capability
> > ID register using the PCI config space mirror at offset 0x88000 from
> > BAR0.  Without this, the device will only fire a single interrupt.
> > VFIO handles the PCI capability ID/next registers as virtual w/o write
> > support, so any write through config space is currently dropped.  Add
> > a check for this and allow the write through the BAR window.  The
> > registers are read-only anyway.
> 
> This is only half the truth, I'm afraid if I'm right its much worse than that.
> 
> At least on some GPUs the MSI ack is done via PCI config space itself,
> and on some its done via the mirror, and yes it matters on some cards
> which way it works.

I was hoping that wouldn't be the case since it seems fairly universal
that PCI config space access should be considered slow and avoided for
things like this.  But, I suppose with MMConfig it's no worse than
device MMIO space.

On my GTX660 I did actually test both paths.  The existing quirk
converts all accesses into the config space mirror into proper config
space accesses, so I changed the kernel to not drop the write.  Then I
figured I'd rather solve this in QEMU using the mirror directly.  So in
my case it didn't matter which path.

I'll keep this in mind though, I may want to enable kernel support for
this first and advertise it via a flag on the config space region, then
I can avoid doing a double write in QEMU and really breaking the state
machine.  Thanks for the feedback,

Alex