Re: [Qemu-devel] vfio-pci issues with multiple devices on the same root port

[Alex Williamson]

On Sat, 2014-12-13 at 21:36 +0100, Peter Lieven wrote:
> Am 12.12.2014 um 23:21 schrieb Alex Williamson:
> > On Fri, 2014-12-12 at 22:38 +0100, Peter Lieven wrote:
> >> Hi,
> >>
> >> we have a Cisco UCS infrastructure where we have fnic Fibre-Channel 
> >> Adapters that we expose to guests. The UCS
> >> infrastruture allows to create virtual HBAs that can be exposed to a host 
> >> so its possible to have quite a lot of them.
> >>
> >> We ran into a strange issue when we started having more than one vServer 
> >> with a FibreChannel Adapter passed
> >> thru with vfio-pci.
> >>
> >> When a hypervisor shuts down it the kernel sees the following error:
> >>
> >>  pcieport 0000:00:07.0: AER: Uncorrected (Non-Fatal) error received: 
> >> id=0038
> >>  pcieport 0000:00:07.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), 
> >> type=Transaction Layer, id=0038(Receiver ID)
> >>  pcieport 0000:00:07.0:   device [8086:340e] error 
> >> status/mask=00200000/00100000
> >>  pcieport 0000:00:07.0:    [21] Unknown Error Bit (First)
> >>  pcieport 0000:00:07.0: broadcast error_detected message
> >>  pcieport 0000:00:07.0: AER: Device recovery failed
> >>
> >> Bit 21 seems to be ACS Violation. And 0000:00:07.0 is the PCIE Root Port 
> >> on that System.
> >>
> >> This wouldn't be a big problem, altough I would like to find out what the 
> >> ACS Violation causes.
> >>
> >> The real problem is that all other vfio-pci cards on that root port get 
> >> notified of this error and the connected vServers are suspended
> >> with RUN_STATE_INTERNAL_ERROR.
> >>
> >> Any ideas to work around this other than hacking qemu to not register an 
> >> error handler or modifying vfio_err_notifier_handler
> >> to not suspend the vServer?
> > You could set bit 21 in the AER uncorrected error mask register to avoid
> > the root port signaling the error.  Is bit 21 already clear in the
> > severity register to make this non-fatal?
> 
> Can you give me a hint where I find those registers and how I mangle them?
> At least the syslog output states non-fatal.

I'd use setpci.  'lspci -v' will give you the offset of the AER
capability, for example:

02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Oland 
[Radeon HD 8570 / R7 240 OEM] (prog-if 00 [VGA controller])
        Subsystem: Dell Device 2121
        Flags: fast devsel, IRQ 16
        Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Memory at f7900000 (64-bit, non-prefetchable) [disabled] [size=256K]
        I/O ports at d000 [disabled] [size=256]
        Expansion ROM at f7940000 [disabled] [size=128K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 
<?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [270] #19
        Kernel driver in use: vfio-pci
        Kernel modules: radeon

In this case AER is at offset 0x150.  The status, mask, and severity
registers are dwords at offset 0x4, 0x8, and 0xc respectively.
Therefore to set bit 21 in the mask register with an AER offset of
0x150, I'd do this:

setpci -s 02:00.0 158.l=200000:200000

This notation sets the bits indicated on the left, using the mask on the
right.  It will do a read-modify-write so that it ORs in just that bit.
Thanks,

Alex

> I am not that familiar with PCI internals, I am more the Block Layer guy.
> 
> >
> >> Is it correct that all children of a root port are notified? Should qemu 
> >> distinguish between fatal and non-fatal errors when
> >> suspending a vServer?
> > Yes, each child is notified.  QEMU only gets an eventfd signal, which is
> > supposed to occur only for fatal errors.  I don't quite understand why
> > this apparently non-fatal error is getting through.  The kernel-side
> > VFIO code is where filtering of fatal vs non-fatal should occur.
> 
> I will look at that. Has there been any bugfix since 3.13 ?
> 
> Peter