Re: [Qemu-devel] [PATCH v8 11/12] vfio: register aer resume notification handler for aer resume

[Alex Williamson]

On Tue, 28 Jun 2016 13:27:21 +0800
Zhou Jie <address@hidden> wrote:

> Hi Alex,
> 
> On 2016/6/28 11:58, Alex Williamson wrote:
> > On Tue, 28 Jun 2016 11:26:33 +0800
> > Zhou Jie <address@hidden> wrote:
> >  
> >> Hi Alex,
> >>  
> >>> The INTx/MSI part needs further definition for the user.  Are we
> >>> actually completely tearing down interrupts with the expectation that
> >>> the user will re-enable them or are we just masking them such that the
> >>> user needs to unmask?  Also note that not all devices support DisINTx.  
> >>
> >> After reset, the "Bus Master Enable" bit of "Command Register"
> >> should be cleared, so MSI/MSI- X interrupt Messages is still disabled.
> >> After reset, the "Interrupt Disable" bit of "Command Register"
> >> should be cleared, so INTx interrupts is enabled.
> >> If the device doesn't support INTx, "Interrupt Disable" bit will
> >> hardware to 0, it is OK here.
> >>
> >> After fatal-error occurs, the user should reset the device and
> >> reinitialize the device.
> >> So I disable the interrupt before host reset the device,
> >> and let user to do the reinitialization.  
> >
> > I'm dubious here.  When DisINTx is not supported by the device or it's
> > marked broken in host quirks, then we can't trust the device to stop
> > sending INTx.  It's hardwired to zero, meaning that it doesn't work or
> > it's been found to be broken in other ways.  So COMMAND register
> > masking is not sufficient for all devices.  
> For Endpoints that generate INTx interrupts, this bit is required.
> For Endpoints that do not generate IN Tx interrupts this bit is
> optional.  If not implemented, this bit must be hardwired to 0b.
> For Root Ports, Switch Ports, and Bridges that generate INTx
> interrupts on their own behalf, this bit is required.
> 
> The above is from "7.5.1.1." of "PCI Express Base Specification 3.1a".
> So I think "Interrupt Disable" bit must be supported by the device
> which can generate INTx interrupts.

And yet we have struct pci_dev.broken_intx_masking and we test for
working DisINTx via pci_intx_mask_supported() rather than simply
looking for a PCIe device.  Some devices are broken and some simply
don't follow the spec, so you're going to need to deal with that or
exclude those devices.
 
> > Also, any time we start
> > changing the state of the device from what the user expects, we risk
> > consistency problems.  We need to consider how the user last saw the
> > device and whether we can legitimately expect them to handle the device
> > in a new state.  If we expect the user to re-initialize the device then
> > would it be more correct to teardown all interrupt signaling such that
> > the device is effectively in the same state as initial handoff when the
> > vfio device fd is opened?  
> Before the user re-initialize the device, host has reseted the device.

How does that happen, aren't we notifying the user at the point the
error occurs, while the device is still in the process or being reset?
My question is how does the user know that the host reset is complete
in order to begin their own re-initialization?

> The interrupt status will be cleared by hardware.
> So the hardware is the same as the state when the
> vfio device fd is opened.

The PCI-core in Linux will save and restore the device state around
reset, how do we know that vfio-pci itself is not racing that reset and
whether PCI-core will restore the state including our interrupt masking
or a state without it?  Do we need to restore the state to the one we
saved when we originally opened the device?  Shouldn't that mean we
teardown the interrupt setup the user had prior to the error event?
 
> > How will the user know when the device is
> > ready to be reset?  Which of the ioctls that you're blocking can they
> > poll w/o any unwanted side-effects or awkward interactions?  Should
> > flag bits in the device info ioctl indicate not only support for this
> > behavior but also the current status?  Thanks,  
> I can block the reset ioctl and config write.
> I will not add flag for the device current status,
> because I don't depend on user to prevent awkward interactions.

Ok, so that's a reason to block rather than return -EAGAIN.  Still we
need some way to indicate to the user whether the device supports this
new interaction rather than the existing behavior.  Thanks,

Alex