Re: [PATCH v8 15/17] vfio-user: handle device interrupts

[Alex Williamson]

On Thu, 28 Apr 2022 10:54:04 +0100
Stefan Hajnoczi <stefanha@redhat.com> wrote:

> On Mon, Apr 25, 2022 at 05:40:01PM +0000, Jag Raman wrote:
> > > On Apr 25, 2022, at 6:27 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > 
> > > On Tue, Apr 19, 2022 at 04:44:20PM -0400, Jagannathan Raman wrote:  
> > >> +static MSIMessage vfu_object_msi_prepare_msg(PCIDevice *pci_dev,
> > >> +                                             unsigned int vector)
> > >> +{
> > >> +    MSIMessage msg;
> > >> +
> > >> +    msg.address = 0;
> > >> +    msg.data = vector;
> > >> +
> > >> +    return msg;
> > >> +}
> > >> +
> > >> +static void vfu_object_msi_trigger(PCIDevice *pci_dev, MSIMessage msg)
> > >> +{
> > >> +    vfu_ctx_t *vfu_ctx = pci_dev->irq_opaque;
> > >> +
> > >> +    vfu_irq_trigger(vfu_ctx, msg.data);
> > >> +}  
> > > 
> > > Why did you switch to vfu_object_msi_prepare_msg() +
> > > vfu_object_msi_trigger() in this revision?  
> > 
> > We previously did not do this switch because the server didn’t get updates
> > to the MSIx table & PBA.
> > 
> > The latest client version (which is not part of this series) forwards 
> > accesses
> > to the MSIx table & PBA over to the server. It also reads the PBA set by the
> > server. These change make it possible for the server to make this switch.  
> 
> Interesting. That's different from kernel VFIO. Before vfio-user commits
> to a new approach it would be worth checking with Alex that he agrees
> with the design.
> 
> I remember sending an email asking about why VFIO MSI-X PBA does not
> offer the full semantics described in the PCIe spec but didn't get a
> response from Alex (Message-Id:
> YkMWp0lUJAHhivJA@stefanha-x1.localdomain).

IIUC, the question is why we redirect the MSI-X interrupt from the KVM
irqfd to be handled in QEMU when the vector is masked.  This is largely
to work around the fact that we haven't had a means to implement mask
and unmask in the kernel, therefore we leave the vector enabled and
only enable the emulated PBA if a masked vector fires.  This works
because nobody really cares about the PBA, nor operates in a mode where
vectors are masked and the PBA is polled.  Drivers that understand the
device likely have better places to poll for service requests than the
PBA.

Ideally, masking a vector would make use of the existing mask and
unmask uAPI via the SET_IRQS ioctl, but we haven't been able to
implement this due to lack of internal kernel APIs to support it.  We
may have those interfaces now, but lacking bandwidth, I haven't checked
recently and we seem to be getting by ok as is.  Thanks,

Alex