Re: [PATCH v8] introduce vfio-user protocol specification

[John Johnson]

> On May 10, 2021, at 3:25 PM, John Levon <levon@movementarian.org> wrote:
> 
> On Mon, May 10, 2021 at 05:57:37PM +0100, Stefan Hajnoczi wrote:
> 
> 
>>> +VFIO_USER_VM_INTERRUPT
>>> +----------------------
>>> +
>>> +Message format
>>> +^^^^^^^^^^^^^^
>>> +
>>> ++----------------+------------------------+
>>> +| Name           | Value                  |
>>> ++================+========================+
>>> +| Message ID     | <ID>                   |
>>> ++----------------+------------------------+
>>> +| Command        | 13                     |
>>> ++----------------+------------------------+
>>> +| Message size   | 20                     |
>>> ++----------------+------------------------+
>>> +| Flags          | Reply bit set in reply |
>>> ++----------------+------------------------+
>>> +| Error          | 0/errno                |
>>> ++----------------+------------------------+
>>> +| Interrupt info | <interrupt>            |
>>> ++----------------+------------------------+
>>> +
>>> +This command message is sent from the server to the client to signal the 
>>> device
>>> +has raised an interrupt.
>> 
>> Except if the client set up irq eventfds?
> 
> Clarified.
> 
>>> +Interrupt info format
>>> +^^^^^^^^^^^^^^^^^^^^^
>>> +
>>> ++-----------+--------+------+
>>> +| Name      | Offset | Size |
>>> ++===========+========+======+
>>> +| Sub-index | 16     | 4    |
>>> ++-----------+--------+------+
>>> +
>>> +* *Sub-index* is relative to the IRQ index, e.g., the vector number used 
>>> in PCI
>>> +  MSI/X type interrupts.
>> 
>> Hmm...this is weird. The server tells the client to raise an MSI-X
>> interrupt but does not include the MSI message that resides in the MSI-X
>> table BAR device region? Or should MSI-X interrupts be delivered to the
>> client via VFIO_USER_DMA_WRITE instead?
>> 
>> (Basically it's not clear to me how MSI-X interrupts would work with
>> vfio-user. Reading how they work in kernel VFIO might let me infer it,
>> but it's probably worth explaining this clearly in the spec.)
> 
> It doesn't. We don't have an implementation, and the qemu patches don't get 
> this
> right either - it treats the sub-index as the IRQ index AKA IRQ type.
> 
> I'd be inclined to just remove this for now, until we have an implementation.
> Thoughts?
> 

        VFIO will set up 2 eventfds for each enabled MSI/X vector.  One is
terminated in KVM for direct injection into the guest.  The other terminates
back in QEMU, and triggers MSI/X SW emulation.  When informing the kernel of
which FDs to use, VFIO prefers the KVM FD, the QMEU one is only used if the
KVM one can’t be created (or is disabled by command line option)

        VFIO_USER_INTERRUPT would need an vector number in the request.  I
noticed this when I did the client, but delayed it because of what JohnL said
in another email: VFIO_USER_INTERRUPT is only be needed if the client and server
are in different VMs and can’t use eventfds.

        I’m fine with removing it for 1.0, since we don’t support cross-VM
emulation yet.

                                                                JJ