Re: [Qemu-devel] Question about the vfio device interrupt

[Alex Williamson]

On Wed, 15 May 2019 17:27:52 +0300
Maxim Levitsky <address@hidden> wrote:

> Hi!
> 
> Let me add my 0.2 cents to this discussion by sharing my summary of INTx 
> emulation in VFIO:
> 
> COMMON:
> * real interrupt is signaled in the kernel
> * vfio-pci kernel interrupt hanlder (vfio_intx_handler), masks the INTx flag
> * vfio-pci kernel interrupt handler triggers an eventfd, if the interrupt is 
> not virtually intx masked
>   (an eventfd which was registred for VFIO_IRQ_SET_DATA_EVENTFD | 
> VFIO_IRQ_SET_ACTION_TRIGGER)
> 
> 
> KERNEL INTERRUPT CONTROLLER CASE:
> 
> * the vfio-pci eventfd is listened by the kvm (set by KVM_IRQFD ioctl), and 
> delivered to guest ioapic
>   (the setup is done by qemu in vfio_intx_enable_kvm, the same eventfd is 
> given to vfio-pci and to kvm)
> * when guest writes eoi, an 'resamplefd' is (which is registered by 
> vfio_intx_enable_kvm too)
>   is triggered.
>   the vfio_intx_enable_kvm tells the vfio pci module 
> (VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_UNMASK)
>   to listen to that 'resamplefd' and unmask the interrupt
>   (here too, a same eventfd is given to kvm and to vfio-pci, but here the kvm 
> signals it, and vfio-pci listens to it).
> 
> 
> USERSPACE QEMU CASE:
> 
>   (only happens in short period while vfio_intx_enable is running and before 
> it calles the vfio_intx_enable_kvm)
>   (or when vfio_intx_enable_kvm fails)
> 
> * the vfio-pci eventfd is listened by the qemu (vfio_intx_interrupt)
> * the vfio_intx_interrupt (in qemu)
>       * masks the interrupt using kernel vfio interface
>       * unmaps all the device bars and sets a timer to map them again
>       * sends the interrupt down the qemu stack till it is reinjected to the 
> guest 
> * any bar access now goes to qemu via EPT fault 
> (vfio_region_read/vfio_region_write)
> * the bar access 
>       * triggers (vfio_eoi->vfio_intx_eoi) which unmasks (using vfio-pci) the 
> interrupt
>         the idea is that first few bar accesses should ack the interrupt and 
> thus avoid flood
>         but some interrupts could still happen (one per bar access)

Full points :)  Thanks,

Alex