Re: Questions about the VFIO BAR region

[Auger Eric]

Hi Li,

On 11/5/19 2:16 AM, Li Qiang wrote:
> 
> 
> Alex Williamson <address@hidden
> <mailto:address@hidden>> 于2019年11月5日周二 上午2:49写道：
> 
>     On Tue, 5 Nov 2019 00:40:39 +0800
>     Li Qiang <address@hidden <mailto:address@hidden>> wrote:
> 
>     > Hello Alex, Auger and all,
>     >
>     > I have a question about the VFIO virtual device BAR.
>     >
>     > In vfio_region_setup, it initialize a ‘region->mem’ MR and set its
>     ops to ‘vfio_regions_ops’.
>     > In ‘vfio_region_mmap’, it maps the physical device’s MMIO to
>     QEMU’s virtual address space
>     > as a raw MR ‘region->mmaps[i].mem’.
>     > And also it set the latter MR as a subregion of the first one.
>     >
>     > So when the guest accesses the BAR, it will direct go to the
>     physical device’s BAR.
>     > My question is here:
>     > When the qemu will use the ‘vfio_regions_ops’ to read/write the BAR?
>     > Also whey in the last of ‘vfio_region_write/read’ we need to call
>     ‘vbasedev->ops->vfio_eoi(vbasedev);’?
> 
>     We support:
> 
>      a) sparse mmaps where the entire BAR is not covered by an mmap
> 
> 
> Got.
> 
>  
> 
>      b) quirks, which layer on top of the mmaps to provide virtualized
>         access
> 
> 
> Do you mean like in 'vfio_probe_ati_bar4_quirk', register a high
> priority subregion of VFIORegion.mem.
> So when the guest write the BAR, vfio_regions_ops will be used. Here
> 'quirks' do you mean such things?
> 
> static void vfio_probe_ati_bar4_quirk(VFIOPCIDevice *vdev, int nr)
> {
>     VFIOQuirk *quirk;
>     VFIOConfigWindowQuirk *window;
> 
>     ...
>     memory_region_init_io(window->addr_mem, OBJECT(vdev),
>                           &vfio_generic_window_address_quirk, window,
>                           "vfio-ati-bar4-window-address-quirk", 4);
>     memory_region_add_subregion_overlap(vdev->bars[nr].region.mem,
>                                         window->address_offset,
>                                         window->addr_mem, 1);
>    ...
> }
Yes that's it. In that case vfio_generic_window_address_quirk ops get
called when attempting to access this overlapping region.
> 
>  
> 
>      c) INTx emulation which disables mmaps MRs in order to detect device
>         access as a generic mechanism for inferring interrupt
>         acknowledgment.
> 
> 
> In the above two cases, in 'vfio_region_write/read' we always access the
> physical device's BAR.
> So as far as I can understand, the physical device(sometimes) will
> trigger interrupts. And the responsible of clear it 
> will be by the 'guest'. So I can't understand why there calls
> 'vbasedev->ops->vfio_eoi'. Could you please give me an
> example.
When a physical level sensitive IRQ hits it is signaled through an
eventfd. The eventfd can be handled by QEMU. In this case,
vfio_intx_interrupt gets called, for PCI. It turns the mmap off (slow
path on) and injects the corresponding virtual IRQ into the guest. The
reason why we turn the mmap off is we need to trap the guest end of
interrupt to eventually deactivate the IRQ at physical level and unmask
it (it was auto-masked by the vfio driver). The first access into the
region is assumed to correspond to the servicing of the pending
interrupt by the guest handler (pending status clear) and at this point
we deactivate the physical IRQ. So .vfio_eoi gets called on the first
read/write BAR access after an INTx gets pending. Note the mmapping
(fast path) is not immediatly turned on after deactivating the physical
INTx. See comment before vfio_intx_mmap_enable.

When an irqfd/resamplefd is used we do not need that trick as the guest
EOI is trapped at KVM level through the virtual interrupt controller.
when the guest EOI is trapped KVM deactivates the associated physical
IRQ and notifies the userspace through the resamplefd for this latter to
unmask the physical IRQ.

Hope this helps.

Thanks

Eric




> 
> 
> Thanks,
> Li Qiang
> 
>  
> 
> 
>     The latter being the reason we call vfio_eoi.  Thanks,
> 
>     Alex
>