Re: [EXT] Re: [PATCH v9 1/9] hw/vfio/common: Remove error print on mmio region translation by viommu

[Auger Eric]

Hi Bharat,

On 5/5/20 12:18 PM, Bharat Bhushan wrote:
> Hi Eric,
> 
> On Tue, May 5, 2020 at 3:16 PM Bharat Bhushan <address@hidden> wrote:
>>
>> hi Eric,
>>
>> On Tue, May 5, 2020 at 3:00 PM Auger Eric <address@hidden> wrote:
>>>
>>> Hi Bharat,
>>>
>>> On 5/5/20 11:25 AM, Bharat Bhushan wrote:
>>>> Hi Eric,
>>>>
>>>> On Fri, Apr 24, 2020 at 7:47 PM Auger Eric <address@hidden> wrote:
>>>>>
>>>>> Hi Bharat,
>>>>>
>>>>> On 4/2/20 11:01 AM, Bharat Bhushan wrote:
>>>>>> Hi Eric/Alex,
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Alex Williamson <address@hidden>
>>>>>>> Sent: Thursday, March 26, 2020 11:23 PM
>>>>>>> To: Auger Eric <address@hidden>
>>>>>>> Cc: Bharat Bhushan <address@hidden>; address@hidden;
>>>>>>> address@hidden; address@hidden; address@hidden;
>>>>>>> address@hidden; Tomasz Nowicki [C] <address@hidden>;
>>>>>>> address@hidden; address@hidden; address@hidden; qemu-
>>>>>>> address@hidden; address@hidden; address@hidden;
>>>>>>> address@hidden; David Gibson <address@hidden>
>>>>>>> Subject: [EXT] Re: [PATCH v9 1/9] hw/vfio/common: Remove error print on 
>>>>>>> mmio
>>>>>>> region translation by viommu
>>>>>>>
>>>>>>> External Email
>>>>>>>
>>>>>>> ----------------------------------------------------------------------
>>>>>>> On Thu, 26 Mar 2020 18:35:48 +0100
>>>>>>> Auger Eric <address@hidden> wrote:
>>>>>>>
>>>>>>>> Hi Alex,
>>>>>>>>
>>>>>>>> On 3/24/20 12:08 AM, Alex Williamson wrote:
>>>>>>>>> [Cc +dwg who originated this warning]
>>>>>>>>>
>>>>>>>>> On Mon, 23 Mar 2020 14:16:09 +0530
>>>>>>>>> Bharat Bhushan <address@hidden> wrote:
>>>>>>>>>
>>>>>>>>>> On ARM, the MSI doorbell is translated by the virtual IOMMU.
>>>>>>>>>> As such address_space_translate() returns the MSI controller MMIO
>>>>>>>>>> region and we get an "iommu map to non memory area"
>>>>>>>>>> message. Let's remove this latter.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Eric Auger <address@hidden>
>>>>>>>>>> Signed-off-by: Bharat Bhushan <address@hidden>
>>>>>>>>>> ---
>>>>>>>>>>  hw/vfio/common.c | 2 --
>>>>>>>>>>  1 file changed, 2 deletions(-)
>>>>>>>>>>
>>>>>>>>>> diff --git a/hw/vfio/common.c b/hw/vfio/common.c index
>>>>>>>>>> 5ca11488d6..c586edf47a 100644
>>>>>>>>>> --- a/hw/vfio/common.c
>>>>>>>>>> +++ b/hw/vfio/common.c
>>>>>>>>>> @@ -426,8 +426,6 @@ static bool vfio_get_vaddr(IOMMUTLBEntry *iotlb,
>>>>>>> void **vaddr,
>>>>>>>>>>                                   &xlat, &len, writable,
>>>>>>>>>>                                   MEMTXATTRS_UNSPECIFIED);
>>>>>>>>>>      if (!memory_region_is_ram(mr)) {
>>>>>>>>>> -        error_report("iommu map to non memory area %"HWADDR_PRIx"",
>>>>>>>>>> -                     xlat);
>>>>>>>>>>          return false;
>>>>>>>>>>      }
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm a bit confused here, I think we need more justification beyond
>>>>>>>>> "we hit this warning and we don't want to because it's ok in this
>>>>>>>>> one special case, therefore remove it".  I assume the special case
>>>>>>>>> is that the device MSI address is managed via the SET_IRQS ioctl and
>>>>>>>>> therefore we won't actually get DMAs to this range.
>>>>>>>> Yes exactly. The guest creates a mapping between one giova and this
>>>>>>>> gpa (corresponding to the MSI controller doorbell) because MSIs are
>>>>>>>> mapped on ARM. But practically the physical device is programmed with
>>>>>>>> an host chosen iova that maps onto the physical MSI controller's
>>>>>>>> doorbell. so the device never performs DMA accesses to this range.
>>>>>>>>
>>>>>>>>   But I imagine the case that
>>>>>>>>> was in mind when adding this warning was general peer-to-peer
>>>>>>>>> between and assigned and emulated device.
>>>>>>>> yes makes sense.
>>>>>>>>
>>>>>>>>   Maybe there's an argument to be made
>>>>>>>>> that such a p2p mapping might also be used in a non-vIOMMU case.  We
>>>>>>>>> skip creating those mappings and drivers continue to work, maybe
>>>>>>>>> because nobody attempts to do p2p DMA with the types of devices we
>>>>>>>>> emulate, maybe because p2p DMA is not absolutely reliable on bare
>>>>>>>>> metal and drivers test it before using it.
>>>>>>>> MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c
>>>>>>>> iommu_dma_get_msi_page).
>>>>>>>> One idea could be to pass that flag through the IOMMU Notifier
>>>>>>>> mechanism into the iotlb->perm. Eventually when we get this in
>>>>>>>> vfio_get_vaddr() we would not print the warning. Could that make sense?
>>>>>>>
>>>>>>> Yeah, if we can identify a valid case that doesn't need a warning, 
>>>>>>> that's fine by me.
>>>>>>> Thanks,
>>>>>>
>>>>>> Let me know if I understood the proposal correctly:
>>>>>>
>>>>>> virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP) with 
>>>>>> VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping.
>>>>>> In qemu, virtio-iommu device will set a new defined flag (say 
>>>>>> IOMMU_MMIO) in iotlb->perm in memory_region_notify_iommu(). 
>>>>>> vfio_get_vaddr() will check same flag and will not print the warning.>
>>>>>> Is above correct?
>>>>> Yes that's what I had in mind.
>>>>
>>>> In that case virtio-iommu driver in guest should not make map
>>>> (VIRTIO_IOMMU_T_MAP) call as it known nothing to be mapped.
>>> sorry I don't catch what you meant. Please can you elaborate?
>>
>> What I understood of the proposal is:
>> Linux:
>>  1) MSI doorbells are mapped using the IOMMU_MMIO flag (dma-iommu.c
>> iommu_dma_get_msi_page)
>>  2) virtio-iommu driver in guest will make map (VIRTIO_IOMMU_T_MAP)
>> with VIRTIO_IOMMU_MAP_F_MMIO flag for MSI mapping.
>>
>> GEMU:
>> 3) virtio-iommu device - If VIRTIO_IOMMU_MAP_F_MMIO flag set then will
>> set a new defined flag (say IOMMU_MMIO) in iotlb->perm in
>> memory_region_notify_iommu()
>> 4. vfio_get_vaddr() will check same flag and will not print the
>> warning. Also vfio_iommu_map_notify() will not do anything.
>>
>> So, rather than going down to step 3 and 4, can we avoid maling map()
>> calling in step-2 itself?
> 
> ohh, We need to setup msi translation mapping, correct.

I sent
[PATCH 0/5] VIRTIO-IOMMU probe request support and MSI bypass on ARM

With this series, the guest does not attempt to map the ITS MSI doorbell
anymore. So you shouldn't get "iommu map to non memory area" anymore if
you rebase on it and remove this patch.

At the moment I only bypass the ITS doorbell but I think it is safe to
do the same with GICv2M doorbell as well. Anyway all assigned devices
will be able to access this latter.

Thanks

Eric
> 
> Thanks
> -Bharat
> 
>>
>> Thanks
>> -Bharat
>>
>>>
>>> Thanks
>>>
>>> Eric
>>>>
>>>> Stay Safe
>>>>
>>>> Thanks
>>>> -Bharat
>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Eric
>>>>>>
>>>>>> Thanks
>>>>>> -Bharat
>>>>>>
>>>>>>>
>>>>>>> Alex
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>