Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device

[Auger Eric]

Hi Bharat,

On 19/06/2017 09:54, Bharat Bhushan wrote:
> Hi Eric,
> 
> I started added replay in virtio-iommu and came across how MSI interrupts 
> with work with VFIO. 
> I understand that on intel this works differently but vsmmu will have same 
> requirement. 
> kvm-msi-irq-route are added using the msi-address to be translated by viommu 
> and not the final translated address.
> While currently the irqfd framework does not know about emulated iommus 
> (virtio-iommu, vsmmuv3/vintel-iommu).
> So in my view we have following options:
> - Programming with translated address when setting up kvm-msi-irq-route
> - Route the interrupts via QEMU, which is bad from performance
> - vhost-virtio-iommu may solve the problem in long term

Sorry for the delay. With regard to the vsmmuv3/vfio integration I think
we need to use the guest physical address otherwise the MSI address will
not be recognized as an MSI doorbell.

Also the fact on ARM we map the MSI doorbell causes an assert in
vfio_get_vaddr() as the vITS doorbell is not a RAM region. We will need
to handle this specifically.

Besides I have not looked specifically at the virtio-iommu/vfio
integration yet.

Thanks

Eric
> 
> Is there any other better option I am missing?
> 
> Thanks
> -Bharat
> 
>> -----Original Message-----
>> From: Auger Eric [mailto:address@hidden
>> Sent: Friday, June 09, 2017 5:24 PM
>> To: Bharat Bhushan <address@hidden>;
>> address@hidden; address@hidden;
>> address@hidden; address@hidden; address@hidden;
>> address@hidden; address@hidden
>> Cc: address@hidden; address@hidden; address@hidden;
>> address@hidden; address@hidden; address@hidden;
>> address@hidden; address@hidden
>> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device
>>
>> Hi Bharat,
>>
>> On 09/06/2017 13:30, Bharat Bhushan wrote:
>>> Hi Eric,
>>>
>>>> -----Original Message-----
>>>> From: Auger Eric [mailto:address@hidden
>>>> Sent: Friday, June 09, 2017 12:14 PM
>>>> To: Bharat Bhushan <address@hidden>;
>>>> address@hidden; address@hidden;
>>>> address@hidden; address@hidden; address@hidden;
>>>> address@hidden; address@hidden
>>>> Cc: address@hidden; address@hidden;
>> address@hidden;
>>>> address@hidden; address@hidden;
>>>> address@hidden; address@hidden; address@hidden
>>>> Subject: Re: [RFC v2 0/8] VIRTIO-IOMMU device
>>>>
>>>> Hi Bharat,
>>>>
>>>> On 09/06/2017 08:16, Bharat Bhushan wrote:
>>>>> Hi Eric,
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Eric Auger [mailto:address@hidden
>>>>>> Sent: Wednesday, June 07, 2017 9:31 PM
>>>>>> To: address@hidden; address@hidden;
>>>>>> address@hidden; address@hidden;
>>>> address@hidden;
>>>>>> address@hidden; address@hidden; jean-
>>>>>> address@hidden
>>>>>> Cc: address@hidden; address@hidden;
>>>> address@hidden;
>>>>>> address@hidden; address@hidden;
>>>>>> address@hidden; address@hidden; address@hidden; Bharat
>>>> Bhushan
>>>>>> <address@hidden>
>>>>>> Subject: [RFC v2 0/8] VIRTIO-IOMMU device
>>>>>>
>>>>>> This series implements the virtio-iommu device. This is a proof of
>>>>>> concept based on the virtio-iommu specification written by
>>>>>> Jean-Philippe
>>>> Brucker [1].
>>>>>> This was tested with a guest using the virtio-iommu driver [2] and
>>>>>> exposed with a virtio-net-pci using dma ops.
>>>>>>
>>>>>> The device gets instantiated using the "-device virtio-iommu-device"
>>>>>> option. It currently works with ARM virt machine only as the
>>>>>> machine must handle the dt binding between the virtio-mmio "iommu"
>>>>>> node and the PCI host bridge node. ACPI booting is not yet supported.
>>>>>>
>>>>>> This should allow to start some benchmarking activities against
>>>>>> pure emulated IOMMU (especially ARM SMMU).
>>>>>
>>>>> I am testing this on ARM64 and see below continuous error prints:
>>>>>
>>>>>   virtio_iommu_translate sid=8 is not known!!
>>>>>   virtio_iommu_translate sid=8 is not known!!
>>>>>   virtio_iommu_translate sid=8 is not known!!
>>>>>   virtio_iommu_translate sid=8 is not known!!
>>>>>   virtio_iommu_translate sid=8 is not known!!
>>>>>   virtio_iommu_translate sid=8 is not known!!
>>>>>   virtio_iommu_translate sid=8 is not known!!
>>>>>   virtio_iommu_translate sid=8 is not known!!
>>>>>   virtio_iommu_translate sid=8 is not known!!
>>>>>   virtio_iommu_translate sid=8 is not known!!
>>>>>
>>>>>
>>>>> Also in guest I do not see device-tree node with virtio-iommu.
>>>> do you mean the virtio-mmio with #iommu-cells property?
>>>>
>>>> This one is created statically by virt machine. I would be surprised
>>>> if it were not there. Are you using the virt = virt2.10 machine.
>>>> Machines before do not support its instantiation.
>>>>
>>>> Please can you add a printf in hw/arm/virt.c create_virtio_mmio() at
>>>> the moment when this node is created. Also you can add a printf in
>>>> bind_virtio_iommu_device() to make sure the binding with the PCI host
>>>> bridge is added on machine init done.
>>>>
>>>> Also worth to check, CONFIG_VIRTIO_IOMMU=y on guest side.
>>>
>>> It works on my side.
>> Great.
>>
>>  The driver config was disabled and also I was using guest kernel which was
>> not have deferred-probing.
>> Yes I did not mention in my cover letter the guest I have been using is based
>> on Jean-Philippe's branch, featuring deferred IOMMU probing. I I have not
>> tried yet with an upstream guest.
>>  Now after fixing it works on my side
>>> I placed some prints to see dma-map are mapping regions in virtio-iommu,
>> it uses emulated iommu.
>>>
>>> I will continue to add VFIO support now on this and more testing !!
>>
>> OK. I will do the VFIO integration first on the vsmmuv3 device as I already
>> prepared the VFIO replay and hopefully we will sync ;-)
>>
>> Thanks
>>
>> Eric
>>>
>>> Thanks
>>> -Bharat
>>>
>>>>
>>>> Thanks
>>>>
>>>> Eric
>>>>
>>>>> I am using qemu-tree you mentioned below and iommu-driver patches
>>>> published by Jean-P.
>>>>> Qemu command line have additional ""-device virtio-iommu-device".
>>>>> What
>>>> I am missing ?
>>>>
>>>>
>>>>>
>>>>> Thanks
>>>>> -Bharat
>>>>>
>>>>>>
>>>>>> Best Regards
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>> This series can be found at:
>>>>>> https://github.com/eauger/qemu/tree/virtio-iommu-rfcv2
>>>>>>
>>>>>> References:
>>>>>> [1] [RFC 0/3] virtio-iommu: a paravirtualized IOMMU, [2] [RFC PATCH
>>>>>> linux]
>>>>>> iommu: Add virtio-iommu driver [3] [RFC PATCH kvmtool 00/15] Add
>>>>>> virtio- iommu
>>>>>>
>>>>>> History:
>>>>>> v1 -> v2:
>>>>>> - fix redifinition of viommu_as typedef
>>>>>>
>>>>>> Eric Auger (8):
>>>>>>   update-linux-headers: import virtio_iommu.h
>>>>>>   linux-headers: Update for virtio-iommu
>>>>>>   virtio_iommu: add skeleton
>>>>>>   virtio-iommu: Decode the command payload
>>>>>>   virtio_iommu: Add the iommu regions
>>>>>>   virtio-iommu: Implement the translation and commands
>>>>>>   hw/arm/virt: Add 2.10 machine type
>>>>>>   hw/arm/virt: Add virtio-iommu the virt board
>>>>>>
>>>>>>  hw/arm/virt.c                                 | 116 ++++-
>>>>>>  hw/virtio/Makefile.objs                       |   1 +
>>>>>>  hw/virtio/trace-events                        |  14 +
>>>>>>  hw/virtio/virtio-iommu.c                      | 623
>>>> ++++++++++++++++++++++++++
>>>>>>  include/hw/arm/virt.h                         |   5 +
>>>>>>  include/hw/virtio/virtio-iommu.h              |  60 +++
>>>>>>  include/standard-headers/linux/virtio_ids.h   |   1 +
>>>>>>  include/standard-headers/linux/virtio_iommu.h | 142 ++++++
>>>>>>  linux-headers/linux/virtio_iommu.h            |   1 +
>>>>>>  scripts/update-linux-headers.sh               |   3 +
>>>>>>  10 files changed, 957 insertions(+), 9 deletions(-)  create mode
>>>>>> 100644 hw/virtio/virtio-iommu.c  create mode 100644
>>>>>> include/hw/virtio/virtio- iommu.h  create mode 100644
>>>>>> include/standard- headers/linux/virtio_iommu.h  create mode 100644
>>>>>> linux-headers/linux/virtio_iommu.h
>>>>>>
>>>>>> --
>>>>>> 2.5.5
>>>>>
>>>
>