Re: [Qemu-devel] [PATCH 00/13] IOMMU: Enable interrupt remapping for Intel IOMMU

[Jan Kiszka]

On 2016-02-19 08:43, Peter Xu wrote:
> Hi, Jan,
> 
> On Fri, Feb 19, 2016 at 07:46:26AM +0100, Jan Kiszka wrote:
>> Hi Peter,
>>
>> On 2016-02-19 04:30, Peter Xu wrote:
>>> This patchset provide very basic functionalities for interrupt
>>> remapping (IR) support of the emulated Intel IOMMU device.
>>
>> Interesting. Some questions:
>>
>> - Were you aware of
>> http://git.kiszka.org/?p=qemu.git;a=shortlog;h=refs/heads/queues/vtd-intremap,
>> and can you comment on how this relates to your approach? Mine included
>> HPET support, e.g.
> 
> No. I do not know we have existing works related. I'd say that I am
> one of those who are not fan of duplicating works. If I know that
> there is existing ones, I would have asked you first.
> 
> Actually there are several people within my working team knows that
> I would be working on this, and I believe none of us do know your
> work too... Or there must be someone telling me so...

I guess we would have to match sets of people know to find out who
forgot to mention the outreachy project ;) - anyway, this can always happen.

> 
> Whatever, I am sorry that finally we got some duplicated work. :-(
> 
> I have not go deeper into your codes yet, I'd say that most of the
> logic is alike, since lots of works are to follow the spec. And the
> implementation is slightly different (actually I just got this
> memory region idea from Paolo yesterday... instead of a worse one to
> declare a specific func ops for MSIs).

I didn't look into your approach in all details yet, and Rita also just
started, she told me. So one question on yours - which looks appealing
from the invasiveness POV - is how you determine the MSI source with
only a single target region? I do find your changes on the IOAPIC, but
none on the PCI infrastructure, which is confusing given that you say
that works, no?

My design introduces a per-source MSI (DMA) target region so that the
IOMMU can do proper remapping by deriving the source device ID from the
targeted region. Then you only need to give the special sources like
IOAPIC and HPET their only regions, and you are done.

> 
> What I was going to do is to at least support:
> 
> - vhost
> - pass through devices

vhost, i.e. virtio, has still problems as current virtio guest drivers
assume that their devices are not under control of any IOMMU unit. I
didn't trace the latest news here, but we were once discussing to
initially exclude virtio devices in DMAR scope descriptions in ACPI
and/or introduce some feature negotiation between guest and host to
identify and reject legacy guest drivers without IOMMU-awareness.

> 
> So what I was planning is that, this series will be the start. And
> the above two is the first-step goal (and I may need kvm-ioapic as
> well though).

KVM support is actually a key thing, that's why we started the project
on integrating the patches with the split irqchip work. There was a
consensus with Paolo long ago that full in-kernel irqchip will no longer
gain any additional support that is needed for IR.

> 
> With the above, we should be able to run DPDK applications in guests
> with either pass-through or vhost-user devices (these should be the
> two scenario which is most possibly to be used AFAIU). This is what
> I was trying to do. HPET is not with high priority in my todo list.
> 
>>
>> - Rita Sinha is currently working on integrating my old patches with the
>> split-irqchip to get KVM working (as an Outreachy project). It's
>> probably a bit unfortunate to consider a different horse that late in to
>> project. What do you think, how could we benefit from each other?
> 
> I'd say that I am still new to QEMU community. So this is the first
> time I encountered this kind of problem... Do you have any
> suggestion?
> 
>>
>> - Radim was telling me to look on this as well. How do your efforts
>> correlate?
> 
> Same as above. Do you have any suggestion on how we'd move on?

Specifically you and Rita should exchange your knowledge on both designs
(here on the list, of course) and try to find an optimal path. We are
unfortunately a bit time-constrained /wrt Outreachy as that project ends
in a couple of weeks. By then, we should have first results for the KVM
integration - not necessarily the perfect ones, but something working
and following the "right" direction.

> 
> One question about your tree: have you posted patches before? and
> what's the relationship between your tree and current QEMU master?

I never posted the patches as they never left the state of "hacks" as
you can quickly spot. However, I think the design is not a hack, but
maybe there is indeed something even smarter.

Rita has rebased the patches recently, maybe you can share your state,
Rita? I did this before, because I'm using them since day #1 to do
regular testing of our Jailhouse hypervisor inside QEMU/KVM. For that
purpose they work very reliably for IOAPIC, HPET and emulated PCI devices.

> 
>>
>>>
>>> By default, IR is disabled to be better compatible with current
>>> QEMU. To enable IR, we can using the following command to boot a
>>> IR-supported VM with basic network (still do not support kvm-ioapic,
>>> so we need to specify kernel_irqchip=off here):
>>>
>>> $ qemu-system-x86_64 -M q35,iommu=on,int_remap=on,kernel_irqchip=off \
>>>      -enable-kvm -m 1024 -s \
>>>      -monitor telnet::3333,server,nowait \
>>>      -netdev user,id=user.0,hostfwd=tcp::5555-:22 \
>>>      -device virtio-net-pci,netdev=user.0 \
>>>      /var/lib/libvirt/images/vm1.qcow2
>>>
>>> When guest boots, we can verify whether IR enabled by grepping the
>>> dmesg like:
>>>
>>> address@hidden ~]# journalctl -k | grep "DMAR-IR"
>>> Feb 19 11:21:23 localhost.localdomain kernel: DMAR-IR: IOAPIC id 0 under 
>>> DRHD base  0xfed90000 IOMMU 0
>>> Feb 19 11:21:23 localhost.localdomain kernel: DMAR-IR: Enabled IRQ 
>>> remapping in xapic mode
>>>
>>> Currently only two devices are supported:
>>>
>>> - Emulated IOAPIC device
>>> - PCI Devices
>>>
>>> TODO List:
>>>
>>> - kvm-ioapic support
>>> - vhost support
>>> - pass through device support
>>> - EIM support
>>> - IR fault reporting
>>> - source-id validation for IRTE
>>> - IRTE cache and better queued invalidation
>>> - migration support (for IOMMU as general?)
>>> - more?
>>>
>>> Peter Xu (13):
>>>   q35: add "int-remap" flag to enable intr
>>>   acpi: enable INTR for DMAR report structure
>>>   intel_iommu: allow queued invalidation for IR
>>>   intel_iommu: set IR bit for ECAP register
>>>   acpi: add DMAR scope definition for root IOAPIC
>>>   intel_iommu: define interrupt remap table addr register
>>>   intel_iommu: handle interrupt remap enable
>>>   intel_iommu: define several structs for IOMMU IR
>>>   intel_iommu: provide helper function vtd_get_iommu
>>>   ioapic-common: add iommu for IOAPICCommonState
>>>   intel_iommu: add IR translation faults defines
>>>   intel_iommu: ioapic: IR support for emulated IOAPIC
>>>   intel_iommu: Add support for PCI MSI remap
>>>
>>>  hw/core/machine.c                 |  20 ++
>>>  hw/i386/acpi-build.c              |  41 +++-
>>>  hw/i386/intel_iommu.c             | 397 
>>> +++++++++++++++++++++++++++++++++++++-
>>>  hw/i386/intel_iommu_internal.h    |  25 +++
>>>  hw/intc/ioapic.c                  |  36 +++-
>>>  hw/intc/ioapic_common.c           |   2 +
>>>  hw/pci-host/q35.c                 |   6 +-
>>>  include/hw/acpi/acpi-defs.h       |  15 ++
>>>  include/hw/boards.h               |   1 +
>>>  include/hw/i386/intel_iommu.h     | 121 ++++++++++++
>>>  include/hw/i386/ioapic_internal.h |   3 +
>>>  include/hw/pci/msi.h              |   4 +
>>>  12 files changed, 651 insertions(+), 20 deletions(-)
>>>
>>
>> Is there a git tree with your patches somewhere?
> 
> Currently not. If needed, I can try to create one. 

Would be helpful for more convenient import and experiments. E.g. I
could quickly through my test setup at them and tell you if they work
for it.

Jan

signature.asc
Description: OpenPGP digital signature