Re: Using virtual IOMMU in guest hypervisors other than KVM and Xen?

[Peter Xu]

On Wed, Oct 16, 2019 at 03:01:22PM -0700, Jintack Lim wrote:
> On Mon, Oct 14, 2019 at 7:50 PM Peter Xu <address@hidden> wrote:
> >
> > On Mon, Oct 14, 2019 at 01:28:49PM -0700, Jintack Lim wrote:
> > > Hi,
> >
> > Hello, Jintack,
> >
> Hi Peter,
> 
> > >
> > > I'm trying to pass through a physical network device to a nested VM
> > > using virtual IOMMU. While I was able to do it successfully using KVM
> > > and Xen guest hypervisors running in a VM respectively, I couldn't do
> > > it with Hyper-V as I described below. I wonder if anyone have
> > > successfully used virtual IOMMU in other hypervisors other than KVM
> > > and Xen? (like Hyper-V or VMware)
> > >
> > > The issue I have with Hyper-V is that Hyper-V gives an error that the
> > > underlying hardware is not capable of doing passthrough. The exact
> > > error message is as follows.
> > >
> > > Windows Power-shell > (Get-VMHost).IovSupportReasons
> > > The chipset on the system does not do DMA remapping, without which
> > > SR-IOV cannot be supported.
> > >
> > > I'm pretty sure that Hyper-V recognizes virtual IOMMU, though; I have
> > > enabled iommu in windows boot loader[1], and I see differences when
> > > booing a Windows VM with and without virtual IOMMU. I also checked
> > > that virtual IOMMU traces are printed.
> >
> > What traces have you checked?  More explicitly, have you seen DMAR
> > enabled and page table setup for that specific device to be
> > pass-throughed?
> 
> Thanks for the pointers. I checked that DMAR is NOT enabled. The only
> registers that Windows guest accessed were Version Register,
> Capability Register, and Extended Capability Register. On the other
> hand, a Linux guest accessed other registers and enabled DMAR.
> Here's a link to the trace I got using QEMU 4.1.0. Do you see anything
> interesting there?
> http://paste.ubuntu.com/p/YcSyxG9Z3x/

Then I feel like Windows is reluctant to enable DMAR due to lacking of
some caps.

> 
> >
> > >
> > > I have tried multiple KVM/QEMU versions including the latest ones
> > > (kernel v5.3, QEMU 4.1.0) as well as two different Windows servers
> > > (2016 and 2019), but I see the same result. [4]
> > >
> > > I'd love to hear if somebody is using virtual IOMMU in Hyper-V or
> > > VMware successfully, especially for passthrough. I also appreciate if
> > > somebody can point out any configuration errors I have.
> > >
> > > Here's the qemu command line I use, basically from the QEMU vt-d
> > > page[2] and Hyper-v on KVM from kvmforum [3].
> > >
> > > ./qemu/x86_64-softmmu/qemu-system-x86_64 -device
> > > intel-iommu,intremap=on,caching-mode=on -smp 6 -m 24G -M
> >
> > Have you tried to use 4-level IOMMU page table (aw-bits=48 on latest
> > QEMU, or x-aw-bits=48 on some old ones)?  IIRC we've encountered
> > issues when trying to pass the SVVP Windows test with this, in which
> > 4-level is required.  I'm not sure whether whether that is required in
> > general usages of vIOMMU in Windows.
> 
> I just tried the option you mentioned, but it didn't change anything.
> BTW, what version of Windows was it?

Sorry I don't remember that. I didn't do the test but I was just
acknowledged that with it the test passed.  I assume you're using the
latest QEMU here because I know Windows could require another
capability (DMA draining) and it should be on by default in latest
qemu master.

At that time the complete cmdline to pass the test should be:

  -device intel-iommu,intremap=on,aw-bits=48,caching-mode=off,eim=on

I also don't remember on why caching-mode needs to be off at that
time (otherwise SVVP fails too).

-- 
Peter Xu