Re: [Qemu-devel] [PATCH 0/3] VFIO-based PCI device assignment for QEMU 1.2

[Alex Williamson]

On Mon, 2012-08-13 at 18:36 +0200, Andreas Hartmann wrote:
> Alex Williamson schrieb:
> > On Mon, 2012-08-13 at 17:48 +0200, Andreas Hartmann wrote:
> >> Alex Williamson wrote:
> >>> On Mon, 2012-08-13 at 08:27 -0500, Anthony Liguori wrote:
> >>>> Alex Williamson <address@hidden> writes:
> >>>>
> >>>>> VFIO kernel support was just merged into Linux, so I'd like to
> >>>>> formally propose inclusion of the QEMU vfio-pci driver for
> >>>>> QEMU 1.2.  Included here is support for x86 PCI device assignment.
> >>>>> PCI INTx is not yet enabled, but devices making use of either MSI
> >>>>> or MSI-X work.  The level irqfd and eoifd support I've proposed
> >>>>> for KVM enable an accelerated patch for this through KVM.  I'd
> >>>>> like to get this base driver in first and enable the remaining
> >>>>> support in-tree.
> >>>>>
> >>>>> I've split this version up a little from the RFC to make it a bit
> >>>>> easier to review.  Review comments from Blue Swirl and Avi are
> >>>>> already incorporated, including Avi's requests to simplify both
> >>>>> the PCI BAR mapping and unmapping paths.
> >>>>
> >>>> Hi Alex,
> >>>>
> >>>> Thanks for pushing this forward!  Hopefully this will finally kill
> >>>> off qemu-kvm.git for good.
> >>>>
> >>>> I think this series is going to have to wait for 1.3 to open up.  We
> >>>> have a very short release window for this release and I'd feel a lot
> >>>> more comfortable having such a significant feature spend some time in
> >>>> the development cycle getting testing/review.
> >>>>
> >>>> I'd like to see a few Reviewed-by's too for this series before it
> >>>> goes in.  I expect they won't be hard to get but I also expect it
> >>>> will take a few more revisions of this series to get there.
> >>>
> >>> That's disappointing, but I can understand your reluctance.  Blue
> >>> Swirl reviewed the RFC and could perhaps add a Reviewed-by.  Alexey
> >>> has been working on the POWER port and I'm sure could provide a
> >>> Reviewed-by.  We also have a few early adopters that are already
> >>> making use of this code.
> >>
> >> I'm running qemu with vfio patch since Jun 05, 2012
> >> (awilliam-qemu-vfio-v0.14.0-rc0-6402-g323cf9f.tar.gz). I didn't
> >> encounter any problem so far.
> >>
> >> If you like, I could compile a more actual version, too (if there have
> >> been any changes).
> > 
> > The only change in the version proposed for qemu is that legacy
> > interrupt support has been removed until we can agree on interfaces in
> > kvm and plumb an EOI path through qemu.  IIRC, the devices you're using
> > require legacy interrupt support.
> > 
> >> To see more about my use case:
> >> http://permalink.gmane.org/gmane.linux.drivers.rt2x00.user/1051
> >>
> >> You may add a Tested-by Andreas Hartmann <address@hidden>
> >> if you like.
> >>
> >> Unfortunately, I'm only running the vfio VM (kvm) with this version of
> >> qemu, but I'm running parallel 4 other VM's with the unchanged version
> >> of qemu (kvm-0.15.0-123.2.x86_64), too.
> >> One of these 4 VM's uses PCIe passthrough.
> >>
> >> I now tried to run all VMs with the new version of qemu. At this point,
> >> I unfortunately run into a problem with the VM which passes through a
> >> PCIe device. The error message is (during start of VM):
> >>
> >> virsh start VM
> >> error: Failed to start domain VM
> >> error: internal error process exited while connecting to monitor: 
> >> qemu-system-x86_64: -device 
> >> pci-assign,host=04:00.0,id=hostdev0,bus=pci.0,addr=0x6: Parameter 'driver' 
> >> expects device type
> >>
> >> The xml file for libvirt looks like this: 
> > 
> > libvirt doesn't yet support vfio-pci and current qemu.git doesn't yet
> > support pci-assign.  To use libvirt for device assignment, your only
> > option right now is to use a qemu-kvm.git based version of qemu.
> 
> If I'm using your qemu instead of qemu from kvm-0.15 (opensuse package),
> this error comes up when passing through a PCIe device, which works
> absolutely fine with kvm 0.15. I would have expected, that your qemu
> works with the legacy way of handling pcie passthrough, too (with
> pci-stub module).

VFIO cannot work with pci-stub, the backends are fundamentally
different.  KVM making use pci-stub to hold onto a device is actually
one of the design problems that VFIO is meant to correct.  The other
significant interface change is use of IOMMU groups, which is actually
why VFIO works for some of your uses while pci-assign does not.

> This would mean, that all users get errors if they use the traditional
> way. IOW: there are changes needed (which?) to move from kvm 0.15 to
> your qemu version.

But the way we solve this is to make libvirt understand how to do both.
Then it can probe the qemu/kvm binary and host system to figure out
which is supported and use the correct device options based on what it
finds.  Trying to do both via the same qemu command line doesn't make
sense to me, especially when the device setup is so different.  Thanks,

Alex