Re: [Qemu-discuss] Error starting domain: vfio-pci

[Bandan Das]

Gabriel Laupre <address@hidden> writes:

>>To rule out if it's a problem with your guest xml. Let me know if you are
> interested and I will write down the steps.
>
> I am way more than interested to try that :) Thanks a lot

Ok, Based on your command line below, I am assuming that you have a
multifunction card. You might need to bind all functions to the
stub driver if they are in the same iommu group or else vfio will complain.

virsh nodedev-list | grep 04_10

The output would be something like
pci_0000_04_10_0
pci_0000_04_10_1 and so on.

For each, assuming they are the same card, you have to bind them to the
stub driver:
virsh nodedev-detach pci_0000_04_10_0
virsh nodedev-detach pci_0000_04_10_1 ...

Shut down all guests and make sure no qemu processes are running
and then run qemu:
/usr/libexec/qemu-kvm -device 
vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3 -monitor stdio

I think this should be enough to spit out the error, there's no need to actually
boot a guest image.

> 2015-05-22 10:46 GMT-07:00 Bandan Das <address@hidden>:
>
>> Gabriel Laupre <address@hidden> writes:
>>
>> >> Thanks for the update. Did you get a chance to run qemu standalone
>> > without libvirt?
>> > How can I do that, what are the purpose to do so and why to try that?
>> > (thank you ^^)
>> To rule out if it's a problem with your guest xml. Let me know if you are
>> interested and I will write down the steps.
>>
>> > 2015-05-21 13:32 GMT-07:00 Bandan Das <address@hidden>:
>> >
>> >> Gabriel Laupre <address@hidden> writes:
>> >>
>> >> >> AppArmor is Debian based, but that doesn't stop you from
>> >> > running/installing it on other distros
>> >> >
>> >> >> $ dmesg | grep apparmor
>> >> >
>> >> > The service isn't started nor present in my machine. Thank you for the
>> >> hint
>> >> > though.
>> >> > I am still stuck with this error and it is not dependent of my Intel
>> >> NIC, I
>> >> > used another one and I still got the same error message.
>> >> > ps: tks to remind me to send to the mailing list:)
>> >>
>> >> Thanks for the update. Did you get a chance to run qemu standalone
>> without
>> >> libvirt ?
>> >>
>> >> > 2015-05-21 12:32 GMT-07:00 Karl Apsite <address@hidden>:
>> >> >
>> >> >> AppArmor is Debian based, but that doesn't stop you from
>> >> >> running/installing it
>> >> >> on other distros
>> >> >>
>> >> >> $ dmesg | grep apparmor
>> >> >>
>> >> >> On 05/20/2015 02:56 PM, Bandan Das wrote:
>> >> >> > Gabriel Laupre <address@hidden> writes:
>> >> >> >
>> >> >> >> Mmmhh,
>> >> >> >> My SELinux is disabled. Is Apparmor not only on debian/ubuntu and
>> >> suse,
>> >> >> am
>> >> >> >> I wrong?  I have no idea on that :)
>> >> >> >
>> >> >> > Yeah, me neither :) I am just trying to rule out all possibilities.
>> >> >> >
>> >> >> >> 2015-05-20 11:23 GMT-07:00 Bandan Das <address@hidden>:
>> >> >> >>
>> >> >> >>> Oh and one more thing! You already answered before but just
>> wanted
>> >> to
>> >> >> >>> confirm
>> >> >> >>> that you don't have apparmor running, right ?
>> >> >> >>>
>> >> >> >>> Bandan Das <address@hidden> writes:
>> >> >> >>>
>> >> >> >>>> Gabriel Laupre <address@hidden> writes:
>> >> >> >>>>
>> >> >> >>>>>> Yes, indeed it is. What distro is this ? Do you have SELinux
>> or
>> >> any
>> >> >> >>> other
>> >> >> >>>>> security feature enabled ? Can you please verify that the file
>> >> has a
>> >> >> >>>>> appropriate label if SELinux is enabled ? (ls -lZ
>> /dev/vfio/vfio)
>> >> >> >>>>> My distrib:
>> >> >> >>>>> address@hidden ~]# cat /proc/version
>> >> >> >>>>> Linux version 3.10.0-229.1.2.el7.x86_64 (
>> >> >> >>> address@hidden)
>> >> >> >>>>> (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP
>> Fri
>> >> >> Mar 27
>> >> >> >>>>> 03:04:26 UTC 2015
>> >> >> >>>>> address@hidden ~]# cat /etc/centos-release
>> >> >> >>>>> CentOS Linux release 7.1.1503 (Core)
>> >> >> >>>>>
>> >> >> >>>>> address@hidden ~]# ls -lZ /dev/vfio/vfio
>> >> >> >>>>> crw-rw-rw- root root ?
>> >> /dev/vfio/vfio
>> >> >> >>>>>
>> >> >> >>>>> SELinux is disabled:
>> >> >> >>>>> address@hidden ~]# getenforce
>> >> >> >>>>> Disabled
>> >> >> >>>>>
>> >> >> >>>>> I guess no other security feature is enabled that I am aware
>> of. I
>> >> >> once
>> >> >> >>> had
>> >> >> >>>>> a message saying that it can be one of the following issues
>> >> (listing
>> >> >> the
>> >> >> >>>>> 5). So I guess it can be any combination of those issues, even
>> >> >> something
>> >> >> >>>>> completely different.
>> >> >> >>>>
>> >> >> >>>> Ugh, I am out of options! Can you please try a few more things:
>> Can
>> >> >> you
>> >> >> >>> try
>> >> >> >>>> running qemu directly and see if you see the same behavior ? If
>> you
>> >> >> still
>> >> >> >>>> haven't tried running as root, please try that too. Also, please
>> >> check
>> >> >> >>> dmesg
>> >> >> >>>> for any vfio related errors.
>> >> >> >>>>
>> >> >> >>>>> libvirtError: internal error: process exited while connecting
>> to
>> >> >> >>>>> monitor: 2015-05-19T21:46:21.935043Z qemu-kvm: -device
>> >> >> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> failed to
>> >> >> >>>>> open /dev/vfio/vfio: Operation not permitted
>> >> >> >>>> Well, this is the first error from vfio_connect_container()
>> when it
>> >> >> does:
>> >> >> >>>>     fd = qemu_open("/dev/vfio/vfio", O_RDWR);
>> >> >> >>>>     if (fd < 0) {
>> >> >> >>>>         error_report("vfio: failed to open /dev/vfio/vfio: %m");
>> >> >> >>>>         ret = -errno;
>> >> >> >>>> ...
>> >> >> >>>>
>> >> >> >>>> The rest are followup errors printed from the other functions in
>> >> the
>> >> >> >>>> stack due to this error.
>> >> >> >>>>
>> >> >> >>>> Bandan
>> >> >> >>>>
>> >> >> >>>>> 2015-05-19T21:46:21.935091Z qemu-kvm: -device
>> >> >> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> failed to
>> >> >> >>>>> setup container for group 24
>> >> >> >>>>> 2015-05-19T21:46:21.935107Z qemu-kvm: -device
>> >> >> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> failed to
>> >> >> >>>>> get group 24
>> >> >> >>>>> 2015-05-19T21:46:21.935135Z qemu-kvm: -device
>> >> >> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> >> >> >>>>> initialization failed.
>> >> >> >>>>> 2015-05-19T21:46:21.935157Z qemu-kvm: -device
>> >> >> >>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> >> >> >>>>> 'vfio-pci' could not be initialized
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> 2015-05-19 21:17 GMT-08:00 Bandan Das <address@hidden>:
>> >> >> >>>>>
>> >> >> >>>>>>
>> >> >> >>>>>>> On May 20, 2015, at 12:29 AM, Gabriel Laupre <
>> address@hidden
>> >> >
>> >> >> >>> wrote:
>> >> >> >>>>>>>
>> >> >> >>>>>>> Thank Bandan,
>> >> >> >>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>>> Assuming you are on an intel box, have you booted your
>> kernel
>> >> with
>> >> >> >>>>>> intel_iommu=on ?
>> >> >> >>>>>>> Yes, I have booted my kernel with the intel_iommu=on. (I
>> don't
>> >> >> >>> remember
>> >> >> >>>>>> how to check that now though ^^)
>> >> >> >>>>>>>
>> >> >> >>>>>>>> Please paste the output of dmesg | grep -e DMAR -e IOMMU ?
>> >> >> >>>>>>> address@hidden ~]# dmesg | grep -e DMAR -e IOMMU
>> >> >> >>>>>>> [    0.000000] ACPI: DMAR 00000000bf79e0c0 00118 (v01    AMI
>> >> >> OEMDMAR
>> >> >> >>>>>> 00000001 MSFT 00000097)
>> >> >> >>>>>>> [    0.000000] Intel-IOMMU: enabled
>> >> >> >>>>>>> [    0.039149] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0
>> cap
>> >> >> >>>>>> c90780106f0462 ecap f020f6
>> >> >> >>>>>>> [    0.550126] IOMMU 0 0xfbffe000: using Queued invalidation
>> >> >> >>>>>>> [    0.550131] IOMMU: Setting RMRR:
>> >> >> >>>>>>> [    0.550149] IOMMU: Setting identity map for device
>> >> 0000:00:1a.0
>> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >>>>>>> [    0.550184] IOMMU: Setting identity map for device
>> >> 0000:00:1a.1
>> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >>>>>>> [    0.550211] IOMMU: Setting identity map for device
>> >> 0000:00:1a.2
>> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >>>>>>> [    0.550241] IOMMU: Setting identity map for device
>> >> 0000:00:1a.7
>> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >>>>>>> [    0.550272] IOMMU: Setting identity map for device
>> >> 0000:00:1d.0
>> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >>>>>>> [    0.550302] IOMMU: Setting identity map for device
>> >> 0000:00:1d.1
>> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >>>>>>> [    0.550329] IOMMU: Setting identity map for device
>> >> 0000:00:1d.2
>> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >>>>>>> [    0.550358] IOMMU: Setting identity map for device
>> >> 0000:00:1d.7
>> >> >> >>>>>> [0xbf7ec000 - 0xbf7fffff]
>> >> >> >>>>>>> [    0.550375] IOMMU: Setting identity map for device
>> >> 0000:00:1a.0
>> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >>>>>>> [    0.550387] IOMMU: Setting identity map for device
>> >> 0000:00:1a.1
>> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >>>>>>> [    0.550399] IOMMU: Setting identity map for device
>> >> 0000:00:1a.2
>> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >>>>>>> [    0.550410] IOMMU: Setting identity map for device
>> >> 0000:00:1a.7
>> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >>>>>>> [    0.550421] IOMMU: Setting identity map for device
>> >> 0000:00:1d.0
>> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >>>>>>> [    0.550433] IOMMU: Setting identity map for device
>> >> 0000:00:1d.1
>> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >>>>>>> [    0.550444] IOMMU: Setting identity map for device
>> >> 0000:00:1d.2
>> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >>>>>>> [    0.550458] IOMMU: Setting identity map for device
>> >> 0000:00:1d.7
>> >> >> >>>>>> [0xec000 - 0xeffff]
>> >> >> >>>>>>> [    0.550471] IOMMU: Prepare 0-16MiB unity mapping for LPC
>> >> >> >>>>>>> [    0.550483] IOMMU: Setting identity map for device
>> >> 0000:00:1f.0
>> >> >> >>> [0x0
>> >> >> >>>>>> - 0xffffff]
>> >> >> >>>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>> Yeah, this looks ok. Actually, taking a second look, I can’t
>> >> think
>> >> >> of
>> >> >> >>>>>> anyway how this could be related to file permissions on
>> >> >> /dev/vfio/vfio.
>> >> >> >>>>>>
>> >> >> >>>>>>>> Why does opening /dev/vfio/vfio fail ? Can you please
>> confirm
>> >> that
>> >> >> >>> you
>> >> >> >>>>>> have read/write permissions as the user you are trying to run
>> ?
>> >> >> >>>>>>> address@hidden ~]# cd /dev/vfio/
>> >> >> >>>>>>> address@hidden vfio]# ls -la | grep vfio
>> >> >> >>>>>>> crw-rw-rw-  1 root root 10, 196 May 18 11:54 vfio
>> >> >> >>>>>>> The right should be okay I guess.
>> >> >> >>>>>>>
>> >> >> >>>>>> Yes, indeed it is. What distro is this ? Do you have SELinux
>> or
>> >> any
>> >> >> >>> other
>> >> >> >>>>>> security feature enabled ? Can you please verify that the file
>> >> has a
>> >> >> >>>>>> appropriate label if SELinux is enabled ? (ls -lZ
>> /dev/vfio/vfio)
>> >> >> >>>>>>
>> >> >> >>>>>> Bandan
>> >> >> >>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>> 2015-05-19 18:54 GMT-08:00 Bandan Das <address@hidden>:
>> >> >> >>>>>>>
>> >> >> >>>>>>> Hello Gabriel,
>> >> >> >>>>>>>
>> >> >> >>>>>>>> On May 19, 2015, at 8:03 PM, Gabriel Laupre <
>> address@hidden
>> >> >
>> >> >> >>> wrote:
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> Hello everyone,
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> I am using a Centos 7.1 machine with the kernel 3.10.229. I
>> >> want
>> >> >> to
>> >> >> >>>>>> use my host with SR-IOV to use a virtual function on my NIC as
>> >> the
>> >> >> >>> vNIC in
>> >> >> >>>>>> my new VM.
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> I have an instance started with a old NIC using macvtap
>> that I
>> >> >> >>> want to
>> >> >> >>>>>> change. I am using the
>> >> >> >>>>>>>> virsh edit instance-00000034
>> >> >> >>>>>>>> command to edit the XML configuration to add the new device
>> I
>> >> want
>> >> >> >>> to
>> >> >> >>>>>> attach.
>> >> >> >>>>>>> …
>> >> >> >>>>>>> Assuming you are on an intel box, have you booted your kernel
>> >> with
>> >> >> >>>>>> intel_iommu=on ?
>> >> >> >>>>>>> Please paste the output of dmesg | grep -e DMAR -e IOMMU ?
>> >> >> >>>>>>>
>> >> >> >>>>>>>> When I try to reboot the VM I get this error:
>> >> >> >>>>>>>> Error starting domain: internal error: process exited while
>> >> >> >>> connecting
>> >> >> >>>>>> to monitor: 2015-05-19T21:46:21.935043Z qemu-kvm: -device
>> >> >> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> failed
>> >> >> to
>> >> >> >>> open
>> >> >> >>>>>> /dev/vfio/vfio: Operation not permitted
>> >> >> >>>>>>> Why does opening /dev/vfio/vfio fail ? Can you please confirm
>> >> that
>> >> >> >>> you
>> >> >> >>>>>> have read/write permissions as the user you are trying to run
>> ?
>> >> >> >>>>>>>
>> >> >> >>>>>>>> 2015-05-19T21:46:21.935091Z qemu-kvm: -device
>> >> >> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> failed
>> >> >> to
>> >> >> >>> setup
>> >> >> >>>>>> container for group 24
>> >> >> >>>>>>>> 2015-05-19T21:46:21.935107Z qemu-kvm: -device
>> >> >> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: vfio:
>> >> failed
>> >> >> to
>> >> >> >>> get
>> >> >> >>>>>> group 24
>> >> >> >>>>>>>> 2015-05-19T21:46:21.935135Z qemu-kvm: -device
>> >> >> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> >> >> >>> initialization
>> >> >> >>>>>> failed.
>> >> >> >>>>>>>> 2015-05-19T21:46:21.935157Z qemu-kvm: -device
>> >> >> >>>>>> vfio-pci,host=04:10.4,id=hostdev0,bus=pci.0,addr=0x3: Device
>> >> >> 'vfio-pci'
>> >> >> >>>>>> could not be initialized
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> total Trace here: http://sprunge.us/XZFB
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> Any idea how to fix that?
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> Thank you very much :)
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> Gabriel
>> >> >> >>>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>>
>> >> >> >>>
>> >> >> >
>> >> >>
>> >>
>>