qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] vfio in the guest: no available reset mechanism


From: Alex Williamson
Subject: Re: [Qemu-devel] vfio in the guest: no available reset mechanism
Date: Fri, 01 Aug 2014 10:35:24 -0600

On Fri, 2014-08-01 at 09:25 -0600, Alex Williamson wrote:
> On Fri, 2014-08-01 at 09:35 +0800, Le Tan wrote:
> > Hi Alex,
> > 
> > 2014-07-30 22:46 GMT+08:00 Alex Williamson <address@hidden>:
> > > On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote:
> > >> Hi Michael,
> > >>
> > >> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <address@hidden>:
> > >> > On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
> > >> >> Hi,
> > >> >> I am testing vfio in L1 with my VT-d emulation project. I assigned one
> > >> >> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
> > >> >> in L1, it complains that:
> > >> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> > >> >> available reset mechanism.
> > >> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
> > >> >> available reset mechanism.
> > >> >>
> > >> >> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
> > >> >> look into this and found that:
> > >> >> val = ahci_ctrl_readl(ctrl, HOST_CTL);
> > >> >> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
> > >> >> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
> > >> >> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
> > >> >> should be cleared by default. So maybe L1 didn't reset the device
> > >> >> before assigning it to L2?
> > >> >> Then the BIOS tried to write back to HOST_CTL and it was stuck here. 
> > >> >> :(
> > >> >>
> > >> >> So can anyone give me some advice? About the state of PCI device or
> > >> >> bus-level reset?
> > >> >>
> > >> >> Here is the detail of the environment and the way I did the vfio.
> > >> >> 1. lspci in L1 said:
> > >> >> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> > >> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> > >> >> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
> > >> >> Interface Controller [8086:2918] (rev 02)
> > >> >> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
> > >> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
> > >> >> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
> > >> >> Controller [8086:2930] (rev 02)
> > >> >> 2. Unbind 00:03.0 and do vfio:
> > >> >> modprobe -r vfio_iommu_type1
> > >> >> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
> > >> >> modprobe vfio-pci
> > >> >> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
> > >> >> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
> > >> >> 3. run L2 with "-device vfio-pci,host=00:03.0"
> > >> >>
> > >> >> Any help is appreciated! Thanks very much!
> > >> >>
> > >> >> Regards,
> > >> >> Le
> > >> >
> > >> > Clearly, bus level reset can't work for the root bus :)
> > >>
> > >> Thanks very much!
> > >> I test the vfio with a second-bus ahci controller and it didn't
> > >> complain about the lack of reset mechanism. :) And the return val of
> > >> HOST_CTL is normal now (the same as emulated ahci controller).
> > >> However, it still paused when the BIOS tried to write to the HOST_CTL.
> > >> Do you have any idea?
> > >> And we should just test vfio and legacy pci-assignment with second bus
> > >> devices, not considering the root-bus devices?
> > >
> > > AHCI seems like a poor choice of device for this work, they typically
> > > don't support any kind of reset and they can be troublesome even for the
> > > L1 assignment.  You really want something with FLR support so that both
> > > the host and L1 guest can potentially reset the device.  That said, you
> > > may still run into bugs with a L1 guest directed FLR.  Thanks,
> > >
> > > Alex
> > >
> > 
> > So what device do you think is suitable for the pci-assign test? e1000?
> > I just tested it with sound card ac97 and USB controller. But I don't
> > know how to attach them to a pcie-to-pci bridge, so maybe they weren't
> > reset before being assigned to L2. But it seems that they can work.
> > 1. With the sound card, I assigned it to L2 via both vfio and legacy
> > pci-assign and I could hear the music played in L2 from host's
> > speakers. Of course, the vfio also complained about the lack of reset
> > mechanism.
> 
> First off, maybe I'm a little confused, are you trying to assigned
> emulated devices for the L1 guest to the L2 guest or are these assigned
> devices to the L1 guest that you want to re-assign to L2?  AFAIK, we
> don't have any emulated devices that support reset, but it wouldn't be
> that hard to add a PM capability to one that advertises a soft reset on
> D3hot->D0 transition and calls the QEMU driver reset function when that
> occurs.  This would provide the most flexibility.

Or even better might be to add an Advanced Features capability to each
device so it can do an explicit FLR.  We don't tend to trust PM reset as
being reliably on all devices.

> The other choice is to create a topology where an individual device can
> be reset via a bus reset, so putting a single device behind a bridge,
> root port, or downstream switch port.
> 
> Otherwise, the emulated e1000 is probably a reasonable choice, network
> drivers generally seem to be pretty good about accepting a device in a
> running state since it might have been used as a boot device.
> 
> If you're looking at re-assigning an L1 assigned device to an L2 guest,
> I'm not sure how that's supposed to work.  The VT-d emulation would need
> to somehow push L2 physical mappings to the host, but the L1 guest
> doesn't know the true isolation constraints of the host devices and
> whether they can actually be mapped separately from other devices.
> Since we try to use as few IOMMU domains as possible, you'd end up
> modifying translations for all devices assigned to the L1 guest.
> 
> > 2. With the USB controller, I used "-usb -usbdevice disk:file" to
> > attach a USB disk to L1. But there were 4 related devices in L1, so I
> > didn't know what should be assigned to L2. the info qtree was like
> > this:
> >  dev: ich9-usb-uhci3, id ""
> >         masterbus = "usb-bus.0"
> >         firstport = 4 (0x4)
> >         bandwidth = 1280 (0x500)
> >         maxframes = 128 (0x80)
> >         addr = 1d.2
> > dev: ich9-usb-uhci2, id ""
> >         masterbus = "usb-bus.0"
> >         firstport = 2 (0x2)
> >         bandwidth = 1280 (0x500)
> >         maxframes = 128 (0x80)
> >         addr = 1d.1
> >  dev: ich9-usb-uhci1, id ""
> >         masterbus = "usb-bus.0"
> >         firstport = 0 (0x0)
> >         bandwidth = 1280 (0x500)
> >         maxframes = 128 (0x80)
> >         addr = 1d.0
> > dev: ich9-usb-ehci1, id ""
> >         maxframes = 128 (0x80)
> >         addr = 1d.7
> >          bus: usb-bus.0
> >           type usb-bus
> >           dev: usb-storage, id ""
> >             drive = ""
> >             logical_block_size = 512 (0x200)
> >             physical_block_size = 512 (0x200)
> >             min_io_size = 0 (0x0)
> >             opt_io_size = 0 (0x0)
> >             bootindex = -1 (0xffffffffffffffff)
> >             discard_granularity = 4294967295 (0xffffffff)
> >             removable = false
> >             port = ""
> >             serial = ""
> >             full-path = true
> >             msos-desc = true
> >             addr 0.1, port 1, speed 480, name QEMU USB MSD, attached
> >             bus: scsi.0
> >               type SCSI
> >               dev: scsi-disk, id ""
> >                 drive = "usb0"
> >                 logical_block_size = 512 (0x200)
> >                 physical_block_size = 512 (0x200)
> >                 min_io_size = 0 (0x0)
> >                 opt_io_size = 0 (0x0)
> > Then I tried to unbind and bind both 1d.7 and 1d.0. Then I assigned
> > 1d.0 to L2. L2 could boot smoothly and I could mount and access the
> > USB disk.
> > So what do you think?
> 
> Sounds like a reasonable test of L1 emulated assigned to L2.  Thanks,
> 
> Alex
> 
> 
> 






reply via email to

[Prev in Thread] Current Thread [Next in Thread]