[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] Multi GPU passthrough via VFIO
|
From: |
Maik Broemme |
|
Subject: |
Re: [Qemu-devel] Multi GPU passthrough via VFIO |
|
Date: |
Thu, 6 Feb 2014 00:47:23 +0100 |
|
User-agent: |
Mutt/1.5.21 (2010-09-15) |
Hi Alex,
Alex Williamson <address@hidden> wrote:
> On Wed, 2014-02-05 at 22:10 +0100, Maik Broemme wrote:
> > Hi Alex,
> >
> > Alex Williamson <address@hidden> wrote:
> > > On Wed, 2014-02-05 at 19:59 +0100, Maik Broemme wrote:
> > > > Hi,
> > > >
> > > > currently VFIO with multi GPU passthrough is working partially and
> > > > hopefully somebody has a hint about the problem. I'm doing passthrough
> > > > of an AMD Radeon R9 290X and AMD Radeon 7870 GHz Edition to a single VM.
> > > >
> > > > If the VM is running Linux this works quite well with radeon or fglrx
> > > > driver. Please see 'dmesg' log attached, when using the radeon driver.
> > > > If needed I can also post one with fglrx driver.
> > > >
> > > > If I do the exact same passthrough to a Windows VM and use latest AMD
> > > > Catalyst 14.1 (2/1/2014) or AMD Catalyst 13.12 (12/18/2013) I can get
> > > > only the first device working (AMD R9 290X) with 'x-vga=on'. I don't
> > > > enable 'x-vga=on' on second device as this should never work. :)
> > >
> > > Why not? The guest is able to change the VGA enable bit in the emulated
> > > bridge registers and access VGA space of each device, just like happens
> > > on bare metal. You'll only get one device initialized from seabios, but
> > > that's the same as would happen on bare metal as well.
> > >
> >
> > Well it was just my guess as it would behave like most physical boxes
> > in this case. :)
> >
> > > > I see
> > > > BIOS boot screen and everything works fine except for the second GPU.
> > > > The windows device manager just show me "Code 12" for the second GPU
> > > > and its HD Audio device. Code 12 means: "This device cannot find enough
> > > > free resources that it can use".
> > >
> > > I've seen the same using Nvidia GRID GPUs (w/o x-vga=on), but only with
> > > the Q35 chipset model, Linux works, Windows reports Code 12. I have no
> > > idea why as all the PCI resources appear to be properly sized and
> > > mapped. FWIW, 2 GRID GPUs assigned to a guest do work with the 440FX
> > > chipset model. Beyond 2 we run out of MMIO resources below 4G and
> > > something bad happens.
> > >
> >
> > Interesting. I will try 440FX a bit later and see if this works. What I
> > can also do is to post system resource conflicts from Windows, the AMD
> > Catalyst Center has it integrated. Maybe this will help?
>
> If you actually see conflicts, then yes. The Code 12 I've seen I was
> never able to identify a conflict. The trouble with 440FX is that
> you'll need to use pci-bridges to isolate VGA space of each GPU.
> Otherwise one card would need to be disabled to ensure the VGA accesses
> go to the other.
>
Okay I've collected all necessary information (hopefully). Some are in
German but if needed I can translate it. Please find it below:
- Conflicts:
E/A-Port 0x000003C0-0x000003DF AMD Radeon R9 200 Series
E/A-Port 0x000003C0-0x000003DF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root
Port 0 - 3420
IRQ 10 AMD Radeon HD 7800 Series
IRQ 10 Intel(R) ICH9 Family SMBus Controller - 2930
Speicheradresse 0xFE800000-0xFE83FFFF AMD Radeon R9 200 Series
Speicheradresse 0xFE800000-0xFE83FFFF Intel(R) 5520/5500/X58 I/O Hub PCI
Express Root Port 0 - 3420
Speicheradresse 0xE0000000-0xEFFFFFFF AMD Radeon HD 7800 Series
Speicheradresse 0xE0000000-0xEFFFFFFF Intel(R) 5520/5500/X58 I/O Hub PCI
Express Root Port 0 - 3420
Speicheradresse 0xA0000-0xBFFFF AMD Radeon R9 200 Series
Speicheradresse 0xA0000-0xBFFFF PCI-Bus
Speicheradresse 0xA0000-0xBFFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root
Port 0 - 3420
Speicheradresse 0xC0000000-0xCFFFFFFF AMD Radeon R9 200 Series
Speicheradresse 0xC0000000-0xCFFFFFFF PCI-Bus
Speicheradresse 0xC0000000-0xCFFFFFFF Intel(R) 5520/5500/X58 I/O Hub PCI
Express Root Port 0 - 3420
E/A-Port 0x000003B0-0x000003BB AMD Radeon R9 200 Series
E/A-Port 0x000003B0-0x000003BB Intel(R) 5520/5500/X58 I/O Hub PCI Express Root
Port 0 - 3420
Speicheradresse 0xFE600000-0xFE63FFFF AMD Radeon HD 7800 Series
Speicheradresse 0xFE600000-0xFE63FFFF Intel(R) 5520/5500/X58 I/O Hub PCI
Express Root Port 0 - 3420
E/A-Port 0x0000C000-0x0000C0FF AMD Radeon HD 7800 Series
E/A-Port 0x0000C000-0x0000C0FF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root
Port 0 - 3420
E/A-Port 0x0000D000-0x0000D0FF AMD Radeon R9 200 Series
E/A-Port 0x0000D000-0x0000D0FF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root
Port 0 - 3420
- Display devices:
Name AMD Radeon R9 200 Series
PNP-Gerätekennung
PCI\VEN_1002&DEV_67B0&SUBSYS_0B001002&REV_00\4&2122111D&0&0018
Adaptertyp AMD Radeon Graphics Processor (0x67B0), Advanced Micro
Devices, Inc.-kompatibel
Adapterbeschreibung AMD Radeon R9 200 Series
Adapter-RAM (1.048.576) Bytes
Installierte Treiber
aticfx64.dll,aticfx64.dll,aticfx64.dll,aticfx32,aticfx32,aticfx32,atiumd64.dll,atidxx64.dll,atidxx64.dll,atiumdag,atidxx32,atidxx32,atiumdva,atiumd6a.cap,atitmm64.dll
Treiberversion 13.350.1005.0
INF-Datei oem7.inf (Abschnitt ati2mtag_Hawaii)
Farbebenen Nicht verfügbar
Farbtabelleneinträge 4294967296
Auflösung 1920 x 1080 x 60 Hz
Bits/Pixel 32
Speicheradresse 0xC0000000-0xCFFFFFFF
Speicheradresse 0xD0000000-0xD07FFFFF
E/A-Port 0x0000D000-0x0000D0FF
Speicheradresse 0xFE800000-0xFE83FFFF
IRQ-Kanal IRQ 4294967287
E/A-Port 0x000003B0-0x000003BB
E/A-Port 0x000003C0-0x000003DF
Speicheradresse 0xA0000-0xBFFFF
Treiber c:\windows\system32\drivers\atikmpag.sys (8.14.1.6367,
622,00 KB (636.928 Bytes), 31.01.2014 20:28)
Name AMD Radeon HD 7800 Series
PNP-Gerätekennung
PCI\VEN_1002&DEV_6818&SUBSYS_32511682&REV_00\4&49049C7&0&0820
Adaptertyp Nicht verfügbar, Advanced Micro Devices, Inc.-kompatibel
Adapterbeschreibung AMD Radeon HD 7800 Series
Adapter-RAM Nicht verfügbar
Installierte Treiber
aticfx64.dll,aticfx64.dll,aticfx64.dll,aticfx32,aticfx32,aticfx32,atiumd64.dll,atidxx64.dll,atidxx64.dll,atiumdag,atidxx32,atidxx32,atiumdva,atiumd6a.cap,atitmm64.dll
Treiberversion 13.350.1005.0
INF-Datei oem7.inf (Abschnitt ati2mtag_R575B)
Farbebenen Nicht verfügbar
Farbtabelleneinträge Nicht verfügbar
Auflösung Nicht verfügbar
Bits/Pixel Nicht verfügbar
Speicheradresse 0xE0000000-0xEFFFFFFF
Speicheradresse 0xFE600000-0xFE63FFFF
E/A-Port 0x0000C000-0x0000C0FF
IRQ-Kanal IRQ 10
Treiber c:\windows\system32\drivers\atikmpag.sys (8.14.1.6367,
622,00 KB (636.928 Bytes), 31.01.2014 20:28)
- I/O:
0x00000000-0x00000CD7 PCI-Bus OK
0x00000060-0x00000060 Standardtastatur (PS/2) OK
0x00000064-0x00000064 Standardtastatur (PS/2) OK
0x00000070-0x00000071 System CMOS/Echtzeituhr OK
0x00000072-0x00000077 System CMOS/Echtzeituhr OK
0x000002F8-0x000002FF Kommunikationsanschluss (COM2) OK
0x00000378-0x0000037F Druckeranschluss (LPT1) OK
0x000003B0-0x000003BB AMD Radeon R9 200 Series OK
0x000003B0-0x000003BB Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0
- 3420 OK
0x000003C0-0x000003DF AMD Radeon R9 200 Series OK
0x000003C0-0x000003DF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0
- 3420 OK
0x000003F2-0x000003F5 Standard-Diskettenlaufwerkcontroller OK
0x000003F7-0x000003F7 Standard-Diskettenlaufwerkcontroller OK
0x000003F8-0x000003FF Kommunikationsanschluss (COM1) OK
0x00000CD8-0x00000CF7 ACPI-Modulgerät OK
0x00000D00-0x0000FFFF PCI-Bus OK
0x0000B100-0x0000B13F Intel(R) ICH9 Family SMBus Controller - 2930 OK
0x0000C000-0x0000C0FF AMD Radeon HD 7800 Series OK
0x0000C000-0x0000C0FF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0
- 3420 OK
0x0000D000-0x0000D0FF AMD Radeon R9 200 Series OK
0x0000D000-0x0000D0FF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0
- 3420 OK
0x0000E000-0x0000E03F Red Hat VirtIO SCSI controller OK
0x0000E080-0x0000E09F Red Hat VirtIO Ethernet Adapter OK
0x0000E0A0-0x0000E0BF Standard AHCI 1.0 Serieller-ATA-Controller OK
- Memory:
0xC0000000-0xCFFFFFFF AMD Radeon R9 200 Series OK
0xC0000000-0xCFFFFFFF PCI-Bus OK
0xC0000000-0xCFFFFFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0
- 3420 OK
0xD0000000-0xD07FFFFF AMD Radeon R9 200 Series OK
0xFE800000-0xFE83FFFF AMD Radeon R9 200 Series OK
0xFE800000-0xFE83FFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0
- 3420 OK
0xFEA40000-0xFEA40FFF Red Hat VirtIO Ethernet Adapter OK
0xFED00000-0xFED003FF Hochpräzisionsereigniszeitgeber OK
0xFEA41000-0xFEA41FFF Red Hat VirtIO SCSI controller OK
0xE0000000-0xEFFFFFFF AMD Radeon HD 7800 Series OK
0xE0000000-0xEFFFFFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0
- 3420 OK
0xFE600000-0xFE63FFFF AMD Radeon HD 7800 Series OK
0xFE600000-0xFE63FFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0
- 3420 OK
0xFEA42000-0xFEA42FFF Standard AHCI 1.0 Serieller-ATA-Controller OK
0xFE860000-0xFE863FFF High Definition Audio-Controller OK
0xA0000-0xBFFFF AMD Radeon R9 200 Series OK
0xA0000-0xBFFFF PCI-Bus OK
0xA0000-0xBFFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420
OK
- IRQ:
IRQ 1 Standardtastatur (PS/2) OK
IRQ 3 Kommunikationsanschluss (COM2) OK
IRQ 4 Kommunikationsanschluss (COM1) OK
IRQ 6 Standard-Diskettenlaufwerkcontroller OK
IRQ 8 System CMOS/Echtzeituhr OK
IRQ 10 AMD Radeon HD 7800 Series OK
IRQ 10 Intel(R) ICH9 Family SMBus Controller - 2930 OK
IRQ 12 PS/2-kompatible Maus OK
IRQ 16 Standard AHCI 1.0 Serieller-ATA-Controller OK
IRQ 20 High Definition Audio-Controller OK
IRQ 81 Microsoft ACPI-konformes System OK
IRQ 82 Microsoft ACPI-konformes System OK
IRQ 83 Microsoft ACPI-konformes System OK
IRQ 84 Microsoft ACPI-konformes System OK
IRQ 85 Microsoft ACPI-konformes System OK
IRQ 86 Microsoft ACPI-konformes System OK
IRQ 87 Microsoft ACPI-konformes System OK
IRQ 88 Microsoft ACPI-konformes System OK
IRQ 89 Microsoft ACPI-konformes System OK
IRQ 90 Microsoft ACPI-konformes System OK
IRQ 91 Microsoft ACPI-konformes System OK
IRQ 92 Microsoft ACPI-konformes System OK
IRQ 93 Microsoft ACPI-konformes System OK
IRQ 94 Microsoft ACPI-konformes System OK
IRQ 95 Microsoft ACPI-konformes System OK
IRQ 96 Microsoft ACPI-konformes System OK
IRQ 97 Microsoft ACPI-konformes System OK
IRQ 98 Microsoft ACPI-konformes System OK
IRQ 99 Microsoft ACPI-konformes System OK
IRQ 100 Microsoft ACPI-konformes System OK
IRQ 101 Microsoft ACPI-konformes System OK
IRQ 102 Microsoft ACPI-konformes System OK
IRQ 103 Microsoft ACPI-konformes System OK
IRQ 104 Microsoft ACPI-konformes System OK
IRQ 105 Microsoft ACPI-konformes System OK
IRQ 106 Microsoft ACPI-konformes System OK
IRQ 107 Microsoft ACPI-konformes System OK
IRQ 108 Microsoft ACPI-konformes System OK
IRQ 109 Microsoft ACPI-konformes System OK
IRQ 110 Microsoft ACPI-konformes System OK
IRQ 111 Microsoft ACPI-konformes System OK
IRQ 112 Microsoft ACPI-konformes System OK
IRQ 113 Microsoft ACPI-konformes System OK
IRQ 114 Microsoft ACPI-konformes System OK
IRQ 115 Microsoft ACPI-konformes System OK
IRQ 116 Microsoft ACPI-konformes System OK
IRQ 117 Microsoft ACPI-konformes System OK
IRQ 118 Microsoft ACPI-konformes System OK
IRQ 119 Microsoft ACPI-konformes System OK
IRQ 120 Microsoft ACPI-konformes System OK
IRQ 121 Microsoft ACPI-konformes System OK
IRQ 122 Microsoft ACPI-konformes System OK
IRQ 123 Microsoft ACPI-konformes System OK
IRQ 124 Microsoft ACPI-konformes System OK
IRQ 125 Microsoft ACPI-konformes System OK
IRQ 126 Microsoft ACPI-konformes System OK
IRQ 127 Microsoft ACPI-konformes System OK
IRQ 128 Microsoft ACPI-konformes System OK
IRQ 129 Microsoft ACPI-konformes System OK
IRQ 130 Microsoft ACPI-konformes System OK
IRQ 131 Microsoft ACPI-konformes System OK
IRQ 132 Microsoft ACPI-konformes System OK
IRQ 133 Microsoft ACPI-konformes System OK
IRQ 134 Microsoft ACPI-konformes System OK
IRQ 135 Microsoft ACPI-konformes System OK
IRQ 136 Microsoft ACPI-konformes System OK
IRQ 137 Microsoft ACPI-konformes System OK
IRQ 138 Microsoft ACPI-konformes System OK
IRQ 139 Microsoft ACPI-konformes System OK
IRQ 140 Microsoft ACPI-konformes System OK
IRQ 141 Microsoft ACPI-konformes System OK
IRQ 142 Microsoft ACPI-konformes System OK
IRQ 143 Microsoft ACPI-konformes System OK
IRQ 144 Microsoft ACPI-konformes System OK
IRQ 145 Microsoft ACPI-konformes System OK
IRQ 146 Microsoft ACPI-konformes System OK
IRQ 147 Microsoft ACPI-konformes System OK
IRQ 148 Microsoft ACPI-konformes System OK
IRQ 149 Microsoft ACPI-konformes System OK
IRQ 150 Microsoft ACPI-konformes System OK
IRQ 151 Microsoft ACPI-konformes System OK
IRQ 152 Microsoft ACPI-konformes System OK
IRQ 153 Microsoft ACPI-konformes System OK
IRQ 154 Microsoft ACPI-konformes System OK
IRQ 155 Microsoft ACPI-konformes System OK
IRQ 156 Microsoft ACPI-konformes System OK
IRQ 157 Microsoft ACPI-konformes System OK
IRQ 158 Microsoft ACPI-konformes System OK
IRQ 159 Microsoft ACPI-konformes System OK
IRQ 160 Microsoft ACPI-konformes System OK
IRQ 161 Microsoft ACPI-konformes System OK
IRQ 162 Microsoft ACPI-konformes System OK
IRQ 163 Microsoft ACPI-konformes System OK
IRQ 164 Microsoft ACPI-konformes System OK
IRQ 165 Microsoft ACPI-konformes System OK
IRQ 166 Microsoft ACPI-konformes System OK
IRQ 167 Microsoft ACPI-konformes System OK
IRQ 168 Microsoft ACPI-konformes System OK
IRQ 169 Microsoft ACPI-konformes System OK
IRQ 170 Microsoft ACPI-konformes System OK
IRQ 171 Microsoft ACPI-konformes System OK
IRQ 172 Microsoft ACPI-konformes System OK
IRQ 173 Microsoft ACPI-konformes System OK
IRQ 174 Microsoft ACPI-konformes System OK
IRQ 175 Microsoft ACPI-konformes System OK
IRQ 176 Microsoft ACPI-konformes System OK
IRQ 177 Microsoft ACPI-konformes System OK
IRQ 178 Microsoft ACPI-konformes System OK
IRQ 179 Microsoft ACPI-konformes System OK
IRQ 180 Microsoft ACPI-konformes System OK
IRQ 181 Microsoft ACPI-konformes System OK
IRQ 182 Microsoft ACPI-konformes System OK
IRQ 183 Microsoft ACPI-konformes System OK
IRQ 184 Microsoft ACPI-konformes System OK
IRQ 185 Microsoft ACPI-konformes System OK
IRQ 186 Microsoft ACPI-konformes System OK
IRQ 187 Microsoft ACPI-konformes System OK
IRQ 188 Microsoft ACPI-konformes System OK
IRQ 189 Microsoft ACPI-konformes System OK
IRQ 190 Microsoft ACPI-konformes System OK
IRQ 4294967287 AMD Radeon R9 200 Series OK
IRQ 4294967288 Red Hat VirtIO Ethernet Adapter OK
IRQ 4294967289 Red Hat VirtIO Ethernet Adapter OK
IRQ 4294967290 Red Hat VirtIO Ethernet Adapter OK
IRQ 4294967291 Red Hat VirtIO SCSI controller OK
IRQ 4294967292 Red Hat VirtIO SCSI controller OK
IRQ 4294967293 Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420
OK
IRQ 4294967294 Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420
OK
I'm no expert but it looks like Windows never enabled MSI for the second
card as qemu shell with 'info pci' show me both with IRQ 10. I'll hope it
helps.
> > > > QEMU is called in both cases via the following. I just replace the
> > > > '-drive' accordingly.
> > > >
> > > > /usr/bin/taskset -c 0,1,2,3 /usr/bin/qemu-system-x86_64 \
> > > > -machine q35,accel=kvm \
> > > > -enable-kvm \
> > > > -nodefaults \
> > > > -nographic \
> > > > -vga none \
> > > > -boot order=nc \
> > > > -cpu host \
> > > > -smp cores=4,threads=1,sockets=1 \
> > > > -m 8192 \
> > > > -rtc base=localtime \
> > > > -k de \
> > > > -drive
> > > > file=/srv/kvm/linux-drive0.img,id=drive0,if=none,cache=none,aio=threads
> > > > \
> > > > -mon chardev=monitor0 \
> > > > -chardev socket,id=monitor0,path=/tmp/linux.monitor,nowait,server \
> > > > -netdev tap,id=net0,vhost=on,helper=/usr/lib/qemu/qemu-bridge-helper \
> > > > -device virtio-net-pci,netdev=net0,mac=00:00:00:02:01:04 \
> > > > -device virtio-blk-pci,drive=drive0,ioeventfd=on \
> > > > -device ioh3420,bus=pcie.0,id=pcie0,port=1,chassis=1,multifunction=on
> > > > \
> > > > -device ioh3420,bus=pcie.0,id=pcie1,port=2,chassis=2,multifunction=on
> > > > \
> > > > -device
> > > > vfio-pci,host=01:00.0,addr=00.0,bus=pcie0,multifunction=on,x-vga=on \
> > > > -device vfio-pci,host=01:00.1,addr=00.1,bus=pcie0 \
> > > > -device vfio-pci,host=02:00.0,addr=00.0,bus=pcie1,multifunction=on \
> > > > -device vfio-pci,host=02:00.1,addr=00.1,bus=pcie1 \
> > > > -no-reboot
> > > >
> > > > My setup is the following:
> > > >
> > > > Kernel: linux-3.13.1
> > > > Seabios: seabios-git-rel.1.7.4.r51.g151d034 (5/2/2014)
> > > > QEMU: qemu-git-2.0.r30666.g31db5b3 (5/2/2014)
> > > >
> > > > Below is the 'lspci' output and I'm using the AMD Radeon HD 5430 as
> > > > device
> > > > for my local X server:
> > > >
> > > > 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI
> > > > to PCI bridge (external gfx0 port B) (rev 02)
> > > > 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory
> > > > Management Unit (IOMMU)
> > > > 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to
> > > > PCI bridge (PCI express gpp port B)
> > > > 00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to
> > > > PCI bridge (PCI express gpp port D)
> > > > 00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to
> > > > PCI bridge (PCI express gpp port H)
> > > > 00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to
> > > > PCI bridge (external gfx1 port B)
> > > > 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > > > SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
> > > > 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > > > SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
> > > > 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > > > SB7x0/SB8x0/SB9x0 USB EHCI Controller
> > > > 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > > > SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
> > > > 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > > > SB7x0/SB8x0/SB9x0 USB EHCI Controller
> > > > 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus
> > > > Controller (rev 42)
> > > > 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00
> > > > Azalia (Intel HDA) (rev 40)
> > > > 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI]
> > > > SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
> > > > 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to
> > > > PCI Bridge (rev 40)
> > > > 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > > > SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
> > > > 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI]
> > > > SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
> > > > 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI]
> > > > SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1)
> > > > 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to
> > > > PCI bridge (PCIE port 2)
> > > > 00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to
> > > > PCI bridge (PCIE port 3)
> > > > 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > > > SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
> > > > 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > > > SB7x0/SB8x0/SB9x0 USB EHCI Controller
> > > > 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
> > > > Processor Function 0
> > > > 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
> > > > Processor Function 1
> > > > 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
> > > > Processor Function 2
> > > > 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
> > > > Processor Function 3
> > > > 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
> > > > Processor Function 4
> > > > 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
> > > > Processor Function 5
> > > > 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
> > > > [AMD/ATI] Hawaii XT [Radeon HD 8970]
> > > > 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8
> > > > 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
> > > > [AMD/ATI] Pitcairn XT [Radeon HD 7870 GHz Edition]
> > > > 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape
> > > > Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> > > > 03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host
> > > > Controller (rev 01)
> > > > 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
> > > > [AMD/ATI] Park [Mobility Radeon HD 5430]
> > > > 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI
> > > > Audio [Radeon HD 5400/6300 Series]
> > > > 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> > > > RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
> > > > 07:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host
> > > > Controller (rev 01)
> > > >
> > > > Another minor issue is that the R9 290X is not reset during shutdown of
> > > > VM (neither Linux nor Windows) but it can be tricked with doing
> > > > "suspend-to-ram" between two starts. That's why I use '-no-reboot'
> > > > option
> > > > in QEMU. The 7870 is doing the reset properly.
> > >
> > >
> > > Is the NoSoftRst "-" on the 290X vs "+" on the 7870 in lspci -vvv by
> > > chance? Thanks,
> > >
> >
> > Here are both. It is funny it is opposite as you described. :)
>
>
> Oops, yes. Does this help?
>
> --- a/hw/misc/vfio.c
> +++ b/hw/misc/vfio.c
> @@ -3136,7 +3136,7 @@ static void vfio_pci_reset_handler(void *opaque)
>
> QLIST_FOREACH(group, &group_list, next) {
> QLIST_FOREACH(vdev, &group->device_list, next) {
> - if (!vdev->reset_works || (!vdev->has_flr &&
> vdev->has_pm_reset)) {
> + if (!vdev->reset_works || !vdev->has_flr) {
> vdev->needs_reset = true;
> }
> }
>
> I can't figure out why I coded it the way that I did. Probably overly
> targeting a specific device. Thanks,
>
This patch works absolutely fine. After applying it to my 'qemu-git', the
device resets works flawlessly. So it would be great to push it upstream
as it looks good.
> Alex
>
> > address@hidden:~# lspci -vvv -s 01:00.0 | grep NoSoftRst
> > Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> >
> > address@hidden:~# lspci -vvv -s 02:00.0 | grep NoSoftRst
> > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> >
> > address@hidden:~# lspci -vvv -s 01:00.0
> > 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > Hawaii XT [Radeon HD 8970] (prog-if 00 [VGA controller])
> > Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0b00
> > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> > Stepping- SERR+ FastB2B- DisINTx+
> > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> > <MAbort- >SERR- <PERR- INTx-
> > Latency: 0, Cache Line Size: 64 bytes
> > Interrupt: pin A routed to IRQ 49
> > Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M]
> > Region 2: Memory at df800000 (64-bit, prefetchable) [size=8M]
> > Region 4: I/O ports at be00 [size=256]
> > Region 5: Memory at fdd80000 (32-bit, non-prefetchable) [size=256K]
> > [virtual] Expansion ROM at d0000000 [disabled] [size=128K]
> > Capabilities: [48] Vendor Specific Information: Len=08 <?>
> > Capabilities: [50] Power Management version 3
> > Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> > PME(D0-,D1+,D2+,D3hot+,D3cold-)
> > Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> > Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
> > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1
> > unlimited
> > ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> > DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> > Unsupported-
> > RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> > MaxPayload 128 bytes, MaxReadReq 512 bytes
> > DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> > TransPend-
> > LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit
> > Latency L0s <64ns, L1 <1us
> > ClockPM- Surprise- LLActRep- BwNot-
> > LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+
> > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> > LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+
> > DLActive- BWMgmt- ABWMgmt-
> > DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-,
> > OBFF Not Supported
> > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-,
> > OBFF Disabled
> > LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
> > Transmit Margin: Normal Operating Range,
> > EnterModifiedCompliance- ComplianceSOS-
> > Compliance De-emphasis: -6dB
> > LnkSta2: Current De-emphasis Level: -3.5dB,
> > EqualizationComplete-, EqualizationPhase1-
> > EqualizationPhase2-, EqualizationPhase3-,
> > LinkEqualizationRequest-
> > Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
> > Address: 00000000fee00000 Data: 0000
> > Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1
> > Len=010 <?>
> > Capabilities: [150 v2] Advanced Error Reporting
> > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> > RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> > AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
> > Capabilities: [270 v1] #19
> > Capabilities: [2b0 v1] Address Translation Service (ATS)
> > ATSCap: Invalidate Queue Depth: 00
> > ATSCtl: Enable+, Smallest Translation Unit: 00
> > Capabilities: [2c0 v1] #13
> > Capabilities: [2d0 v1] #1b
> > Kernel driver in use: vfio-pci
> > Kernel modules: radeon
> >
> > address@hidden:~# lspci -vvv -s 02:00.0
> > 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
> > Pitcairn XT [Radeon HD 7870 GHz Edition] (prog-if 00 [VGA controller])
> > Subsystem: XFX Pine Group Inc. Device 3251
> > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> > Stepping- SERR+ FastB2B- DisINTx+
> > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> > <MAbort- >SERR- <PERR- INTx-
> > Latency: 0, Cache Line Size: 64 bytes
> > Interrupt: pin A routed to IRQ 48
> > Region 0: Memory at a0000000 (64-bit, prefetchable) [size=256M]
> > Region 2: Memory at fda80000 (64-bit, non-prefetchable) [size=256K]
> > Region 4: I/O ports at ee00 [size=256]
> > [virtual] Expansion ROM at fda00000 [disabled] [size=128K]
> > Capabilities: [48] Vendor Specific Information: Len=08 <?>
> > Capabilities: [50] Power Management version 3
> > Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> > PME(D0-,D1+,D2+,D3hot+,D3cold-)
> > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> > Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
> > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1
> > unlimited
> > ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> > DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> > Unsupported-
> > RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> > MaxPayload 128 bytes, MaxReadReq 512 bytes
> > DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> > TransPend-
> > LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit
> > Latency L0s <64ns, L1 <1us
> > ClockPM- Surprise- LLActRep- BwNot-
> > LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+
> > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> > LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive-
> > BWMgmt- ABWMgmt-
> > DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-,
> > OBFF Not Supported
> > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-,
> > OBFF Disabled
> > LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
> > Transmit Margin: Normal Operating Range,
> > EnterModifiedCompliance- ComplianceSOS-
> > Compliance De-emphasis: -6dB
> > LnkSta2: Current De-emphasis Level: -3.5dB,
> > EqualizationComplete-, EqualizationPhase1-
> > EqualizationPhase2-, EqualizationPhase3-,
> > LinkEqualizationRequest-
> > Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
> > Address: 00000000fee00000 Data: 0000
> > Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1
> > Len=010 <?>
> > Capabilities: [150 v2] Advanced Error Reporting
> > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
> > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
> > RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
> > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> > AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
> > Capabilities: [270 v1] #19
> > Capabilities: [2b0 v1] Address Translation Service (ATS)
> > ATSCap: Invalidate Queue Depth: 00
> > ATSCtl: Enable+, Smallest Translation Unit: 00
> > Capabilities: [2c0 v1] #13
> > Capabilities: [2d0 v1] #1b
> > Kernel driver in use: vfio-pci
> > Kernel modules: radeon
> >
> > > Alex
> > >
> >
> > --Maik
>
>
>
--Maik
- [Qemu-devel] Multi GPU passthrough via VFIO, Maik Broemme, 2014/02/05
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Alex Williamson, 2014/02/05
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Maik Broemme, 2014/02/05
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Alex Williamson, 2014/02/05
- Re: [Qemu-devel] Multi GPU passthrough via VFIO,
Maik Broemme <=
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Maik Broemme, 2014/02/05
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Alex Williamson, 2014/02/05
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Maik Broemme, 2014/02/06
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Maik Broemme, 2014/02/07
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Alex Williamson, 2014/02/07
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Maik Broemme, 2014/02/07
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Maik Broemme, 2014/02/13
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Alex Williamson, 2014/02/13
- Re: [Qemu-devel] Multi GPU passthrough via VFIO, Maik Broemme, 2014/02/14