[Bug 1921444] Re: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest

[xudong]

** Attachment added: "guest_dmesg.log"
   
https://bugs.launchpad.net/qemu/+bug/1921444/+attachment/5481022/+files/guest_dmesg.log

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1921444

Title:
  Q35 doesn't support to hot add the 2nd PCIe device to KVM guest

Status in QEMU:
  New

Bug description:
  KVM: https://git.kernel.org/pub/scm/virt/kvm/kvm.git  branch: next, commit: 
4a98623d
  Qemu: https://git.qemu.org/git/qemu.git  branch: master, commit: 9e2e9fe3

  Created a KVM guest with Q35 chipset, and try to hot add 2 PCIe device
  to guest with qemu internal command device_add, the 1st device can be
  added successfully, but the 2nd device failed to hot add.

  If guest chipset is legacy i440fx, the 2 device can be added
  successfully.

  1. Enable VT-d in BIOS
  2. load KVM modules in Linux OS: modprobe kvm; modprobe kvm_intel
  3. Bind 2 device to vfio-pci
      echo 0000:b1:00.0 > /sys/bus/pci/drivers/i40e/unbind
      echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id 
      echo 0000:b1:00.1 > /sys/bus/pci/drivers/i40e/unbind
      echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id 

  4. create guest with Q35 chipset:
  qemu-system-x86_64 --accel kvm -m 4096 -smp 4 -drive 
file=/home/rhel8.2.qcow2,if=none,id=virtio-disk0 -device 
virtio-blk-pci,drive=virtio-disk0 -cpu host -machine q35 -device 
pcie-root-port,id=root1 -daemonize

  5. hot add the 1st device to guest successfully
  in guest qemu monitor "device_add vfio-pci,host=b1:00.0,id=nic0,bus=root1"
  6. hot add the 2nd device to guest
  in guest qemu monitor "device_add vfio-pci,host=b1:00.1,id=nic1,bus=root1"
  The 2nd device doesn't be added in guest, and the 1st device is removed from 
guest. 

  Guest partial log:
  [  110.452272] pcieport 0000:00:04.0: pciehp: Slot(0): Attention button 
pressed
  [  110.453314] pcieport 0000:00:04.0: pciehp: Slot(0) Powering on due to 
button press
  [  110.454156] pcieport 0000:00:04.0: pciehp: Slot(0): Card present
  [  110.454792] pcieport 0000:00:04.0: pciehp: Slot(0): Link Up
  [  110.580927] pci 0000:01:00.0: [8086:1572] type 00 class 0x020000
  [  110.582560] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x007fffff 64bit 
pref]
  [  110.583453] pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x00007fff 64bit 
pref]
  [  110.584278] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
  [  110.585051] pci 0000:01:00.0: Max Payload Size set to 128 (was 512, max 
2048)
  [  110.586621] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
  [  110.588140] pci 0000:01:00.0: BAR 0: no space for [mem size 0x00800000 
64bit pref]
  [  110.588954] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x00800000 
64bit pref]
  [  110.589797] pci 0000:01:00.0: BAR 6: assigned [mem 0xfe800000-0xfe87ffff 
pref]
  [  110.590703] pci 0000:01:00.0: BAR 3: assigned [mem 0xfe000000-0xfe007fff 
64bit pref]
  [  110.592085] pcieport 0000:00:04.0: PCI bridge to [bus 01]
  [  110.592755] pcieport 0000:00:04.0:   bridge window [io  0x1000-0x1fff]
  [  110.594403] pcieport 0000:00:04.0:   bridge window [mem 
0xfe800000-0xfe9fffff]
  [  110.595847] pcieport 0000:00:04.0:   bridge window [mem 
0xfe000000-0xfe1fffff 64bit pref]
  [  110.597867] PCI: No. 2 try to assign unassigned res
  [  110.597870] release child resource [mem 0xfe000000-0xfe007fff 64bit pref]
  [  110.597871] pcieport 0000:00:04.0: resource 15 [mem 0xfe000000-0xfe1fffff 
64bit pref] released
  [  110.598881] pcieport 0000:00:04.0: PCI bridge to [bus 01]
  [  110.600789] pcieport 0000:00:04.0: BAR 15: assigned [mem 
0x180000000-0x180bfffff 64bit pref]
  [  110.601731] pci 0000:01:00.0: BAR 0: assigned [mem 0x180000000-0x1807fffff 
64bit pref]
  [  110.602849] pci 0000:01:00.0: BAR 3: assigned [mem 0x180800000-0x180807fff 
64bit pref]
  [  110.604069] pcieport 0000:00:04.0: PCI bridge to [bus 01]
  [  110.604941] pcieport 0000:00:04.0:   bridge window [io  0x1000-0x1fff]
  [  110.606237] pcieport 0000:00:04.0:   bridge window [mem 
0xfe800000-0xfe9fffff]
  [  110.607401] pcieport 0000:00:04.0:   bridge window [mem 
0x180000000-0x180bfffff 64bit pref]
  [  110.653661] i40e: Intel(R) Ethernet Connection XL710 Network Driver
  [  110.654443] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
  [  110.655314] i40e 0000:01:00.0: enabling device (0140 -> 0142)
  [  110.672396] i40e 0000:01:00.0: fw 6.0.48442 api 1.7 nvm 6.01 0x800035b1 
1.1747.0 [8086:1572] [8086:0008]
  [  110.750054] i40e 0000:01:00.0: MAC address: 3c:fd:fe:c0:59:98
  [  110.751792] i40e 0000:01:00.0: FW LLDP is enabled
  [  110.764644] i40e 0000:01:00.0 eth1: NIC Link is Up, 10 Gbps Full Duplex, 
Flow Control: None
  [  110.779390] i40e 0000:01:00.0: PCI-Express: Speed 8.0GT/s Width x8
  [  110.789841] i40e 0000:01:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 4 
RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
  [  111.817553] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
  [  205.130288] pcieport 0000:00:04.0: pciehp: Slot(0): Attention button 
pressed
  [  205.131743] pcieport 0000:00:04.0: pciehp: Slot(0): Powering off due to 
button press
  [  205.133233] pcieport 0000:00:04.0: pciehp: Slot(0): Card not present
  [  205.135728] i40e 0000:01:00.0: i40e_ptp_stop: removed PHC on eth1

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1921444/+subscriptions