Failure of hot plugging secondary virtio_blk into q35 Windows 2019

[Annie.li]

Hello,

I've found an issue when hot-plugging the secondary virtio_blk device 
into q35 Windows guest(2019) with upstream qemu 6.1.0(+1 patch). The 
first disk can be hot-plugged successfully.

The qemu options for PCIe root port is,

 -device 
pcie-root-port,port=2,chassis=2,id=pciroot2,bus=pcie.0,addr=0x2,multifunction=on 
\
 -device 
pcie-root-port,port=3,chassis=3,id=pciroot3,bus=pcie.0,addr=0x3,multifunction=on 
\
 -device 
pcie-root-port,port=4,chassis=4,id=pciroot4,bus=pcie.0,addr=0x4,multifunction=on 
\
 -device 
pcie-root-port,port=5,chassis=5,id=pciroot5,bus=pcie.0,addr=0x5,multifunction=on 
\
 -device 
pcie-root-port,port=6,chassis=6,id=pciroot6,bus=pcie.0,addr=0x6,multifunction=on 
\

The command to hotplug 1st virtio_blk disk is following, the PCI slot of 
the 1st virtio_blk is Pci slot 0(PCI bus 1, device 0, function 0).

 drive_add auto 
file=block_10.qcow2,format=qcow2,if=none,id=drive10,cache=none

 device_add virtio-blk-pci,drive=drive10,id=block-disk10,bus=pciroot2

Following is the related "info mtree" after the 1st virtio_blk device is 
hot plugged

memory-region: pci_bridge_pci
  0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
    00000000febff000-00000000febfffff (prio 1, i/o): virtio-blk-pci-msix
      00000000febff000-00000000febff01f (prio 0, i/o): msix-table
      00000000febff800-00000000febff807 (prio 0, i/o): msix-pba
    0000000fffffc000-0000000fffffffff (prio 1, i/o): virtio-pci
      0000000fffffc000-0000000fffffcfff (prio 0, i/o): virtio-pci-common
      0000000fffffd000-0000000fffffdfff (prio 0, i/o): virtio-pci-isr
      0000000fffffe000-0000000fffffefff (prio 0, i/o): virtio-pci-device
      0000000ffffff000-0000000fffffffff (prio 0, i/o): virtio-pci-notify

memory-region: pci_bridge_pci
  0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci

memory-region: pci_bridge_pci
  0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci

memory-region: pci_bridge_pci
  0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci

memory-region: pci_bridge_pci
  0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci

Right after the secondary virtio_blk device is hot-plugged, a yellow 
mark shows on the first virtio_blk device in the Windows guest. The PCI 
slot info of the 2nd virtio_blk is Pci slot 0(PCI bus 2, device 0, 
function 0). The debug log of Windows virtio_blk driver shows a 
"ScsiStopAdapter" adapter control operation is triggered first, and then 
"StorSurpriseRemoval". From the following "info mtree", it seems the 2nd 
virtio_blk device is occupying the same memory resource as the above 1st 
virtio_blk device. Maybe this causes the failure of the 1st virtio_blk 
device and then the system assume it is surprisingly removed?

The command to hotplug 2nd virtio_blk disk,

 drive_add auto 
file=block_11.qcow2,format=qcow2,if=none,id=drive11,cache=none

 device_add virtio-blk-pci,drive=drive11,id=block-disk11,bus=pciroot3

Following is the related "info mtree" after the 2nd virtio_blk device is 
hot-plugged,

memory-region: pci_bridge_pci
  0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci

memory-region: pci_bridge_pci
  0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci
    00000000febff000-00000000febfffff (prio 1, i/o): virtio-blk-pci-msix
      00000000febff000-00000000febff01f (prio 0, i/o): msix-table
      00000000febff800-00000000febff807 (prio 0, i/o): msix-pba
    0000000fffffc000-0000000fffffffff (prio 1, i/o): virtio-pci
      0000000fffffc000-0000000fffffcfff (prio 0, i/o): virtio-pci-common
      0000000fffffd000-0000000fffffdfff (prio 0, i/o): virtio-pci-isr
      0000000fffffe000-0000000fffffefff (prio 0, i/o): virtio-pci-device
      0000000ffffff000-0000000fffffffff (prio 0, i/o): virtio-pci-notify

memory-region: pci_bridge_pci
  0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci

memory-region: pci_bridge_pci
  0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci

memory-region: pci_bridge_pci
  0000000000000000-ffffffffffffffff (prio 0, i/o): pci_bridge_pci


Note: I've patched the upstream 6.1.0 qemu with following patch,

https://patchwork.kernel.org/project/qemu-devel/patch/20210916132838.3469580-3-ani@anisinha.ca/

the acpi-pci-hotplug memory is following as expected,

  0000000000000cc0-0000000000000cd7 (prio 0, i/o): acpi-pci-hotplug
  0000000000000cd8-0000000000000ce3 (prio 0, i/o): acpi-mem-hotplug

Thanks

Annie