[Qemu-devel] Qemu and 32 PCIe devices

[Marcin Juszkiewicz]

Hello

Few days ago I had an issue with getting PCIe hotplug working on AArch64
machine. Enabled PCI hotplug in kernel and then got hit by some issues.

Out setup is a bunch of aarch64 servers and we use OpenStack to provide
access to arm64 systems. OpenStack uses libvirt to control VMs and
allows to add network interfaces and disk volumes to running systems.

By libvirt AArch64 is treated as PCIe machine without legacy PCI slots.
So to hotplug anything you first need to have enough pcie-root-port
entries as it is described in Qemu docs/pcie.txt and by patch to libvirt
documentation [1][2].

1. https://bugs.linaro.org/attachment.cgi?id=782
2. https://www.redhat.com/archives/libvir-list/2017-July/msg01033.html


But things get complicated once you are going to 32 PCIe devices limit
(which in our setup will rather not happen). UEFI first takes ages to
boot just to land in UEFI shell as it forgot all PCIe devices. With 31
devices it boots (also after long time).


I attached two xml files with VM definitions (for use with virsh). Also
attached their qemu command lines. One has 31 PCI devices, second has
32. Both use [30] as rootfs.

30.
https://builds.96boards.org/snapshots/reference-platform/components/developer-cloud/debian/cloud-image/30/debian-cloud-image.qcow2

Output from 31 PCI devices VM:

address@hidden:~# lspci;lspci |wc -l
00:00.0 Host bridge: Red Hat, Inc. Device 0008
00:01.0 PCI bridge: Red Hat, Inc. Device 000c
00:01.1 PCI bridge: Red Hat, Inc. Device 000c
00:01.2 PCI bridge: Red Hat, Inc. Device 000c
00:01.3 PCI bridge: Red Hat, Inc. Device 000c
00:01.4 PCI bridge: Red Hat, Inc. Device 000c
00:01.5 PCI bridge: Red Hat, Inc. Device 000c
00:01.6 PCI bridge: Red Hat, Inc. Device 000c
00:01.7 PCI bridge: Red Hat, Inc. Device 000c
00:02.0 PCI bridge: Red Hat, Inc. Device 000c
00:02.1 PCI bridge: Red Hat, Inc. Device 000c
00:02.2 PCI bridge: Red Hat, Inc. Device 000c
00:02.3 PCI bridge: Red Hat, Inc. Device 000c
00:02.4 PCI bridge: Red Hat, Inc. Device 000c
00:02.5 PCI bridge: Red Hat, Inc. Device 000c
00:02.6 PCI bridge: Red Hat, Inc. Device 000c
01:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
02:00.0 SCSI storage controller: Red Hat, Inc Device 1048 (rev 01)
03:00.0 Communication controller: Red Hat, Inc Device 1043 (rev 01)
04:00.0 SCSI storage controller: Red Hat, Inc Device 1042 (rev 01)
05:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
06:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
07:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
08:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
09:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
0a:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
0b:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
0c:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
0d:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
0e:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
0f:00.0 Ethernet controller: Red Hat, Inc Device 1041 (rev 01)
31
address@hidden:~# lspci -t
-[0000:00]-+-00.0
           +-01.0-[01]----00.0
           +-01.1-[02]----00.0
           +-01.2-[03]----00.0
           +-01.3-[04]----00.0
           +-01.4-[05]----00.0
           +-01.5-[06]----00.0
           +-01.6-[07]----00.0
           +-01.7-[08]----00.0
           +-02.0-[09]----00.0
           +-02.1-[0a]----00.0
           +-02.2-[0b]----00.0
           +-02.3-[0c]----00.0
           +-02.4-[0d]----00.0
           +-02.5-[0e]----00.0
           \-02.6-[0f]----00.0

>From what I was told some parts of that issue lies in UEFI, some in
Qemu, some in Linux kernel.

vm-pci-32.xml
Description: Text Data

vm-pci-31.xml
Description: Text Data

pci-32.txt
Description: Text document

pci-31.txt
Description: Text document