[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
virtio-iommu hotplug issue
From: |
Akihiko Odaki |
Subject: |
virtio-iommu hotplug issue |
Date: |
Thu, 13 Apr 2023 13:49:43 +0900 |
User-agent: |
Mozilla/5.0 (X11; Linux aarch64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 |
Hi,
Recently I encountered a problem with the combination of Linux's
virtio-iommu driver and QEMU when a SR-IOV virtual function gets
disabled. I'd like to ask you what kind of solution is appropriate here
and implement the solution if possible.
A PCIe device implementing the SR-IOV specification exports a virtual
function, and the guest can enable or disable it at runtime by writing
to a configuration register. This effectively looks like a PCI device is
hotplugged for the guest. In such a case, the kernel assumes the
endpoint is detached from the virtio-iommu domain, but QEMU actually
does not detach it.
This inconsistent view of the removed device sometimes prevents the VM
from correctly performing the following procedure, for example:
1. Enable a VF.
2. Disable the VF.
3. Open a vfio container.
4. Open the group which the PF belongs to.
5. Add the group to the vfio container.
6. Map some memory region.
7. Close the group.
8. Close the vfio container.
9. Repeat 3-8
When the VF gets disabled, the kernel assumes the endpoint is detached
from the IOMMU domain, but QEMU actually doesn't detach it. Later, the
domain will be reused in step 3-8.
In step 7, the PF will be detached, and the kernel thinks there is no
endpoint attached and the mapping the domain holds is cleared, but the
VF endpoint is still attached and the mapping is kept intact.
In step 9, the same domain will be reused again, and the kernel requests
to create a new mapping, but it will conflict with the existing mapping
and result in -EINVAL.
This problem can be fixed by either of:
- requesting the detachment of the endpoint from the guest when the PCI
device is unplugged (the VF is disabled)
- detecting that the PCI device is gone and automatically detach it on
QEMU-side.
It is not completely clear for me which solution is more appropriate as
the virtio-iommu specification is written in a way independent of the
endpoint mechanism and does not say what should be done when a PCI
device is unplugged.
Regards,
Akihiko Odaki
- virtio-iommu hotplug issue,
Akihiko Odaki <=