|
From: | Akihiko Odaki |
Subject: | Re: virtio-iommu hotplug issue |
Date: | Fri, 14 Apr 2023 11:51:27 +0900 |
User-agent: | Mozilla/5.0 (X11; Linux aarch64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 |
On 2023/04/13 22:39, Eric Auger wrote:
Hi, On 4/13/23 13:01, Akihiko Odaki wrote:On 2023/04/13 19:40, Jean-Philippe Brucker wrote:Hello, On Thu, Apr 13, 2023 at 01:49:43PM +0900, Akihiko Odaki wrote:Hi, Recently I encountered a problem with the combination of Linux's virtio-iommu driver and QEMU when a SR-IOV virtual function gets disabled. I'd like to ask you what kind of solution is appropriate here and implement the solution if possible. A PCIe device implementing the SR-IOV specification exports a virtual function, and the guest can enable or disable it at runtime by writing to a configuration register. This effectively looks like a PCI device is hotplugged for the guest.Just so I understand this better: the guest gets a whole PCIe device PF that implements SR-IOV, and so the guest can dynamically create VFs? Out of curiosity, is that a hardware device assigned to the guest with VFIO, or a device emulated by QEMU?Yes, that's right. The guest can dynamically create and delete VFs. The device is emulated by QEMU: igb, an Intel NIC recently added to QEMU and projected to be released as part of QEMU 8.0.From below description In understand you then bind this emulated device to VFIO on guest, correct?
Yes, that's correct.
In such a case, the kernel assumes the endpoint is detached from the virtio-iommu domain, but QEMU actually does not detach it.The QEMU virtio-iommu device executes commands from the virtio-iommu driver and my understanding is the VFIO infra is not in trouble here. As suggested by Jean, a detach command probably is missed.
VFIO just illustrates the problem and the origin of the problem is indeed virtio-iommu.
Regards, Akihiko Odaki
This inconsistent view of the removed device sometimes prevents the VM from correctly performing the following procedure, for example: 1. Enable a VF. 2. Disable the VF. 3. Open a vfio container. 4. Open the group which the PF belongs to. 5. Add the group to the vfio container. 6. Map some memory region. 7. Close the group. 8. Close the vfio container. 9. Repeat 3-8 When the VF gets disabled, the kernel assumes the endpoint is detached from the IOMMU domain, but QEMU actually doesn't detach it. Later, the domain will be reused in step 3-8. In step 7, the PF will be detached, and the kernel thinks there is no endpoint attached and the mapping the domain holds is cleared, but the VF endpoint is still attached and the mapping is kept intact. In step 9, the same domain will be reused again, and the kernel requests to create a new mapping, but it will conflict with the existing mapping and result in -EINVAL. This problem can be fixed by either of: - requesting the detachment of the endpoint from the guest when the PCI device is unplugged (the VF is disabled)Yes, I think this is an issue in the virtio-iommu driver, which should be sending a DETACH request when the VF is disabled, likely from viommu_release_device(). I'll work on a fix unless you would like to do itIt will be nice if you prepare a fix. I will test your patch with my workload if you share it with me.I can help testing too Thanks EricRegards, Akihiko Odaki- detecting that the PCI device is gone and automatically detach it on QEMU-side. It is not completely clear for me which solution is more appropriate as the virtio-iommu specification is written in a way independent of the endpoint mechanism and does not say what should be done when a PCI device is unplugged.Yes, I'm not sure it's in scope for the specification, it's more about software guidance Thanks, Jean
[Prev in Thread] | Current Thread | [Next in Thread] |