Re: [PATCH v7 00/17] Add a host IOMMU device abstraction to check with vIOMMU

[Eric Auger]

Hi Zhenzhong,

On 6/5/24 10:30, Zhenzhong Duan wrote:
> Hi,
>
> This series introduce a HostIOMMUDevice abstraction and sub-classes.
> Also HostIOMMUDeviceCaps structure in HostIOMMUDevice and a new interface
> between vIOMMU and HostIOMMUDevice.
>
> A HostIOMMUDevice is an abstraction for an assigned device that is protected
> by a physical IOMMU (aka host IOMMU). The userspace interaction with this
> physical IOMMU can be done either through the VFIO IOMMU type 1 legacy
> backend or the new iommufd backend. The assigned device can be a VFIO device
> or a VDPA device. The HostIOMMUDevice is needed to interact with the host
> IOMMU that protects the assigned device. It is especially useful when the
> device is also protected by a virtual IOMMU as this latter use the translation
> services of the physical IOMMU and is constrained by it. In that context the
> HostIOMMUDevice can be passed to the virtual IOMMU to collect physical IOMMU
> capabilities such as the supported address width. In the future, the virtual
> IOMMU will use the HostIOMMUDevice to program the guest page tables in the
> first translation stage of the physical IOMMU.
>
> HostIOMMUDeviceClass::realize() is introduced to initialize
> HostIOMMUDeviceCaps and other fields of HostIOMMUDevice variants.
>
> HostIOMMUDeviceClass::get_cap() is introduced to query host IOMMU
> device capabilities.
>
> The class tree is as below:
>
>                               HostIOMMUDevice
>                                      | .caps
>                                      | .realize()
>                                      | .get_cap()
>                                      |
>             .-----------------------------------------------.
>             |                        |                      |
> HostIOMMUDeviceLegacyVFIO  {HostIOMMUDeviceLegacyVDPA}  HostIOMMUDeviceIOMMUFD
>             |                        |                      | [.iommufd]
>                                                             | [.devid]
>                                                             | [.ioas_id]
>                                                             | [.attach_hwpt()]
>                                                             | [.detach_hwpt()]
>                                                             |
>                                             .----------------------.
>                                             |                      |
>                          HostIOMMUDeviceIOMMUFDVFIO  
> {HostIOMMUDeviceIOMMUFDVDPA}
>                                           | [.vdev]                | {.vdev}
>
> * The attributes in [] will be implemented in nesting series.
> * The classes in {} will be implemented in future.
> * .vdev in different class points to different agent device,
> * i.e., VFIODevice or VDPADevice.
>
> PATCH1-4: Introduce HostIOMMUDevice and its sub classes
> PATCH5-10: Implement .realize() and .get_cap() handler
> PATCH11-14: Create HostIOMMUDevice instance and pass to vIOMMU
> PATCH15-17: Implement compatibility check between host IOMMU and 
> vIOMMU(intel_iommu)
>
> Test done:
> make check
> vfio device hotplug/unplug with different backend on linux
> reboot, kexec
> build test on linux and windows11
>
> Qemu code can be found at:
> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_preq_v7
>
> Besides the compatibility check in this series, in nesting series, this
> host IOMMU device is extended for much wider usage. For anyone interested
> on the nesting series, here is the link:
> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_rfcv2
>
> Thanks
> Zhenzhong
>
> Changelog:
> v7:
> - drop config CONFIG_HOST_IOMMU_DEVICE (Cédric)
> - introduce HOST_IOMMU_DEVICE_CAP_AW_BITS_MAX (Eric)
> - use iova_ranges method in iommufd.realize() (Eric)
> - introduce HostIOMMUDevice::name to facilitate tracing (Eric)
> - implement a custom destroy hash function (Cédric)
> - drop VTDHostIOMMUDevice and save HostIOMMUDevice in hash table (Eric)
> - move patch5 after patch1 (Eric)
> - squash patch3 and 4, squash patch12 and 13 (Eric)
> - refine comments (Eric)
> - collect Eric's R-B

for the whole series:
Reviewed-by: Eric Auger <eric.auger@redhat.com>

I exercised part of it using the virtio-iommu and this series on top
[RFC v2 0/7] VIRTIO-IOMMU/VFIO: Fix host iommu geometry handling for
hotplugged devices

Thanks

Eric
>
> v6:
> - open coded host_iommu_device_get_cap() to avoid #ifdef in intel_iommu.c 
> (Cédric)
>
> v5:
> - pci_device_set_iommu_device return true (Cédric)
> - fix build failure on windows (thanks Cédric found that issue)
>
> v4:
> - move properties vdev, iommufd and devid to nesting series where need it 
> (Cédric)
> - fix 32bit build with clz64 (Cédric)
> - change check_cap naming to get_cap (Cédric)
> - return bool if error is passed through errp (Cédric)
> - drop HostIOMMUDevice[LegacyVFIO|IOMMUFD|IOMMUFDVFIO] declaration (Cédric)
> - drop HOST_IOMMU_DEVICE_CAP_IOMMUFD (Cédric)
> - replace include directive with forward declaration (Cédric)
>
> v3:
> - refine declaration and doc for HostIOMMUDevice (Cédric, Philippe)
> - introduce HostIOMMUDeviceCaps, .realize() and .check_cap() (Cédric)
> - introduce helper range_get_last_bit() for range operation (Cédric)
> - separate pci_device_get_iommu_bus_devfn() in a prereq patch (Cédric)
> - replace HIOD_ abbreviation with HOST_IOMMU_DEVICE_ (Cédric)
> - add header in include/sysemu/iommufd.h (Cédric)
>
> v2:
> - use QOM to abstract host IOMMU device and its sub-classes (Cédric)
> - move host IOMMU device creation in attach_device() (Cédric)
> - refine pci_device_set/unset_iommu_device doc further (Eric)
> - define host IOMMU info format of different backend
> - implement get_host_iommu_info() for different backend (Cédric)
> - drop cap/ecap update logic (MST)
> - check aw-bits from get_host_iommu_info() in legacy mode
>
> v1:
> - use HostIOMMUDevice handle instead of union in VFIODevice (Eric)
> - change host_iommu_device_init to host_iommu_device_create
> - allocate HostIOMMUDevice in host_iommu_device_create callback
>   and set the VFIODevice base_hdev handle (Eric)
> - refine pci_device_set/unset_iommu_device doc (Eric)
> - use HostIOMMUDevice handle instead of union in VTDHostIOMMUDevice (Eric)
> - convert HostIOMMUDevice to sub object pointer in vtd_check_hdev
>
> rfcv2:
> - introduce common abstract HostIOMMUDevice and sub struct for different BEs 
> (Eric, Cédric)
> - remove iommufd_device.[ch] (Cédric)
> - remove duplicate iommufd/devid define from VFIODevice (Eric)
> - drop the p in aliased_pbus and aliased_pdevfn (Eric)
> - assert devfn and iommu_bus in pci_device_get_iommu_bus_devfn (Cédric, Eric)
> - use errp in iommufd_device_get_info (Eric)
> - split and simplify cap/ecap check/sync code in intel_iommu.c (Cédric)
> - move VTDHostIOMMUDevice declaration to intel_iommu_internal.h (Cédric)
> - make '(vtd->cap_reg >> 16) & 0x3fULL' a MACRO and add missed '+1' (Cédric)
> - block migration if vIOMMU cap/ecap updated based on host IOMMU cap/ecap
> - add R-B
>
> Yi Liu (2):
>   hw/pci: Introduce pci_device_[set|unset]_iommu_device()
>   intel_iommu: Implement [set|unset]_iommu_device() callbacks
>
> Zhenzhong Duan (15):
>   backends: Introduce HostIOMMUDevice abstract
>   backends/host_iommu_device: Introduce HostIOMMUDeviceCaps
>   vfio/container: Introduce TYPE_HOST_IOMMU_DEVICE_LEGACY_VFIO device
>   backends/iommufd: Introduce TYPE_HOST_IOMMU_DEVICE_IOMMUFD[_VFIO]
>     devices
>   range: Introduce range_get_last_bit()
>   vfio/container: Implement HostIOMMUDeviceClass::realize() handler
>   backends/iommufd: Introduce helper function
>     iommufd_backend_get_device_info()
>   vfio/iommufd: Implement HostIOMMUDeviceClass::realize() handler
>   vfio/container: Implement HostIOMMUDeviceClass::get_cap() handler
>   backends/iommufd: Implement HostIOMMUDeviceClass::get_cap() handler
>   vfio: Create host IOMMU device instance
>   hw/pci: Introduce helper function pci_device_get_iommu_bus_devfn()
>   vfio/pci: Pass HostIOMMUDevice to vIOMMU
>   intel_iommu: Extract out vtd_cap_init() to initialize cap/ecap
>   intel_iommu: Check compatibility with host IOMMU capabilities
>
>  MAINTAINERS                           |   2 +
>  include/hw/i386/intel_iommu.h         |   2 +
>  include/hw/pci/pci.h                  |  38 ++++-
>  include/hw/vfio/vfio-common.h         |   8 +
>  include/hw/vfio/vfio-container-base.h |   3 +
>  include/qemu/range.h                  |  11 ++
>  include/sysemu/host_iommu_device.h    |  91 ++++++++++++
>  include/sysemu/iommufd.h              |  19 +++
>  backends/host_iommu_device.c          |  33 +++++
>  backends/iommufd.c                    |  76 ++++++++--
>  hw/i386/intel_iommu.c                 | 203 ++++++++++++++++++++------
>  hw/pci/pci.c                          |  75 +++++++++-
>  hw/vfio/common.c                      |  16 +-
>  hw/vfio/container.c                   |  41 +++++-
>  hw/vfio/helpers.c                     |  17 +++
>  hw/vfio/iommufd.c                     |  37 ++++-
>  hw/vfio/pci.c                         |  19 ++-
>  backends/meson.build                  |   1 +
>  18 files changed, 623 insertions(+), 69 deletions(-)
>  create mode 100644 include/sysemu/host_iommu_device.h
>  create mode 100644 backends/host_iommu_device.c
>