Re: [Qemu-devel] [PATCH v6 00/16] KVM platform device passthrough

[Alex Williamson]

On Tue, 2014-09-09 at 08:31 +0100, Eric Auger wrote:
> This RFC series aims at enabling KVM platform device passthrough.
> It implements a VFIO platform device, derived from VFIO PCI device.
> 
> The VFIO platform device uses the host VFIO platform driver which must
> be bound to the assigned device prior to the QEMU system start.
> 
> - the guest can directly access the device register space
> - assigned device IRQs are transparently routed to the guest by
>   QEMU/KVM (3 methods currently are supported: user-level eventfd
>   handling, irqfd, forwarded IRQs)
> - iommu is transparently programmed to prevent the device from
>   accessing physical pages outside of the guest address space
> 
> This patch series is made of the following patch files:
> 
> 1-7) Modifications to PCI code to prepare for VFIO platform device
> 8) split of PCI specific code and generic code (move)
> 9-11) creation of the VFIO calxeda xgmac platform device, without irqfd
>       support (MMIO direct access and IRQ assignment).
> 12) fake injection test modality (to test multiple IRQ)
> 13) addition of irqfd/virqfd support
> 14-16) forwarded IRQ
> 
> Dependency List:
> 
> QEMU dependencies:
> [1] [PATCH v2 0/9] Dynamic sysbus device allocation support, Alex Graf
>     http://lists.gnu.org/archive/html/qemu-ppc/2014-07/msg00047.html
> [2] [RFC v3] machvirt dynamic sysbus device instantiation, Eric Auger
> [3] [PATCH v2 0/2] actual checks of KVM_CAP_IRQFD and KVM_CAP_IRQFD_RESAMPLE,
>     Eric Auger
>     http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg00589.html
> [4] [RFC] vfio: migration to trace points, Eric Auger
>     http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg00569.html
> 
> Kernel Dependencies:
> [5] [RFC Patch v6 0/20] VFIO support for platform devices, Antonios Motakis
>     https://www.mail-archive.com/address@hidden/msg103247.html
> [6] [PATCH v3] ARM: KVM: add irqfd support, Eric Auger
>     https://lkml.org/lkml/2014/9/1/141
> [7] arm/arm64: KVM: Various VGIC cleanups and improvements, Christoffer Dall
>     http://comments.gmane.org/gmane.linux.ports.arm.kernel/340430
> [8] [RFC v2 0/9] KVM-VFIO IRQ forward control, Eric Auger
>     https://lkml.org/lkml/2014/9/1/344
> [9] [RFC PATCH 0/9] ARM: Forwarding physical interrupts to a guest VM,
>     Marc Zyngier
>     http://lwn.net/Articles/603514/
> 
> kernel pieces can be found at:
> http://git.linaro.org/people/eric.auger/linux.git
> (branch 3.17rc3_irqfd_forward_integ_v2)
> QEMU pieces can be found at:
> http://git.linaro.org/people/eric.auger/qemu.git (branch vfio_integ_v6)
> 
> The patch series was tested on Calxeda Midway (ARMv7) where one xgmac
> is assigned to KVM host while the second one is assigned to the guest.
> Reworked PCI device is not tested.
> 
> Wiki for Calxeda Midway setup:
> https://wiki.linaro.org/LEG/Engineering/Virtualization/Platform_Device_Passthrough_on_Midway
> 
> History:
> 
> v5->v6:
> - rebase on 2.1rc5 PCI code
> - forwarded IRQ first integraton

Why?  Are there acceleration paths that you're concerned cannot be
implemented or we do not already have a proof of concept for?  The base
kernel patch series you depend on is 3 months old yet this series
continues to grow and add new dependencies.  Please let's prioritize
getting something upstream instead of adding more blockers to prevent
that.  Thanks,

Alex

> - vfio_device property renamed into host property
> - split IRQ setup in different functions that match the 3 supported
>   injection techniques (user handled eventfd, irqfd, forwarded IRQ):
>   removes dynamic switch between injection methods
> - introduce fake interrupts as a test modality:
>   x makes possible to test multiple IRQ user-side handling.
>   x this is a test feature only: enable to trigger a fd as if the
>     real physical IRQ hit. No virtual IRQ is injected into the guest
>     but handling is simulated so that the state machine can be tested
> - user handled eventfd:
>   x add mutex to protect IRQ state & list manipulation,
>   x correct misleading comment in vfio_intp_interrupt.
>   x Fix bugs using fake interrupt modality
> - irqfd no more advertised in this patchset (handled in [3])
> - VFIOPlatformDeviceClass becomes abstract and Calxeda xgmac device
>   and class is re-introduced (as per v4)
> - all DPRINTF removed in platform and replaced by trace-points
> - corrects compilation with configure --disable-kvm
> - simplifies the split for vfio_get_device and introduce a unique
>   specialized function named vfio_populate_device
> - group_list renamed into vfio_group_list
> - hw/arm/dyn_sysbus_devtree.c currently only support vfio-calxeda-xgmac
>   instantiation. Needs to be specialized for other VFIO devices
> - fix 2 bugs in dyn_sysbus_devtree(reg_attr index and compat)
> 
> v4->v5:
> - rebase on v2.1.0 PCI code
> - take into account Alex Williamson comments on PCI code rework
>   - trace updates in vfio_region_write/read
>   - remove fd from VFIORegion
>   - get/put ckeanup
> - bug fix: bar region's vbasedev field duly initialization
> - misc cleanups in platform device
> - device tree node generation removed from device and handled in
>   hw/arm/dyn_sysbus_devtree.c
> - remove "hw/vfio: add an example calxeda_xgmac": with removal of
>   device tree node generation we do not have so many things to
>   implement in that derived device yet. May be re-introduced later
>   on if needed typically for reset/migration.
> - no GSI routing table anymore
> 
> v3->v4 changes (Eric Auger, Alvise Rigo)
> - rebase on last VFIO PCI code (v2.1.0-rc0)
> - full git history rework to ease PCI code change review
> - mv include files in hw/vfio
> - DPRINTF reformatting temporarily moved out
> - support of VFIO virq (removal of resamplefd handler on user-side)
> - integration with sysbus dynamic instantiation framwork
> - removal of unrealize and cleanup routines until it is better
>   understood what is really needed
> - Support of VFIO for Amba devices should be handled in an inherited
>   device to specialize the device tree generation (clock handle currently
>   missing in framework however)
> - "Always use eventfd as notifying mechanism" temporarily moved out
> - static instantiation is not mainstream (although it remains possible)
>   note if static instantiation is used, irqfd must be setup in machine file
>   when virtual IRQ is known
> - create the GSI routing table on qemu side
> 
> v2->v3 changes (Alvise Rigo, Eric Auger):
> - Following Alex W recommandations, further efforts to factorize the
>   code between PCI:introduction of VFIODevice and VFIORegion
>   as base classes
> - unique reset handler for platform and PCI
> - cleanup following Kim's comments
> - multiple IRQ support mechanics should be in place although not
>   tested
> - Better handling of MMIO multiple regions
> - New features and fixes by Alvise (multiple compat string, exec
>   flag, force eventfd usage, amba device tree support)
> - irqfd support
> 
> v1->v2 changes (Kim Phillips, Eric Auger):
> - IRQ initial support (legacy mode where eventfds are handled on
>   user side)
> - hacked dynamic instantiation
> 
> v1 (Kim Phillips):
> - initial split between PCI and platform
> - MMIO support only
> - static instantiation
> 
> Best Regards
> 
> Eric
> 
> 
> Eric Auger (15):
>   hw/vfio/pci: Rename VFIODevice into VFIOPCIDevice
>   hw/vfio/pci: introduce VFIODevice
>   hw/vfio/pci: Introduce VFIORegion
>   hw/vfio/pci: split vfio_get_device
>   hw/vfio/pci: rename group_list into vfio_group_list
>   hw/vfio/pci: use name field in format strings
>   hw/vfio: create common module
>   hw/vfio/platform: add vfio-platform support
>   hw/vfio: calxeda xgmac device
>   hw/arm/dyn_sysbus_devtree: enable vfio-calxeda-xgmac dynamic
>     instantiation
>   vfio/platform: add fake injection modality
>   hw/vfio/platform: Add irqfd support
>   linux-headers: Update KVM headers from linux-next tag ToBeFilled
>   VFIO: COMMON: vfio_kvm_device_fd moved in the common header
>   VFIO: PLATFORM: add forwarded irq support
> 
> Kim Phillips (1):
>   vfio: move hw/misc/vfio.c to hw/vfio/pci.c     Move vfio.h into
>     include/hw/vfio
> 
>  LICENSE                              |    2 +-
>  MAINTAINERS                          |    2 +-
>  hw/Makefile.objs                     |    1 +
>  hw/arm/dyn_sysbus_devtree.c          |  141 +++
>  hw/misc/Makefile.objs                |    1 -
>  hw/ppc/spapr_pci_vfio.c              |    2 +-
>  hw/vfio/Makefile.objs                |    6 +
>  hw/vfio/calxeda_xgmac.c              |   57 ++
>  hw/vfio/common.c                     |  959 +++++++++++++++++++
>  hw/{misc/vfio.c => vfio/pci.c}       | 1670 
> +++++++---------------------------
>  hw/vfio/platform.c                   |  874 ++++++++++++++++++
>  include/hw/vfio/vfio-calxeda-xgmac.h |   41 +
>  include/hw/vfio/vfio-common.h        |  157 ++++
>  include/hw/vfio/vfio-platform.h      |   95 ++
>  include/hw/{misc => vfio}/vfio.h     |    0
>  linux-headers/linux/kvm.h            |    9 +
>  trace-events                         |  136 +--
>  17 files changed, 2739 insertions(+), 1414 deletions(-)
>  create mode 100644 hw/vfio/Makefile.objs
>  create mode 100644 hw/vfio/calxeda_xgmac.c
>  create mode 100644 hw/vfio/common.c
>  rename hw/{misc/vfio.c => vfio/pci.c} (65%)
>  create mode 100644 hw/vfio/platform.c
>  create mode 100644 include/hw/vfio/vfio-calxeda-xgmac.h
>  create mode 100644 include/hw/vfio/vfio-common.h
>  create mode 100644 include/hw/vfio/vfio-platform.h
>  rename include/hw/{misc => vfio}/vfio.h (100%)
>