[Qemu-devel] [RFC v4 00/13] KVM platform device passthrough

[Eric Auger]

This RFC series aims at enabling KVM platform device passthrough.
It implements a VFIO platform device which is bound to be dynamically
instantiated using -device option.

The VFIO platform device uses an host VFIO platform driver which must
be bound to the assigned device prior to the QEMU system start.

- the guest can directly access the device register space
- assigned device IRQs are transparently routed to the guest by
  QEMU/KVM (2 methods currently are supported)
- iommu is transparently programmed to prevent the device from
  accessing physical pages outside of the guest address space

The patch series was fully reworked between v3 and v4 to ease the
review of PCI modifications. Dynamic instantiation from command
line was cleaned up thanks to Alex Graf "Dynamic sysbus device
allocation support" patch series and its porting onto machvirt.

the patch relies on the following QEMU patch series:
- Alex Graf's "Dynamic sysbus device allocation support"
  http://lists.gnu.org/archive/html/qemu-ppc/2014-07/msg00047.html
- "machvirt dynamic sysbus device instantiation"
  Port Alex mechanics from e500 to virt. Propose to implement
  device tree generation in devices instead of machine file

The patch series is made of the following patch files:

1-6) Modifications to PCI code to prepare for VFIO platform device:
7) split of PCI specific code and generic code (move)
8) EXEC_FLAG setting
9) creation of the VFIO platform device, without irqfd support
   (MMIO direct access and IRQ assignment).
10-11) addition of irqfd/virqfd support
12) capability to dynamically instantiate the device
13) example derived VFIO device: calxeda xgmac

v3->v4 changes (Eric Auger, Alvise Rigo)
- rebase on last VFIO PCI code (v2.1.0-rc0)
- full git history rework to ease PCI code change review
- mv include files in hw/vfio
- DPRINTF reformatting temporarily moved out
- support of VFIO virq (removal of resamplefd handler on user-side)
- integration with sysbus dynamic instantiation framwork
- removal of unrealize and cleanup routines until it is better
  understood what is really needed
- Support of VFIO for Amba devices should be handled in an inherited
  device to specialize the device tree generation (clock handle currently
  missing in framework however)
- "Always use eventfd as notifying mechanism" temporarily moved out
- static instantiation is not mainstream (although it remains possible)
  note if static instantiation is used, irqfd must be setup in machine file
  when virtual IRQ is known
- create the GSI routing table on qemu side

v2->v3 changes (Alvise Rigo, Eric Auger):
- Following Alex W recommandations, further efforts to factorize the
  code between PCI:introduction of VFIODevice and VFIORegion
  as base classes
- unique reset handler for platform and PCI
- cleanup following Kim's comments
- multiple IRQ support mechanics should be in place although not
  tested
- Better handling of MMIO multiple regions
- New features and fixes by Alvise (multiple compat string, exec
  flag, force eventfd usage, amba device tree support)
- irqfd support

v1->v2 changes (Kim Phillips, Eric Auger):
- IRQ initial support (legacy mode where eventfds are handled on
  user side)
- hacked dynamic instantiation

v1 (Kim Phillips):
- initial split between PCI and platform
- MMIO support only
- static instantiation

This patch has the following kernel side dependencies:

- [RFC Patch v6 0/20] VFIO support for platform devices
https://www.mail-archive.com/address@hidden/msg103247.html
- [Patch] ARM: KVM: Handle IPA unmapping on memory region deletion
https://patches.linaro.org/27691/
- [PATCH v3] ARM: KVM: add irqfd and irq routing support
https://patches.linaro.org/32261/
- [PATCH] ARM: KVM: Enable the KVM-VFIO device
https://lists.cs.columbia.edu/pipermail/kvmarm/2014-March/008629.html
- [PATCH v2] ARM: KVM: user_mem_abort: support stage 2 MMIO page mapping
http://www.spinics.net/lists/kvm/msg105083.html

The patch series was tested on Calxeda Midway (ARMv7) where one xgmac
is assigned to KVM host while the second one is assigned to the guest.

Unfortunately a single IRQ is exercised.

Next steps:
- use of "ARM: Forwarding physical interrupts to a guest VM"
- unbind/migration/reset problematics

Here are the instructions to test on a Calxeda Midway:

https://wiki.linaro.org/LEG/Engineering/Virtualization/Platform_Device_Passthrough_on_Midway

git://git.linaro.org/people/eric.auger/linux.git (branch irqfd_integ_v3)
git://git.linaro.org/people/eric.auger/qemu.git (branch vfio_integ_v4)

Best Regards

Eric


Alvise Rigo (1):
  hw/vfio/common: Add EXEC_FLAG to VFIO DMA mappings

Eric Auger (11):
  hw/vfio/pci: Rename VFIODevice into VFIOPCIDevice
  hw/vfio/pci: Remove unneeded include files
  hw/vfio/pci: introduce VFIODevice
  hw/vfio/pci: Introduce VFIORegion
  hw/vfio/pci: split vfio_get_device
  hw/vfio: create common module
  hw/vfio/platform: add vfio-platform support
  hw/intc/arm_gic_kvm: enable irqfd and set routing table
  hw/vfio/platform: Add irqfd support
  hw/vfio/platform: add default dt generation for vfio device
  hw/vfio: add an example calxeda_xgmac

Kim Phillips (1):
  vfio: move hw/misc/vfio.c to hw/vfio/pci.c     Move vfio.h into
    include/hw/vfio

 LICENSE                          |    2 +-
 MAINTAINERS                      |    2 +-
 hw/Makefile.objs                 |    1 +
 hw/intc/arm_gic_kvm.c            |   11 +
 hw/misc/Makefile.objs            |    1 -
 hw/ppc/spapr_pci_vfio.c          |    2 +-
 hw/vfio/Makefile.objs            |    6 +
 hw/vfio/calxeda_xgmac.c          |  165 +++++
 hw/vfio/common.c                 | 1003 +++++++++++++++++++++++++
 hw/{misc/vfio.c => vfio/pci.c}   | 1514 +++++++-------------------------------
 hw/vfio/platform.c               |  766 +++++++++++++++++++
 include/hw/vfio/vfio-common.h    |  149 ++++
 include/hw/vfio/vfio-platform.h  |   75 ++
 include/hw/{misc => vfio}/vfio.h |    0
 linux-headers/linux/vfio.h       |    2 +
 15 files changed, 2465 insertions(+), 1234 deletions(-)
 create mode 100644 hw/vfio/Makefile.objs
 create mode 100644 hw/vfio/calxeda_xgmac.c
 create mode 100644 hw/vfio/common.c
 rename hw/{misc/vfio.c => vfio/pci.c} (71%)
 create mode 100644 hw/vfio/platform.c
 create mode 100644 include/hw/vfio/vfio-common.h
 create mode 100644 include/hw/vfio/vfio-platform.h
 rename include/hw/{misc => vfio}/vfio.h (100%)

-- 
1.8.3.2