[RFC PATCH v8 00/21] Net Control VQ support with asid in vDPA SVQ

[Eugenio Pérez]

Control virtqueue is used by networking device for accepting various
commands from the driver. It's a must to support multiqueue and other
configurations.

Shadow VirtQueue (SVQ) already makes possible migration of virtqueue
states, effectively intercepting them so qemu can track what regions of memory
are dirty because device action and needs migration. However, this does not
solve networking device state seen by the driver because CVQ messages, like
changes on MAC addresses from the driver.

To solve that, this series uses SVQ infraestructure proposed to intercept
networking control messages used by the device. This way, qemu is able to
update VirtIONet device model and to migrate it.

However, to intercept all queues would slow device data forwarding. To solve
that, only the CVQ must be intercepted all the time. This is achieved using
the ASID infraestructure, that allows different translations for different
virtqueues. The most updated kernel part of ASID is proposed at [1].

You can run qemu in two modes after applying this series: only intercepting
cvq with x-cvq-svq=on or intercept all the virtqueues adding cmdline x-svq=on:

-netdev 
type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,id=vhost-vdpa0,x-cvq-svq=on,x-svq=on

First three patches enable the update of the virtio-net device model for each
CVQ message acknoledged by the device.

Patches from 5 to 9 enables individual SVQ to copy the buffers to QEMU's VA.
This allows simplyfing the memory mapping, instead of map all the guest's
memory like in the data virtqueues.

Patch 10 allows to inject control messages to the device. This allows to set
state to the device both at QEMU startup and at live migration destination. In
the future, this may also be used to emulate _F_ANNOUNCE.

Patch 11 updates kernel headers, but it assign random numbers to needed ioctls
because they are still not accepted in the kernel.

Patches 12-16 enables the set of the features of the net device model to the
vdpa device at device start.

Last ones enables the sepparated ASID and SVQ.

Comments are welcomed.

TODO:
* Fallback on regular CVQ if QEMU cannot isolate in its own ASID by any
  reason, blocking migration. This is tricky, since it can cause that the VM
  cannot be migrated anymore, so some way of block it must be used.
* Review failure paths, some are with TODO notes, other don't.

Changes from rfc v7:
* Don't map all guest space in ASID 1 but copy all the buffers. No need for
  more memory listeners.
* Move net backend start callback to SVQ.
* Wait for device CVQ commands used by the device at SVQ start, avoiding races.
* Changed ioctls, but they're provisional anyway.
* Reorder commits so refactor and code adding ones are closer to usage.
* Usual cleaning: better tracing, doc, patches messages, ...

Changes from rfc v6:
* Fix bad iotlb updates order when batching was enabled
* Add reference counting to iova_tree so cleaning is simpler.

Changes from rfc v5:
* Fixes bad calculus of cvq end group when MQ is not acked by the guest.

Changes from rfc v4:
* Add missing tracing
* Add multiqueue support
* Use already sent version for replacing g_memdup
* Care with memory management

Changes from rfc v3:
* Fix bad returning of descriptors to SVQ list.

Changes from rfc v2:
* Fix use-after-free.

Changes from rfc v1:
* Rebase to latest master.
* Configure ASID instead of assuming cvq asid != data vqs asid.
* Update device model so (MAC) state can be migrated too.

[1] https://lkml.kernel.org/kvm/20220224212314.1326-1-gdawar@xilinx.com/

Eugenio Pérez (21):
  virtio-net: Expose ctrl virtqueue logic
  vhost: Add custom used buffer callback
  vdpa: control virtqueue support on shadow virtqueue
  virtio: Make virtqueue_alloc_element non-static
  vhost: Add vhost_iova_tree_find
  vdpa: Add map/unmap operation callback to SVQ
  vhost: move descriptor translation to vhost_svq_vring_write_descs
  vhost: Add SVQElement
  vhost: Add svq copy desc mode
  vhost: Add vhost_svq_inject
  vhost: Update kernel headers
  vdpa: delay set_vring_ready after DRIVER_OK
  vhost: Add ShadowVirtQueueStart operation
  vhost: Make possible to check for device exclusive vq group
  vhost: add vhost_svq_poll
  vdpa: Add vhost_vdpa_start_control_svq
  vdpa: Add asid attribute to vdpa device
  vdpa: Extract get features part from vhost_vdpa_get_max_queue_pairs
  vhost: Add reference counting to vhost_iova_tree
  vdpa: Add x-svq to NetdevVhostVDPAOptions
  vdpa: Add x-cvq-svq

 qapi/net.json                                |  13 +-
 hw/virtio/vhost-iova-tree.h                  |   7 +-
 hw/virtio/vhost-shadow-virtqueue.h           |  61 ++-
 include/hw/virtio/vhost-vdpa.h               |   3 +
 include/hw/virtio/vhost.h                    |   3 +
 include/hw/virtio/virtio-net.h               |   4 +
 include/hw/virtio/virtio.h                   |   1 +
 include/standard-headers/linux/vhost_types.h |  11 +-
 linux-headers/linux/vhost.h                  |  25 +-
 hw/net/vhost_net.c                           |   5 +-
 hw/net/virtio-net.c                          |  84 +++--
 hw/virtio/vhost-iova-tree.c                  |  35 +-
 hw/virtio/vhost-shadow-virtqueue.c           | 378 ++++++++++++++++---
 hw/virtio/vhost-vdpa.c                       | 206 +++++++++-
 hw/virtio/virtio.c                           |   2 +-
 net/vhost-vdpa.c                             | 294 ++++++++++++++-
 hw/virtio/trace-events                       |  10 +-
 17 files changed, 1012 insertions(+), 130 deletions(-)

-- 
2.27.0