Re: [Qemu-devel] [PATCH 0/4] add failover feature for assigned network devices

[Laine Stump]

On 6/11/19 11:51 AM, Michael S. Tsirkin wrote:
> On Tue, Jun 11, 2019 at 11:42:54AM -0400, Laine Stump wrote:
> > On 5/17/19 8:58 AM, Jens Freimann wrote:
> > > This is another attempt at implementing the host side of the
> > > net_failover concept
> > > (https://www.kernel.org/doc/html/latest/networking/net_failover.html)
> > >
> > > Changes since last RFC:
> > > - work around circular dependency of commandline options. Just add
> > >     failover=on to the virtio-net standby options and reference it from
> > >     primary (vfio-pci) device with standby=<id>
> > > - add patch 3/4 to allow migration of vfio-pci device when it is part of a
> > >     failover pair, still disallow for all other devices
> > > - add patch 4/4 to allow unplug of device during migrationm, make an
> > >     exception for failover primary devices. I'd like feedback on how to
> > >     solve this more elegant. I added a boolean to DeviceState, have it
> > >     default to false for all devices except for primary devices.
> > > - not tested yet with surprise removal
> > > - I don't expect this to go in as it is, still needs more testing but
> > >     I'd like to get feedback on above mentioned changes.
> > >
> > > The general idea is that we have a pair of devices, a vfio-pci and a
> > > emulated device. Before migration the vfio device is unplugged and data
> > > flows to the emulated device, on the target side another vfio-pci device
> > > is plugged in to take over the data-path. In the guest the net_failover
> > > module will pair net devices with the same MAC address.
> > >
> > > * In the first patch the infrastructure for hiding the device is added
> > >     for the qbus and qdev APIs.
> > >
> > > * In the second patch the virtio-net uses the API to defer adding the vfio
> > >     device until the VIRTIO_NET_F_STANDBY feature is acked.
> > >
> > > Previous discussion:
> > >     RFC v1 https://patchwork.ozlabs.org/cover/989098/
> > >     RFC v2 https://www.mail-archive.com/address@hidden/msg606906.html
> > >
> > > To summarize concerns/feedback from previous discussion:
> > > 1.- guest OS can reject or worse _delay_ unplug by any amount of time.
> > >     Migration might get stuck for unpredictable time with unclear reason.
> > >     This approach combines two tricky things, hot/unplug and migration.
> > >     -> We can surprise-remove the PCI device and in QEMU we can do all
> > >        necessary rollbacks transparent to management software. Will it be
> > >        easy, probably not.
> > > 2. PCI devices are a precious ressource. The primary device should never
> > >     be added to QEMU if it won't be used by guest instead of hiding it in
> > >     QEMU.
> > >     -> We only hotplug the device when the standby feature bit was
> > >        negotiated. We save the device cmdline options until we need it for
> > >        qdev_device_add()
> > >        Hiding a device can be a useful concept to model. For example a
> > >        pci device in a powered-off slot could be marked as hidden until the 
> > > slot is
> > >        powered on (mst).
> > > 3. Management layer software should handle this. Open Stack already has
> > >     components/code to handle unplug/replug VFIO devices and metadata to
> > >     provide to the guest for detecting which devices should be paired.
> > >     -> An approach that includes all software from firmware to
> > >        higher-level management software wasn't tried in the last years. This is
> > >        an attempt to keep it simple and contained in QEMU as much as possible.
> > > 4. Hotplugging a device and then making it part of a failover setup is
> > >      not possible
> > >     -> addressed by extending qdev hotplug functions to check for hidden
> > >        attribute, so e.g. device_add can be used to plug a device.
> > >
> > >
> > > I have tested this with a mlx5 NIC and was able to migrate the VM with
> > > above mentioned workarounds for open problems.
> > >
> > > Command line example:
> > >
> > > qemu-system-x86_64 -enable-kvm -m 3072 -smp 3 \
> > >           -machine q35,kernel-irqchip=split -cpu host   \
> > >           -k fr   \
> > >           -serial stdio   \
> > >           -net none \
> > >           -qmp unix:/tmp/qmp.socket,server,nowait \
> > >           -monitor telnet:127.0.0.1:5555,server,nowait \
> > >           -device pcie-root-port,id=root0,multifunction=on,chassis=0,addr=0xa \
> > >           -device pcie-root-port,id=root1,bus=pcie.0,chassis=1 \
> > >           -device pcie-root-port,id=root2,bus=pcie.0,chassis=2 \
> > >           -netdev 
> > > tap,script=/root/bin/bridge.sh,downscript=no,id=hostnet1,vhost=on \
> > >           -device 
> > > virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:6f:55:cc,bus=root2,failover=on
> > >  \
> > >           /root/rhel-guest-image-8.0-1781.x86_64.qcow2
> > >
> > > Then the primary device can be hotplugged via
> > >    (qemu) device_add vfio-pci,host=5e:00.2,id=hostdev0,bus=root1,standby=net1
> >
> >
> > I guess this is the commandline on the migration destination, and as far as
> > I understand from this example, on the destination we (meaning libvirt or
> > higher level management application) must *not* include the assigned device
> > on the qemu commandline, but must instead hotplug the device later after the
> > guest CPUs have been restarted on the destination.
> >
> > So if I'm understanding correctly, the idea is that on the migration source,
> > the device may have been hotplugged, or may have been included when qemu was
> > originally started. Then qemu automatically handles the unplug of the device
> > on the source, but it seems qemu does nothing on the destination, leaving
> > that up to libvirt or a higher layer to implement.
>
> Good point. I don't see why it would not work just as well
> with device present straight away.

Will the guest get properly notified about the device if it had been 
unplugged (from the guest POV) prior to the migration (so the last thing 
the guest knows is that there is no device), but is suddenly/magically 
back in place from the instant the CPUs start on the destination? 
Doesn't there need to be some sort of notification sent from qemu to the 
guest OS to let it know that a "new" device has been plugged in? (I've 
always assumed this was the case, but it's really just guessing on my 
part :-)

It will certainly make things simpler if the device can be present in 
the qemu commandline. If that's the case, then I *think* only item (2) 
below will need solving.

> Did I miss something?
>
> I think Jens was just testing local machine migration
> and of course you can only assign a device to 1 VM at a time.


That's useful and convenient for a smoke test, but doesn't account for 
the (almost 100%) possibility of having a different device address 
(maybe even a different model of device) on source and destination, or 
for the need to unbind/rebind vfio-pci and the host net driver.


> > Then in order for this to work, libvirt (or OpenStack or oVirt or whoever)
> > needs to understand that the device in the libvirt config (it will still be
> > in the libvirt config, since from libvirt's POV it hasn't been unplugged):
> >
> > 1) shouldn't be included in the qemu commandline on the destination,
> >
> > 2) will almost surely need to be replaced with a different device on the
> > destination (since it's almost certain that the destination won't have an
> > available device at the same PCI address)
> >
> > 3) will probably need to be unbinded from the VF net driver (does this need
> > to happen before migration is finished? If we want to lower the probability
> > of a failure after we're already committed to the migration, then I think we
> > must, but libvirt isn't set up for that in any way).
> >
> > 4) will need to be hotplugged after the migration has finished *and* after
> > the guest CPUs have been restarted on the destination.
> >
> >
> > While it will be possible to assure that there is a destination device, and
> > to replace the old device with new in the config (and maybe, either with
> > some major reworking of device assignment code, or offloading the
> > responsibility to the management application(s), possible to re-bind the
> > device to the vfio-pci driver), prior to marking the migration as
> > "successful" (thus committing to running it on the destination), we can't
> > say as much for actually assigning the device. So if the assignment fails,
> > then what happens?
> >
> >
> > So a few issues I see that will need to be solved by [someone] (apparently
> > either libvirt or management):
> >
> > a) there isn't anything in libvirt's XML grammar that allows us to signify a
> > device that is "present in the config but shouldn't be included in the
> > commandline"
> >
> > b) someone will need to replace the device from the source with an
> > equivalent device on the destination in the libvirt XML. There are other
> > cases of management modifying the XML during migration (I think), but this
> > does point out that putting the "auto-unplug code into qemu isn't turning
> > this into a trivial
> >
> > c) there is nothing in libvirt's migration logic that can cause a device to
> > be re-binded to vfio-pci prior to completion of a migration. Unless this is
> > added to libvirt (or the re-bind operation is passed off to the management
> > application), we will need to live with the possibility that hotplugging the
> > device will fail due to failed re-bind *after* we've committed to the
> > migration.
> >
> > d) once the guest CPUs are restarted on the destination, [someone] (libvirt
> > or management) needs to hotplug the new device on the destination. (I'm
> > guessing that a hotplug can only be done while the guest CPUs are running;
> > correct me if this is wrong!)
> >
> > This sounds like a lot of complexity for something that was supposed to be
> > handled completely/transparently by qemu :-P.