Re: [Qemu-devel] [libvirt] [RFC 0/2] Attempt to implement the standby feature for assigned network devices

[Daniel P . Berrangé]

On Wed, Dec 05, 2018 at 06:09:16PM +0100, Peter Krempa wrote:
> On Wed, Dec 05, 2018 at 10:18:29 -0600, Michael Roth wrote:
> > Quoting Sameeh Jubran (2018-10-25 13:01:10)
> > > On Thu, Oct 25, 2018 at 5:06 PM Sameeh Jubran <address@hidden> wrote:
> > > > From: Sameeh Jubran <address@hidden>
> > > > Migration support:
> > > >
> > > > Pre migration or during setup phase of the migration we should send an
> > > > unplug request to the guest to unplug the primary device. I haven't had
> > > > the chance to implement that part yet but should do soon. Do you know
> > > > what's the best approach to do so? I wanted to have a callback to the
> > > > virtio-net device which tries to send an unplug request to the guest and
> > > > if succeeds then the migration continues. It needs to handle the case 
> > > > where
> > > > the migration fails and then it has to replug the primary device back.
> > > I think that the "add_migration_state_change_notifier" API call can be 
> > > used
> > > from within the virtio-net device to achieve this, what do you think?
> > 
> > I think it would be good to hear from the libvirt folks (on Cc:) on this as
> > having QEMU unplug a device without libvirt's involvement seems like it
> > could cause issues. Personally I think it seems cleaner to just have QEMU
> > handle the 'hidden' aspects of the device and leave it to QMP/libvirt to do
> > the unplug beforehand. On the libvirt side I could imagine adding an option
> > like virsh migrate --switch-to-standby-networking or something along
> > that line to do it automatically (if we decide doing it automatically is
> > even needed on that end).
> 
> I remember talking about this approach some time ago.
> 
> In general the migration itself is a very complex process which has too
> many places where it can fail. The same applies to device hotunplug.
> This series proposes to merge those two together into an even more
> complex behemoth.
> 
> Few scenarios which don't have clear solution come into my mind:
> - Since unplug request time is actually unbounded. The guest OS may
>   arbitrarily reject it or execute it at any later time, migration may get
>   stuck in a halfway state without any clear rollback or failure scenario.

IMHO this is the really big deal. Doing this inside QEMU can arbitrarily
delay the start of migration, but this is opaque to mgmt apps because it
all becomes hidden behind the migrate command. It is common for mgmt apps
to serialize migration operations, otherwise they compete for limited
network bandwidth making it less likely that any will complete. If we're
waiting for a guest OS to do the unplug though, we don't want to be
stopping other migrations from being started in the mean time. Having
the unplugs done from the mgmt app explicitly gives it the flexibility
to decide how to order and serialize things to suit its needs.

> - After migration, device hotplug may fail for whatever reason, leaving
>   networking crippled and again no clear single-case rollback scenario.

I'd say s/crippled/degraded/. Anyway depending on the the reason that
triggered the migration, you may not even want to rollback to the source
host, despite the VFIO hotplug failing on the target.

If the original host was being evacuated in order to upgrade it to the latest
security patches, or due to hardware problems, it can be preferrable to just
let the VM start running on the target host with just emulated NICs only and
worry about getting working VFIO later.

> Then there's stuff which requires libvirt/management cooperation
> - picking of the network device on destination
> - making sure that the device is present etc.
> 
> From managements point of view, bundling all this together is really not
> a good idea since it creates a very big matrix of failure scenarios. In
> general even libvirt will prefer that upper layer management drives this
> externally, since any rolback scenario will result in a policy decision
> of what to do in certain cases, and what timeouts to pick.

Indeed, leaving these policies decisions to the mgmt app has been a
better approach in general over time, as the view of what's the best
way to approach a problem has changed over time.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|