[Qemu-devel] Re: [PATCH] net: delay peer host device delete

[Michael S. Tsirkin]

On Mon, Sep 20, 2010 at 01:14:12PM -0500, Anthony Liguori wrote:
> On 09/20/2010 12:14 PM, Michael S. Tsirkin wrote:
> >On Mon, Sep 20, 2010 at 11:56:56AM -0500, Anthony Liguori wrote:
> >>On 09/20/2010 11:47 AM, Michael S. Tsirkin wrote:
> >>>On Mon, Sep 20, 2010 at 11:41:45AM -0500, Anthony Liguori wrote:
> >>>>On 09/20/2010 11:30 AM, Michael S. Tsirkin wrote:
> >>>>>With -netdev, virtio devices present offload
> >>>>>features to guest, depending on the backend used.
> >>>>>Thus, removing host ntedev peer while guest is
> >>>>>active leads to guest-visible inconsistency and/or crashes.
> >>>>>See e.g. https://bugzilla.redhat.com/show_bug.cgi?id=623735
> >>>>>
> >>>>>As a solution, while guest (NIC) peer device exists,
> >>>>>we must prevent the host peer from being deleted.
> >>>>>
> >>>>>This patch does this by adding peer_deleted flag in nic state:
> >>>>>if host device is going away while guest device
> >>>>>is around, set this flag and keep host device around
> >>>>>for as long as guest device exists.
> >>>>Having an unclear life cycle really worries me.
> >>>>
> >>>>Wouldn't the more correct solution be to avoid removing the netdev
> >>>>device until after the peer has successfully been removed?
> >>>>
> >>>>Regards,
> >>>>
> >>>>Anthony Liguori
> >>>This is exactly what the patch does.
> >>At the management layer instead of doing it magically in the backend.
> >The amount of pain this inflicts on management would be considerable.
> >Hotplug commands were designed to be asynchronous
> >(starts the process, does not wait for it to complete), maybe that
> >was a mistake but we can not change semantics at will now.
> >
> >Add new commands, okay, but existing ones should work and get fixed
> >if there's a bug.
> 
> But having commands that are impossible to use correctly is not very good.

So we will have to fix the existing commands so they can be used
correctly. Since the device is removed from the list
shown to the monitor, I do not really see why the user
cares that the backend is actually still around
until the device is removed.

> Here's what makes sense to me:
> 
> 1) async device remove + poll device status/removal notification +
> remove backend
> 
> The management tool needs to determine when the device is gone and
> remove the backend.
> 
> 2) sync device remove + remove backend
> 
> Command does not return until device is removed
> 
> 3) async device and backend removal + poll device/backend removal +
> removal notification
> 
> One command that removes the device and any associated backend.  We
> need to indicate to the management layer when this operation is
> complete.
> 
> I think (2) is the most elegant but also the most difficult to
> implement today.  I think (1) is the least invasive to implement but
> has the most management tool complexity.  (3) is probably the best
> compromise in terms of complexity and ease of implementation.
> 
> Just for comparison, your patch does:
> 
> 4) async device removal + remove backend
> 
> Whereas remove backend may or may not cause removal depending on
> whether device removal has happened.  So it's really async removal
> but it doesn't happen deterministically on it's own.  What happens
> if you call remove backend before starting async device removal?

It won't be removed until device is removed.

> What if the guest never removes the device?

Not really different from guest never reacting to nic hotplug.
If you want to fix this, we'll need a "force" flag to delete.

>  What if a reset
> happens?

I think reset will complete the hotplug.  If it does not we need to fix
it anyway.

> One advantage of (1) is that there is no tricky life cycle
> considerations.  If we did (3), we would have to think through what
> happens if a guest doesn't respond to an unplug request.
> 
> Regards,
> 
> Anthony Liguori

All very well, but this ignores the issue:

We have told management that a way to remove a frontend backend pair is
by giving two commands.  Management has implemented this. Now we need to
have qemu do the right thing.




> >>IOW, if device_del returns and the device isn't actually deleted,
> >>that's a bug and addressing it like this just means we'll trip over
> >>it somewhere else.
> >>
> >>We'll have the same problem with drive_del.
> >Let's fix it there as well then.
> >
> >>Regards,
> >>
> >>Anthony Liguori
> >>
> >>>>>Signed-off-by: Michael S. Tsirkin<address@hidden>
> >>>>>---
> >>>>>  net.c |   21 ++++++++++++++++++++-
> >>>>>  net.h |    1 +
> >>>>>  2 files changed, 21 insertions(+), 1 deletions(-)
> >>>>>
> >>>>>diff --git a/net.c b/net.c
> >>>>>index 3d0fde7..10855d1 100644
> >>>>>--- a/net.c
> >>>>>+++ b/net.c
> >>>>>@@ -286,12 +286,31 @@ void qemu_del_vlan_client(VLANClientState *vc)
> >>>>>      if (vc->vlan) {
> >>>>>          QTAILQ_REMOVE(&vc->vlan->clients, vc, next);
> >>>>>      } else {
> >>>>>+        /* Even if client will not be deleted yet, remove it from list 
> >>>>>so it
> >>>>>+         * does not appear in monitor.  */
> >>>>>+        QTAILQ_REMOVE(&non_vlan_clients, vc, next);
> >>>>>+        /* Detect that guest-visible (NIC) peer is active, and delay 
> >>>>>deletion.
> >>>>>+         * */
> >>>>>+        if (vc->peer&&    vc->peer->info->type == NET_CLIENT_TYPE_NIC) {
> >>>>>+            NICState *nic = DO_UPCAST(NICState, nc, vc->peer);
> >>>>>+            assert(!nic->peer_deleted);
> >>>>>+            nic->peer_deleted = true;
> >>>>>+            return;
> >>>>>+        }
> >>>>>          if (vc->send_queue) {
> >>>>>              qemu_del_net_queue(vc->send_queue);
> >>>>>          }
> >>>>>-        QTAILQ_REMOVE(&non_vlan_clients, vc, next);
> >>>>>          if (vc->peer) {
> >>>>>              vc->peer->peer = NULL;
> >>>>>+            /* If this is a guest-visible (NIC) device,
> >>>>>+             * and peer has already been removed from monitor,
> >>>>>+             * delete it here. */
> >>>>>+            if (vc->info->type == NET_CLIENT_TYPE_NIC) {
> >>>>>+                NICState *nic = DO_UPCAST(NICState, nc, vc);
> >>>>>+                if (nic->peer_deleted) {
> >>>>>+                    qemu_del_vlan_client(vc->peer);
> >>>>>+                }
> >>>>>+            }
> >>>>>          }
> >>>>>      }
> >>>>>
> >>>>>diff --git a/net.h b/net.h
> >>>>>index 518cf9c..44c31a9 100644
> >>>>>--- a/net.h
> >>>>>+++ b/net.h
> >>>>>@@ -72,6 +72,7 @@ typedef struct NICState {
> >>>>>      VLANClientState nc;
> >>>>>      NICConf *conf;
> >>>>>      void *opaque;
> >>>>>+    bool peer_deleted;
> >>>>>  } NICState;
> >>>>>
> >>>>>  struct VLANState {
>