Re: [PATCH v3] virtio-net: prevent offloads reset on migration

[Dr. David Alan Gilbert]

* Michael S. Tsirkin (address@hidden) wrote:
> On Thu, Oct 10, 2019 at 02:11:15PM +0200, Mikhail Sennikovsky wrote:
> > Currently offloads disabled by guest via the 
> > VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET
> > command are not preserved on VM migration.
> > Instead all offloads reported by guest features (via 
> > VIRTIO_PCI_GUEST_FEATURES)
> > get enabled.
> > What happens is: first the VirtIONet::curr_guest_offloads gets restored
> > and offloads are getting set correctly:
> > 
> >  #0  qemu_set_offload (nc=0x555556a11400, csum=1, tso4=0, tso6=0, ecn=0, 
> > ufo=0) at net/net.c:474
> >  #1  virtio_net_apply_guest_offloads (n=0x555557701ca0) at 
> > hw/net/virtio-net.c:720
> >  #2  virtio_net_post_load_device (opaque=0x555557701ca0, version_id=11) at 
> > hw/net/virtio-net.c:2334
> >  #3  vmstate_load_state (f=0x5555569dc010, vmsd=0x555556577c80 
> > <vmstate_virtio_net_device>, opaque=0x555557701ca0, version_id=11)
> >      at migration/vmstate.c:168
> >  #4  virtio_load (vdev=0x555557701ca0, f=0x5555569dc010, version_id=11) at 
> > hw/virtio/virtio.c:2197
> >  #5  virtio_device_get (f=0x5555569dc010, opaque=0x555557701ca0, size=0, 
> > field=0x55555668cd00 <__compound_literal.5>) at hw/virtio/virtio.c:2036
> >  #6  vmstate_load_state (f=0x5555569dc010, vmsd=0x555556577ce0 
> > <vmstate_virtio_net>, opaque=0x555557701ca0, version_id=11) at 
> > migration/vmstate.c:143
> >  #7  vmstate_load (f=0x5555569dc010, se=0x5555578189e0) at 
> > migration/savevm.c:829
> >  #8  qemu_loadvm_section_start_full (f=0x5555569dc010, mis=0x5555569eee20) 
> > at migration/savevm.c:2211
> >  #9  qemu_loadvm_state_main (f=0x5555569dc010, mis=0x5555569eee20) at 
> > migration/savevm.c:2395
> >  #10 qemu_loadvm_state (f=0x5555569dc010) at migration/savevm.c:2467
> >  #11 process_incoming_migration_co (opaque=0x0) at migration/migration.c:449
> > 
> > However later on the features are getting restored, and offloads get reset 
> > to
> > everything supported by features:
> > 
> >  #0  qemu_set_offload (nc=0x555556a11400, csum=1, tso4=1, tso6=1, ecn=0, 
> > ufo=0) at net/net.c:474
> >  #1  virtio_net_apply_guest_offloads (n=0x555557701ca0) at 
> > hw/net/virtio-net.c:720
> >  #2  virtio_net_set_features (vdev=0x555557701ca0, features=5104441767) at 
> > hw/net/virtio-net.c:773
> >  #3  virtio_set_features_nocheck (vdev=0x555557701ca0, val=5104441767) at 
> > hw/virtio/virtio.c:2052
> >  #4  virtio_load (vdev=0x555557701ca0, f=0x5555569dc010, version_id=11) at 
> > hw/virtio/virtio.c:2220
> >  #5  virtio_device_get (f=0x5555569dc010, opaque=0x555557701ca0, size=0, 
> > field=0x55555668cd00 <__compound_literal.5>) at hw/virtio/virtio.c:2036
> >  #6  vmstate_load_state (f=0x5555569dc010, vmsd=0x555556577ce0 
> > <vmstate_virtio_net>, opaque=0x555557701ca0, version_id=11) at 
> > migration/vmstate.c:143
> >  #7  vmstate_load (f=0x5555569dc010, se=0x5555578189e0) at 
> > migration/savevm.c:829
> >  #8  qemu_loadvm_section_start_full (f=0x5555569dc010, mis=0x5555569eee20) 
> > at migration/savevm.c:2211
> >  #9  qemu_loadvm_state_main (f=0x5555569dc010, mis=0x5555569eee20) at 
> > migration/savevm.c:2395
> >  #10 qemu_loadvm_state (f=0x5555569dc010) at migration/savevm.c:2467
> >  #11 process_incoming_migration_co (opaque=0x0) at migration/migration.c:449
> > 
> > Fix this by making the virtio_net_set_features not reset
> > the curr_guest_offloads in case the VM runstate is RUN_STATE_INMIGRATE
> > 
> > Signed-off-by: Mikhail Sennikovsky <address@hidden>
> 
> I thought more about it and I really think the problem here is
> different: we really should first set features and then apply offloads.
> 
> It is simply dumb that we are calling device specific
> code before initializing generic virtio.
> 
> The logical order of initialization should be:
> 
> init base class
> init derived class
> 
> 
> instead we have it all spread out :(

Yeh; we've hit problems there before.

> 
> What to do about it?
> Could we maybe teach vmstate to record post load hooks
> it needs to run *somewhere*?
> Then we could load device specific state and record post load
> hooks, then load generic state, run generic post load,
> then run device specific post load hooks.
> 
> Dave, what do you think?

I worry about changing all existing virtio devices; who knows
what existing ordering constraints those devices might have on their
post_load's - for example one field in the vmstate might rely on
another post_load having already run and set a flag; that's not
that unusual.

I can imagine having a Notifier* chain; you initialise it at the start
of virtio_load, provide a virtio_load_post_notifier_add(...) and any
post_load that wants delaying can add itself and then you call it
at the right point in virtio_load; but I'd be too scared to change
the ordering of all post_loads of all devices.  You might be
OK to do that for the top level post_load of a virtio device,
because there are only a few to check.

Dave

> 
> 
> > ---
> >  hw/net/virtio-net.c | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)
> > 
> > diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> > index b9e1cd7..49be172 100644
> > --- a/hw/net/virtio-net.c
> > +++ b/hw/net/virtio-net.c
> > @@ -767,7 +767,14 @@ static void virtio_net_set_features(VirtIODevice 
> > *vdev, uint64_t features)
> >      n->rsc6_enabled = virtio_has_feature(features, VIRTIO_NET_F_RSC_EXT) &&
> >          virtio_has_feature(features, VIRTIO_NET_F_GUEST_TSO6);
> >  
> > -    if (n->has_vnet_hdr) {
> > +    /*
> > +     * In case of RUN_STATE_INMIGRATE the virtio_net_set_features
> > +     * is called as part of VM state restore process.
> > +     * At this stage we do not want the curr_guest_offloads to be reset,
> > +     * i.e. want to preserve them in the same state as was set
> > +     * by the guest on the source machine.
> > +     */
> > +    if (n->has_vnet_hdr && !runstate_check(RUN_STATE_INMIGRATE)) {
> >          n->curr_guest_offloads =
> >              virtio_net_guest_offloads_by_features(features);
> >          virtio_net_apply_guest_offloads(n);
> > -- 
> > 2.7.4
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK