Re: [Qemu-devel] [BUG] QEMU crashes with dpdk virtio pmd

[Gonglei (Arei)]

CCing Paolo and Stefan, since it has a relationship with bh in Qemu.

> -----Original Message-----
> From: Jason Wang [mailto:address@hidden
> 
> 
> On 2017年04月25日 19:37, wangyunjian wrote:
> > The q->tx_bh will free in virtio_net_del_queue() function, when remove 
> > virtio
> queues
> > if the guest doesn't support multiqueue. But it might be still referenced by
> others (eg . virtio_net_set_status()),
> > which need so set NULL.
> >
> > diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> > index 7d091c9..98bd683 100644
> > --- a/hw/net/virtio-net.c
> > +++ b/hw/net/virtio-net.c
> > @@ -1522,9 +1522,12 @@ static void virtio_net_del_queue(VirtIONet *n,
> int index)
> >       if (q->tx_timer) {
> >           timer_del(q->tx_timer);
> >           timer_free(q->tx_timer);
> > +        q->tx_timer = NULL;
> >       } else {
> >           qemu_bh_delete(q->tx_bh);
> > +        q->tx_bh = NULL;
> >       }
> > +    q->tx_waiting = 0;
> >       virtio_del_queue(vdev, index * 2 + 1);
> >   }
> 
> Thanks a lot for the fix.
> 
> Two questions:
> 
> - If virtio_net_set_status() is the only function that may access tx_bh,
> it looks like setting tx_waiting to zero is sufficient?

Currently yes, but we don't assure that it works for all scenarios, so
we set the tx_bh and tx_timer to NULL to avoid to possibly access wild pointer,
which is the common method for usage of bh in Qemu.

I have another question about the root cause of this issure.

This below trace is the path of setting tx_waiting to one in 
virtio_net_handle_tx_bh() :

Breakpoint 1, virtio_net_handle_tx_bh (vdev=0x0, vq=0x7f335ad13900) at 
/data/wyj/git/qemu/hw/net/virtio-net.c:1398
1398    {
(gdb) bt
#0  virtio_net_handle_tx_bh (vdev=0x0, vq=0x7f335ad13900) at 
/data/wyj/git/qemu/hw/net/virtio-net.c:1398
#1  0x00007f3357bddf9c in virtio_bus_set_host_notifier (bus=<optimized out>, 
address@hidden, address@hidden) at hw/virtio/virtio-bus.c:297
#2  0x00007f3357a0055d in vhost_dev_disable_notifiers (address@hidden, 
address@hidden) at /data/wyj/git/qemu/hw/virtio/vhost.c:1422
#3  0x00007f33579e3373 in vhost_net_stop_one (net=0x7f335ad84dc0, 
dev=0x7f335c6f5f90) at /data/wyj/git/qemu/hw/net/vhost_net.c:289
#4  0x00007f33579e385b in vhost_net_stop (address@hidden, ncs=<optimized out>, 
address@hidden) at /data/wyj/git/qemu/hw/net/vhost_net.c:367
#5  0x00007f33579e15de in virtio_net_vhost_status (status=<optimized out>, 
n=0x7f335c6f5f90) at /data/wyj/git/qemu/hw/net/virtio-net.c:176
#6  virtio_net_set_status (vdev=0x7f335c6f5f90, status=0 '\000') at 
/data/wyj/git/qemu/hw/net/virtio-net.c:250
#7  0x00007f33579f8dc6 in virtio_set_status (address@hidden, address@hidden 
'\000') at /data/wyj/git/qemu/hw/virtio/virtio.c:1146
#8  0x00007f3357bdd3cc in virtio_ioport_write (val=0, addr=18, 
opaque=0x7f335c6eda80) at hw/virtio/virtio-pci.c:387
#9  virtio_pci_config_write (opaque=0x7f335c6eda80, addr=18, val=0, 
size=<optimized out>) at hw/virtio/virtio-pci.c:511
#10 0x00007f33579b2155 in memory_region_write_accessor (mr=0x7f335c6ee470, 
addr=18, value=<optimized out>, size=1, shift=<optimized out>, mask=<optimized 
out>, attrs=...) at /data/wyj/git/qemu/memory.c:526
#11 0x00007f33579af2e9 in access_with_adjusted_size (address@hidden, 
address@hidden, address@hidden, access_size_min=<optimized out>, 
access_size_max=<optimized out>, address@hidden
    0x7f33579b20f0 <memory_region_write_accessor>, address@hidden, 
address@hidden) at /data/wyj/git/qemu/memory.c:592
#12 0x00007f33579b2e15 in memory_region_dispatch_write (address@hidden, 
address@hidden, data=0, address@hidden, address@hidden) at 
/data/wyj/git/qemu/memory.c:1319
#13 0x00007f335796cd93 in address_space_write_continue (mr=0x7f335c6ee470, l=1, 
addr1=18, len=1, buf=0x7f335773d000 "", attrs=..., addr=49170, 
as=0x7f3358317060 <address_space_io>) at /data/wyj/git/qemu/exec.c:2834
#14 address_space_write (as=<optimized out>, addr=<optimized out>, attrs=..., 
buf=<optimized out>, len=<optimized out>) at /data/wyj/git/qemu/exec.c:2879
#15 0x00007f335796d3ad in address_space_rw (as=<optimized out>, address@hidden, 
attrs=..., address@hidden, buf=<optimized out>, address@hidden, address@hidden) 
at /data/wyj/git/qemu/exec.c:2981
#16 0x00007f33579ae226 in kvm_handle_io (count=1, size=1, direction=<optimized 
out>, data=<optimized out>, attrs=..., port=49170) at 
/data/wyj/git/qemu/kvm-all.c:1803
#17 kvm_cpu_exec (address@hidden) at /data/wyj/git/qemu/kvm-all.c:2032
#18 0x00007f335799b632 in qemu_kvm_cpu_thread_fn (arg=0x7f335ae82070) at 
/data/wyj/git/qemu/cpus.c:1118
#19 0x00007f3352983dc5 in start_thread () from /usr/lib64/libpthread.so.0
#20 0x00007f335113571d in clone () from /usr/lib64/libc.so.6

It calls qemu_bh_schedule(q->tx_bh) at the bottom of virtio_net_handle_tx_bh(),
I don't know why virtio_net_tx_bh() doesn't be invoked, so that the 
q->tx_waiting is not zero.
[ps: we added logs in virtio_net_tx_bh() to verify that]

Some other information: 

It won't crash if we don't use vhost-net.


Thanks,
-Gonglei

> - Can you post a formal patch for this?
> 
> Thanks
> 
> > From: wangyunjian
> > Sent: Monday, April 24, 2017 6:10 PM
> > To: address@hidden; Michael S. Tsirkin <address@hidden>; 'Jason
> Wang' <address@hidden>
> > Cc: wangyunjian <address@hidden>; caihe <address@hidden>
> > Subject: [Qemu-devel][BUG] QEMU crashes with dpdk virtio pmd
> >
> > Qemu crashes, with pre-condition:
> > vm xml config with multiqueue, and the vm's driver virtio-net support
> multi-queue
> >
> > reproduce steps:
> > i. start dpdk testpmd in VM with the virtio nic
> > ii. stop testpmd
> > iii. reboot the VM
> >
> > This commit "f9d6dbf0  remove virtio queues if the guest doesn't support
> multiqueue" is introduced.
> >
> > Qemu version: QEMU emulator version 2.9.50 (v2.9.0-137-g32c7e0a)
> > VM DPDK version:  DPDK-1.6.1
> >
> > Call Trace:
> > #0  0x00007f60881fe5d7 in raise () from /usr/lib64/libc.so.6
> > #1  0x00007f60881ffcc8 in abort () from /usr/lib64/libc.so.6
> > #2  0x00007f608823e2f7 in __libc_message () from /usr/lib64/libc.so.6
> > #3  0x00007f60882456d3 in _int_free () from /usr/lib64/libc.so.6
> > #4  0x00007f608900158f in g_free () from /usr/lib64/libglib-2.0.so.0
> > #5  0x00007f6088fea32c in iter_remove_or_steal () from
> /usr/lib64/libglib-2.0.so.0
> > #6  0x00007f608edc0986 in object_property_del_all (obj=0x7f6091e74800)
> at qom/object.c:410
> > #7  object_finalize (data=0x7f6091e74800) at qom/object.c:467
> > #8  object_unref (address@hidden) at qom/object.c:903
> > #9  0x00007f608eaf1fd3 in phys_section_destroy (mr=0x7f6091e74800) at
> git/qemu/exec.c:1154
> > #10 phys_sections_free (map=0x7f6090b72bb0) at git/qemu/exec.c:1163
> > #11 address_space_dispatch_free (d=0x7f6090b72b90) at
> git/qemu/exec.c:2514
> > #12 0x00007f608ee91ace in call_rcu_thread (opaque=<optimized out>) at
> util/rcu.c:272
> > #13 0x00007f6089b0ddc5 in start_thread () from /usr/lib64/libpthread.so.0
> > #14 0x00007f60882bf71d in clone () from /usr/lib64/libc.so.6
> >
> > Call Trace:
> > #0  0x00007fdccaeb9790 in ?? ()
> > #1  0x00007fdcd82d09fc in object_property_del_all (obj=0x7fdcdb8acf60) at
> qom/object.c:405
> > #2  object_finalize (data=0x7fdcdb8acf60) at qom/object.c:467
> > #3  object_unref (address@hidden) at qom/object.c:903
> > #4  0x00007fdcd8001fd3 in phys_section_destroy (mr=0x7fdcdb8acf60) at
> git/qemu/exec.c:1154
> > #5  phys_sections_free (map=0x7fdcdc86aa00) at git/qemu/exec.c:1163
> > #6  address_space_dispatch_free (d=0x7fdcdc86a9e0) at
> git/qemu/exec.c:2514
> > #7  0x00007fdcd83a1ace in call_rcu_thread (opaque=<optimized out>) at
> util/rcu.c:272
> > #8  0x00007fdcd301ddc5 in start_thread () from /usr/lib64/libpthread.so.0
> > #9  0x00007fdcd17cf71d in clone () from /usr/lib64/libc.so.6
> >
> >