[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
From: |
Grundmann, Christian |
Subject: |
Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc |
Date: |
Tue, 17 Nov 2015 10:36:16 +0000 |
Hi,
@ Can you please use a 'thread apply all bt full' the full gives a little
more info.
gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set
pagination off" -ex "thread apply all bt full"
[New LWP 52281]
[New LWP 52288]
[New LWP 52286]
[New LWP 52291]
[New LWP 52292]
[New LWP 52287]
[New LWP 52293]
[New LWP 52290]
[New LWP 56455]
[New LWP 52289]
[New LWP 52282]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine
rhel6.5.0,accel=kvm,us'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
Thread 11 (Thread 0x7f6d47719700 (LWP 52282)):
#0 0x00007f6d4f8f2949 in syscall () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d57426272 in qemu_event_wait ()
No symbol table info available.
#2 0x00007f6d57434526 in call_rcu_thread ()
No symbol table info available.
#3 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)):
#0 0x00007f6d55cf08a0 in sem_timedwait () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007f6d574260c7 in qemu_sem_timedwait ()
No symbol table info available.
#2 0x00007f6d573a98ac in worker_thread ()
No symbol table info available.
#3 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 8 (Thread 0x7f6d44713700 (LWP 52290)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 5 (Thread 0x7f6d43711700 (LWP 52292)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 3 (Thread 0x7f6d46717700 (LWP 52286)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 2 (Thread 0x7f6d45715700 (LWP 52288)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)):
#0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
No symbol table info available.
#1 0x00007f6d572a4489 in malloc_and_trace ()
No symbol table info available.
#2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0
No symbol table info available.
#3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0
No symbol table info available.
#4 0x00007f6d57204ffd in virtio_blk_handle_output ()
No symbol table info available.
#5 0x00007f6d573b76b6 in qemu_iohandler_poll ()
No symbol table info available.
#6 0x00007f6d573b7296 in main_loop_wait ()
No symbol table info available.
#7 0x00007f6d571b6a4e in main ()
No symbol table info available.
@ Also, if you've not already got it installed can you please install the
debuginfo package for qemu, it gives a lot more information in backtraces.
Sorry it's a ovirt-node System where I can't you yum
@ Does this part always look the same in your backtraces?
The most are the same, found one a little bit different :
Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)):
#0 0x00007f3785d18353 in
tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
unsigned long, int) () from /lib64/libtcmalloc.so.4
No symbol table info available.
#1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from
/lib64/libtcmalloc.so.4
No symbol table info available.
#2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4
No symbol table info available.
#3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0
No symbol table info available.
#4 0x00007f37885fec89 in g_slice_free1 () from /lib64/libglib-2.0.so.0
No symbol table info available.
#5 0x00007f378a1f232e in virtio_blk_rw_complete ()
No symbol table info available.
#6 0x00007f378a39f1ae in bdrv_co_em_bh ()
No symbol table info available.
#7 0x00007f378a398394 in aio_bh_poll ()
No symbol table info available.
#8 0x00007f378a3a7409 in aio_dispatch_clients ()
No symbol table info available.
#9 0x00007f378a39820e in aio_ctx_dispatch ()
No symbol table info available.
#10 0x00007f37885e299a in g_main_context_dispatch () from
/lib64/libglib-2.0.so.0
No symbol table info available.
#11 0x00007f378a3a6288 in main_loop_wait ()
No symbol table info available.
#12 0x00007f378a1a5a4e in main ()
No symbol table info available.
@ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
No nothing abnormal
@ 2) Did you hit any IO errors and need to tell the VM to continue after a
problem?
Ovirt tells me "no Storage space error". Which is something like the disk is
growing to fast i think. I use Snapshots so on heavy write the disk has to grow
a lot.
Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays
offline.
@ 3) If this is pretty repeatable, then it would be interesting to try changing
to a different
disk emulation and see if the problem goes away - e.g. virtio-scsi would
be a good one to try.
Ok will try that and report
Thx Christian
-----Ursprüngliche Nachricht-----
Von: Dr. David Alan Gilbert [mailto:address@hidden
Gesendet: Dienstag, 17. November 2015 10:59
An: Grundmann, Christian <address@hidden>
Cc: 'address@hidden' <address@hidden>; address@hidden
Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
* Grundmann, Christian (address@hidden) wrote:
> Hi,
> Dan sent me over to you,
> please let me know if i can provide additional informations
Hi Christian,
Thanks for reporting this,
> Softwareversions:
> ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
>
> qemu-img-ev-2.3.0-29.1.el7.x86_64
> qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> kernel-3.10.0-229.14.1.el7.x86_64
> gperftools-libs-2.4-7.el7.x86_64
>
> Commandline:
> /usr/libexec/qemu-kvm -name myvmname -S -machine
> rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime mlock=off
> -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios
> type=1,manufacturer=oVirt,product=oVirt
> Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-5A
> 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319
> -nographic -no-user-config -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,serv
> er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc
> base=2015-11-15T20:04:35,driftfix=slew -global
> kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on
> -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
> virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
> virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
> -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
> -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> -drive
> file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84-8
> 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffaecf
> /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,fo
> rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,werr
> or=stop,rerror=stop,aio=native -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id
> =virtio-disk0,bootindex=1 -netdev
> tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci.0
> ,addr=0x3 -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a
> 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=chann
> el0,name=com.redhat.rhevm.vdsm -chardev
> socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a
> 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait -device
> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=chann
> el1,name=org.qemu.guest_agent.0 -device
> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
>
> Stack Trace:
>
> gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set
> pagination off" -ex "thread apply all bt"
Can you please use a 'thread apply all bt full' the full gives a little more
info.
Also, if you've not already got it installed can you please install the
debuginfo package for qemu, it gives a lot more information in backtraces.
> Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
> #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4
> #1 0x00007fa8b186b489 in malloc_and_trace ()
> #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
> #3 0x00007fa8afbd666e in g_slice_alloc () from
> /lib64/libglib-2.0.so.0
> #4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
> #5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
> #6 0x00007fa8b197e296 in main_loop_wait ()
> #7 0x00007fa8b177da4e in main ()
Does this part always look the same in your backtraces?
The segfault in tc_malloc is probably due to a heap corruption, or double free
or similar - although it can be a bit tricky to find out what did it, since the
corruption might have happened a bit before the place it crashed.
Some other ideas:
1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
2) Did you hit any IO errors and need to tell the VM to continue after a
problem?
3) If this is pretty repeatable, then it would be interesting to try changing
to a different
disk emulation and see if the problem goes away - e.g. virtio-scsi would
be a good one to try.
Dave
>
>
> Thx Christian
>
> -----Ursprüngliche Nachricht-----
> Von: Dan Kenigsberg [mailto:address@hidden
> Gesendet: Freitag, 13. November 2015 20:00
> An: Grundmann, Christian <address@hidden>
> Cc: 'address@hidden' <address@hidden>
> Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
>
> On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> > Hi,
> > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso"
> > (is there something better to use?) fort he nodes, and have random
> > crashes of VMs The dumps are always the Same
> >
> > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump [Thread
> > debugging using libthread_db enabled] Using host libthread_db
> > library "/lib64/libthread_db.so.1".
> > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine
> > rhel6.5.0,accel=kvm,usb=o'.
> > Program terminated with signal 11, Segmentation fault.
> > #0 0x00007f0c559c4353 in
> > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::
> > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> >
> >
> > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist
> > centos7 or 3.6
>
> Due to the low-leveled-ness of the problem, I'd guess it's a
> qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
>
> Please report the precise version of qemu,kernel,libvirt and gperftools-libs
> to qemu-devel mailing list and the complete stack trace and qemu command
> line, if possible.
>
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK
- [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Grundmann, Christian, 2015/11/16
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Dr. David Alan Gilbert, 2015/11/17
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc,
Grundmann, Christian <=
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Dr. David Alan Gilbert, 2015/11/17
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Grundmann, Christian, 2015/11/17
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Grundmann, Christian, 2015/11/17
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Dr. David Alan Gilbert, 2015/11/17
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Grundmann, Christian, 2015/11/19
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Paolo Bonzini, 2015/11/19
- Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc, Dr. David Alan Gilbert, 2015/11/20