Re: [Qemu-discuss] need your help for some crash trouble of qemu-system-x86_64

[Zhenhao Zhang]

hello Wei & Fam,

    Thanks very much for your reply。

在 2016/12/21 12:24, wyang 写道:
> On 2016年12月21日 07:57, Fam Zheng wrote:
> > On Tue, 12/20 18:29, Zhenhao Zhang wrote:
> > > hello everyone,
> > Hi Zhenhao, thanks for the report.
> >
> > >      I am sorry to trouble you. but, I meet some trouble when using 
> > > qemu+kvm
> > > +libvirt.  The qemu-system-x86_64 crash and get some errer info from 
> > > logs
> > > :./var/log/libvirt/qemu/instance-000006d3.log, as bellow,
> > >
> > > 2016-05-25 10:28:52.565+0000: starting up libvirt version: 1.2.16, 
> > > package:
> > > 1.2.16-2ubuntu11.15.10.4~cloud0, qemu version: 2.3.0 (Debian
> > > 1:2.3+dfsg-5ubuntu9.2~cloud0)
> > > LC_ALL=C 
> > > PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> > > QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name 
> > > instance-000006d3 -S
> > > -machine pc-i440fx-vivid,accel=kvm,usb=off -cpu 
> > > Haswell-noTSX,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
> > > -m 8192 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
> > > 5578b710-3518-44c3-91fe-e2e601f569f0 -smbios 
> > > type=1,manufacturer=OpenStack
> > > Foundation,product=OpenStack 
> > > Nova,version=12.0.2,serial=00000000-0000-0000-0000-0cc47a555870,uuid=5578b710-3518-44c3-91fe-e2e601f569f0,family=Virtual
> > > Machine -no-user-config -nodefaults -chardev 
> > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000006d3.monitor,server,nowait
> > > -mon chardev=charmonitor,id=monitor,mode=control -rtc 
> > > base=utc,driftfix=slew
> > > -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
> > > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 
> > > -drive 
> > > file=rbd:vms/5578b710-3518-44c3-91fe-e2e601f569f0_disk:id=cinder:key=AQBrR6tWbFwlOhAAmtmc/eQX8/K6OjQugVx+mg==:auth_supported=cephx\;none:mon_host=192.168.200.1\:6789\;192.168.200.2\:6789\;192.168.200.3\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback
> > > -device 
> > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> > > -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 -device 
> > > virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:63:24:27,bus=pci.0,addr=0x3
> > > -chardev 
> > > file,id=charserial0,path=/var/lib/nova/instances/5578b710-3518-44c3-91fe-e2e601f569f0/console.log
> > > -device isa-serial,chardev=charserial0,id=serial0 -chardev
> > > pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 
> > > -device
> > > usb-tablet,id=input0 -vnc 0.0.0.0:1 -k en-us -device
> > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
> > > char device redirected to /dev/pts/3 (label charserial1)
> > > *** Error in `/usr/bin/qemu-system-x86_64': double free or 
> > > corruption (out):
> > > 0x00007f83bc0cc1d0 ***
> > > 2016-06-28 22:36:16.361+0000: shutting down
> > >
> > >    not all vms have crashed and the creashed vms  images incloud : 
> > > centos6.5
> > > ubuntu14.04 and ubuntu16.04.  Maybe it not caused by the guest images,I
> > > think.
> > Yes, a double free or corruption is very likely a bug of QEMU, not 
> > the guest.
> >
> > Can you post a backtrace of the crash?
>
> Zhenhao,  its better to post backtrace after installing debug symbols 
> for package qemu.
 I am so sorry to said that, I have not installed this for qemu . 
So,all the things I can get is here. I will install it and Thanks again 
for your reminder. Could you give me  more suggestions to help get its 
root cause ,when it is happend again.
> > What operations are being performed when it crashes?

I am not the owner of the vms, so I can not get more info about this. I 
have try to ask  owners,  one person say that only ftp service is 
running on his vm  and no one use it.  his OS is ubuntu14.04LTS Server. 
the other way, this VM only start up and running long time.
> I'm using qemu 2.3.0, so I'm interest in how to reproduce it, i will 
> try it.
I have 20 host with same hardware and the same software version of 
OS,KVM,QEMU and Libvirt . more than 200 vms, only 10 vms meet this 
trouble on 5 host.  So I don't know how to reproduce it.
> Thanks
> Wei
> >   Can you reproduce it with
> > QEMU 2.8.0?
> >
> > Fam
May be I can update one host to use new QEMU version , but, I don't 
knonw when this trouble will happend again and how to catch hold of it.
> > > *I don't have any idear about this, any tips are grateful.*
> > >
> > > my environment info as:
> > >
> > > *OS: *
> > >
> > >          # uname -a
> > >          Linux VM154S019059 4.2.0-35-generic #40~14.04.1-Ubuntu SMP 
> > > Fri Mar
> > > 18 16:37:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> > >
> > >          # cat /etc/lsb-release
> > >              DISTRIB_ID=Ubuntu
> > >              DISTRIB_RELEASE=14.04
> > >              DISTRIB_CODENAME=trusty
> > >              DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"
> > >
> > > *QEMU INFO：*
> > >
> > > dpkg -l|grep qemu
> > > ii  ipxe-qemu 1.0.0+git-20131111.c3d1e78-2ubuntu1.1 all          PXE 
> > > boot
> > > firmware - ROM images for qemu
> > > ii  qemu-block-extra:amd64 1:2.3+dfsg-5ubuntu9.2~cloud0          amd64
> > > extra block backend modules for qemu-system and qemu-utils
> > > ii  qemu-slof 20140630+dfsg-1ubuntu1~14.04 all          Slimline
> > > Open Firmware -- QEMU PowerPC version
> > > ii  qemu-system 1:2.3+dfsg-5ubuntu9.2~cloud0 amd64        QEMU full
> > > system emulation binaries
> > > ii  qemu-system-arm 1:2.3+dfsg-5ubuntu9.2~cloud0 amd64        QEMU
> > > full system emulation binaries (arm)
> > > ii  qemu-system-common 1:2.3+dfsg-5ubuntu9.2~cloud0 amd64
> > > QEMU full system emulation binaries (common files)
> > > ii  qemu-system-mips 1:2.3+dfsg-5ubuntu9.2~cloud0 amd64        QEMU
> > > full system emulation binaries (mips)
> > > ii  qemu-system-misc 1:2.3+dfsg-5ubuntu9.2~cloud0 amd64        QEMU
> > > full system emulation binaries (miscelaneous)
> > > ii  qemu-system-ppc 1:2.3+dfsg-5ubuntu9.2~cloud0 amd64        QEMU
> > > full system emulation binaries (ppc)
> > > ii  qemu-system-sparc 1:2.3+dfsg-5ubuntu9.2~cloud0 amd64
> > > QEMU full system emulation binaries (sparc)
> > > ii  qemu-system-x86 1:2.3+dfsg-5ubuntu9.2~cloud0 amd64        QEMU
> > > full system emulation binaries (x86)
> > > ii  qemu-utils 1:2.3+dfsg-5ubuntu9.2~cloud0 amd64        QEMU
> > > utilities
> > > *
> > > *
> > >
> > > *
> > > *
> > >
> > > *kvm model:*
> > >
> > > # modinfo kvm
> > > filename: /lib/modules/4.2.0-35-generic/kernel/arch/x86/kvm/kvm.ko
> > > license:        GPL
> > > author:         Qumranet
> > > srcversion:     54A62F24A6A2743053CCCC6
> > > depends:
> > > intree:         Y
> > > vermagic:       4.2.0-35-generic SMP mod_unload modversions
> > > signer:         Build time autogenerated kernel key
> > > sig_key: 78:A2:B1:7E:20:A3:96:76:A3:5E:2C:3E:16:0E:70:1F:99:E0:65:9B
> > > sig_hashalgo:   sha512
> > > parm:           allow_unsafe_assigned_interrupts:Enable device 
> > > assignment on
> > > platforms without interrupt remapping support. (bool)
> > > parm:           ignore_msrs:bool
> > > parm:           min_timer_period_us:uint
> > > parm:           kvmclock_periodic_sync:bool
> > > parm:           tsc_tolerance_ppm:uint
> > > parm:           lapic_timer_advance_ns:uint
> > > parm:           halt_poll_ns:uint
> > >
> > > # modinfo kvm-amd
> > > filename: /lib/modules/4.2.0-35-generic/kernel/arch/x86/kvm/kvm-amd.ko
> > > license:        GPL
> > > author:         Qumranet
> > > srcversion:     BBB11CD7C18B7875B850EC4
> > > alias:          cpu:type:x86,ven*fam*mod*:feature:*00C2*
> > > depends:        kvm
> > > intree:         Y
> > > vermagic:       4.2.0-35-generic SMP mod_unload modversions
> > > signer:         Build time autogenerated kernel key
> > > sig_key: 78:A2:B1:7E:20:A3:96:76:A3:5E:2C:3E:16:0E:70:1F:99:E0:65:9B
> > > sig_hashalgo:   sha512
> > > parm:           npt:int
> > > parm:           nested:int
> > >
> > > *libvirt*
> > >
> > >         # libvirtd --version
> > >         libvirtd (libvirt) 1.2.16
> > >
> > > Best Regards,
> > >
> > > Zhang Zhenhao
Thanks again.

Best Regards,
Zhang Zhenhao