Re: [Qemu-devel] Unresponsive linux guest once migrated

[Dr. David Alan Gilbert]

* Marcin Gibu??a (address@hidden) wrote:
> >Can you give:
> >   1) A backtrace from the guest
> >      thread apply all bt full
> >      in gdb
> 
> You mean from gdb attached to hanged guest? I'll try to get it. From
> what I remember it looks rather "normal" - busy executing guest
> code.

yes; if you can send it a sysrq to trigger a backtrace it might also
be worth a try - I'm just trying to find what the guest is really doing
when it's apparentyly 'hung'.


> >   2) What's the earliest/newest qemu versions you've seen this on?
> 
> 1.4 - 1.6
> Don't know about earlier versions because I didn't use migration on
> them. Haven't tried 1.7 yet (I know about XBZRLE fixes, but it
> happened without it as well...).

If you were going to try one thing, I'd prefer you to try the head
of git, i.e.the very latest.

> >   3) What guest OS are you running?
> 
> All flavors of Centos, Ubuntu, Redhat, etc. Also Windows. But never
> seen a crash with Windows so far.
> 
> Seems that few people who also have this issue, reports success with
> kvmclock disabled (either in qemu or kernel command line).

OK.

> >   4) What host OS are you running?
> 
> Distro is Gentoo based (with no crazy compiler options). I've been
> using kernel 3.4 - 3.10.
> 
> >   5) What CPU are you running on?
> 
> AMD Opteron(tm) Processor 6164 HE
> 
> >   6) What does your qemu command line look like?
> 
> Example VM:
> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name
> 3b5e37ea-04be-4a6b-8d63-f1a5853f2138 -S -machine
> pc-i440fx-1.5,accel=kvm,usb=off -cpu 
> qemu64,+misalignsse,+abm,+lahf_lm,+rdtscp,+popcnt,+x2apic,-svm,+kvmclock
> -m 1024 -realtime mlock=on -smp 2,sockets=4,cores=12,threads=1 -uuid
> 3b5e37ea-04be-4a6b-8d63-f1a5853f2138 -smbios type=0,vendor=HAL 9000
> -smbios type=1,manufacturer=cloud -no-user-config -nodefaults
> -chardev 
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/3b5e37ea-04be-4a6b-8d63-f1a5853f2138.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc
> base=utc,clock=vm,driftfix=slew -no-hpet -no-kvm-pit-reinjection
> -no-shutdown -boot menu=off -device
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive 
> file=/dev/stor1c/2e7fd7aa-8588-47ed-a091-af2b81c9e935,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native,bps_rd=57671680,bps_wr=57671680,iops_rd=275,iops_wr=275
> -device 
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
> -drive if=none,id=drive-ide0-0-0,readonly=on,format=raw -device
> ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
> -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=27 -device 
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:11:11:11:11,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -chardev 
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/f16x86_64.agent,server,nowait
> -device 
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
> -device usb-tablet,id=input0 -vnc 0.0.0.0:4,password -vga cirrus
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -sandbox
> on
> 
> I've tried playing with different CPU model (Opteron_G3) and flags,
> it didn't make any difference.
> 
> >   7) How exactly are you migrating?
> 
> Via libvirt live migration. Seen it with and without XBZRLE enabled.
> 
> >   8) You talk about having to wait a few hours to trigger it - do
> >      you have a more exact description of a test?
> 
> Yes, that's where it gets weird. I've never seen this on fresh VM.
> It needs to be idle for couple of hours at least. And even then it
> doesn't always hang.

So your OS is just sitting at a text console, running nothing special?
When you reboot after the migration what's the last thing you see
in the guests logs? Is there anything from after the migration?

> >   9) Is there any output from qemu stderr/stdout in your qemu logs?
> 
> Nothing unusual. From QEMU point of view guest is up and running.
> Only its OS is hanged (but not panicked, there is no backtrace,
> oops or BUG on its screen).

Dave
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK