[Qemu-discuss] Hang on reboot in FreeBSD guest on Linux KVM host

[John Nielsen]

I am trying to solve a problem with x86_64 FreeBSD virtual machines running on 
a Linux+libvirt+KVM hypervisor. To be honest I'm not sure if the problem is in 
FreeBSD or the hypervisor, but I'm trying to approach it from both directions.

The _second_ time FreeBSD boots in a virtual machine with more than one core, 
the boot hangs just before the kernel would normally bring up the additional 
processors. The VM will boot fine a first time, but running either "shutdown -r 
now" OR "reboot" will lead to a hung second boot. Stopping and starting the 
host qemu-kvm process is the only way to continue.

Interestingly the problem does not manifest itself after a "virsh reset" on the 
host. I can also avoid it by patching the guest kernel to skip the SMP part of 
the shutdown routine.

Clearly something in the normal FreeBSD shutdown code ls leaving the VM in a 
bad state that hinders the next boot, but I haven't been able to identify what 
exactly. Can someone on the list suggest ways to debug this further? Unless 
it's a FreeBSD bug, I'd like to find a solution or workaround that doesn't 
involve modifying the guest OS.

One more thing: the problem only appears on one of two clusters of host 
machines. The hosts within each cluster are identical, and the two clusters are 
_nearly_ identical to each other. All are running the same software, including:
  CentOS 6.5
  kernel 3.12.13 (custom)
  libvirt-1.1.4-2.el6
  qemu-kvm-1.7.0-2.el6
  seabios-1.7.3.1-1.el6

The only substantial difference on the hardware side is the CPU. The hosts 
where the problem occurs use "Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz", while 
the hosts that don't show the problem use the prior revision, "Intel(R) Xeon(R) 
CPU E5-2650 0 @ 2.00GHz".

All ideas appreciated.

Thanks,

JN