[Qemu-discuss] qemu -> kvm eventually stops responding when VNC is prepped and waiting for a connection

[Brad Barnett]

FYI, Debian Jessie.

# qemu-system-x86_64 --version
QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-12+deb8u6), Copyright (c)
2003-2008 Fabrice Bellard

Through a variety of scripts and what not, I boot up multiple KVM
instances on a variety of boxes.

Over the 6 months, I've noticed that randomly KVM instances will stop
responding.  They aren't using CPU.  Disk I/O seems non-existent.

Yet, they do not respond to ping, or any network traffic.  Further, they
seem to 'slowly' degrade.  It typically always starts with a variety of:

Feb 24 13:54:36 xxx kernel: [111360.168074] INFO: task kworker/u16:0:6 blocked 
for more than 120 seconds.
Feb 24 13:54:36 xxx kernel: [111360.168179]       Not tainted 3.16.0-4-amd64 #1
Feb 24 13:54:36 xxx kernel: [111360.168244] "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 24 13:54:36 xxx kernel: [111360.168360] kworker/u16:0   D ffff880236d504a8  
   0     6      2 0x00000000
Feb 24 13:54:36 xxx kernel: [111360.168371] Workqueue: writeback 
bdi_writeback_workfn (flush-254:0)

The task could be a variety of things, but mostly seem to revolve around
core kernel processes, or disk.

For this reason, I've been poking at the wrong side of the tracks.  And,
debugging disk i/o... even though these boxes are not heavily disk i/o
laden.

However, recently I discovered that the instant I connect to VNC?

The box immediately and instantly recovers.

I've now verified this three times over the last month (it can take weeks
for a box to get into this state.)  In each case, the symptoms are identical.

The box slows down, eventually starts to have issues as above, then
eventually stops responding entirely.  Connect with VNC?  Instant recovery.

It would seem that there is some sort of blocking going on, when VNC is
up and running in my case... waiting for a connection.

It should be noted that these boxes are prepped with a permanent VNC
password, and ready to go.  I am not running a test run where qemu is
prepped with the VNC command line, but no password/etc.

Current command line (which causes this issue):

qemu-system-x86_64 -enable-kvm -net nic,macaddr=xx:xx:xx:xx:xx:xx -net
vde,sock=/var/run/vde2/mytap.ctl -m 2G -smp 2 -name xxxxxx -drive
file=/xxx/xxx.raw,if=virtio,boot=on -D /xx/log/xxxxxxxx.logfile
-pidfile /run/xxxxxx.pid -monitor stdio
-display vnc=127.0.0.1:2,password -vga qxl -qmp
unix:/xxxx/xxx/xxx.sock,server,nowait

Each qemu instance sits in its own screen instance.  Once qemu starts,
the following script runs.  The 'stuff' screen command essentially sends
commands to screens console/stdin.

Yes, I know I can use the socket, but I want qemu/kvm + screen running
anyhow in this way.  So, the scripts take advantage of that.. and address
qemu/kvm's console directly.

#!/bin/bash

end=`echo -ne '\015'`;
( sleep 10
screen -S screennameofinstance -X select '0'
screen -S screennameofinstance -X stuff 'change vnc password'$end
screen -S screennameofinstance -X stuff 'PASSWORD'$end
screen -S screennameofinstance -X stuff 'expire_password vnc never'$end ) &


I have just recently stopped using this script on all boxes.  The command
line remains the same as above, but I am not sending the password
setup/expire command now to each machine's console.


Has anyone heard of this before?  I Googled, no luck..

Thanks