Re: [Qemu-discuss] Fwd: qemu VM cannot be killed

[Michael S. Tsirkin]

Yes - I suspect a packet is stuck somewhere in networking stack.
This is what vhost is waiting for.

Yes, host reboot is the only way out.

RHEL disables zero copy tx in vhost to avoid these issues.

On Wed, Nov 15, 2017 at 02:55:32PM +0100, Lukáš Kubín wrote:
> CC-ing Michael and Jason as I was suggested in OFTC:#virt forum. Thanks!
> 
> ---------- Forwarded message ----------
> From: Lukáš Kubín <address@hidden>
> Date: Wed, Nov 15, 2017 at 1:39 PM
> Subject: qemu VM cannot be killed
> To: address@hidden
> 
> 
> Hi, we've experienced an issue with kvm instance which got stuck at reboot.
> It's an OpenStack environment, with OpenContrail networking (vrouter agent
> running on host), Ubuntu 16.04.
> 
> Machine was first called to reboot from guest OS by user, had issues with NFS
> unmount during that, user sent a hard-reboot call from OpenStack again then.
> Then we (platform operator) got involved, tried to "virsh destroy" it with 
> this
> output:
> 
> 
>     error: Failed to destroy domain instance-00004243
>     error: Failed to terminate process 140529 with SIGKILL: Device or resource
>     busy
> 
> 
> Neither "kill -9" sent to the qemu process helped.
> 
> Good guys at OFTC:#virt have guided me to collect the following traces and ask
> for help here:
> 
> 
>     # cat /proc/140529/wchan
>     vhost_net_ubuf_put_and_wait
> 
>     # cat /proc/140529/stack
>     [<ffffffffc03c2f74>] vhost_net_ubuf_put_and_wait+0x54/0xa0 [vhost_net]
>     [<ffffffffc03c33b4>] vhost_net_ioctl+0x354/0x8a0 [vhost_net]
>     [<ffffffffa26478c1>] do_vfs_ioctl+0xa1/0x5f0
>     [<ffffffffa2647e89>] SyS_ioctl+0x79/0x90
>     [<ffffffffa2c9b7f6>] entry_SYSCALL_64_fastpath+0x1e/0xa8
>     [<ffffffffffffffff>] 0xffffffffffffffff
> 
> 
> The versions we use are:
> 
>   • kernel 4.8.0-41-generic
>   • qemu-kvm 1:2.5+dfsg-5ubuntu10.2~xenial0+contrail1
>   • libvirt-bin 1.3.1-1ubuntu10.1~xenial1+contrail1
> 
> What can be the cause for this error? What can we do in such a situation to
> destroy the VM - is physical server reboot the only option?
> 
> Thanks and greetings,
> 
> Lukáš
>