Re: Deadlock between bdrv_drain_all_begin and prepare_mmio_access

[Kevin Wolf]

Am 24.07.2022 um 23:41 hat Liang Yan geschrieben:
> Hello All,
> 
> I am facing a lock situation between main-loop thread 1 and vcpu thread 4
> when doing a qmp snapshot. QEMU is running on 6.0.x, checked the upstream
> code and did not see any big change since between. Guest is a Windows 10 VM.
> Unfortunately, I could not get into the windows vm or reproduce the issue by
> myself. No iothread is used here, native aio only.
> 
> From the code,
> 
> -> AIO_WAIT_WHILE(NULL, bdrv_drain_all_poll());
> 
> --> aio_poll(qemu_get_aio_context(), true);
> 
> Mainloop mutex is locked when start snapshot in thread 1, vcpu released
> thread lock when address_space_rw and try to get thread lock again in
> prepare_mmio_access.
> 
> It seems main loop thread is stuck at aio_poll with blocking, but I can not
> figure out what the addr=4275044592 belongs to from mmio read.
> 
> I do not quite understand what really happens here, either block jobs never
> drained out or maybe a block io read from vcpu and cause a deadlock? I hope
> domain experts here could help figure out the root cause, thanks in advance
> and let me know if need any further information.

This does not look like a deadlock to me: Thread 4 is indeed waiting for
thread 1 to release the lock, but I don't think thread 1 is waiting in
any way for thread 4.

In thread 1, bdrv_drain_all_begin() waits for all in-flight I/O requests
to complete. So it looks a bit like some I/O request got stuck. If you
want to debug this a bit further, try to check what it is that makes
bdrv_drain_poll() still return true.

Please also add the QEMU command line you're using, especially the
configuration of the block device backends (for example, does this use
Linux AIO, the thread pool or io_uring?).

Kevin