Re: test-aio-multithread assertion

[Peter Maydell]

On Mon, 14 Oct 2019 at 11:50, Paolo Bonzini <address@hidden> wrote:
>
> On 14/10/19 12:26, Peter Maydell wrote:
> > On Thu, 5 Sep 2019 at 14:17, Peter Maydell <address@hidden> wrote:
> >> Here's a different intermittent assertion on Netbsd from 
> >> test-aio-multithread:
> >>
> >> MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
> >> tests/test-aio-multithread -m=quick
> >> -k --tap < /dev/null | ./scripts/tap-driver.pl
> >> --test-name="test-aio-multithread"
> >> PASS 1 test-aio-multithread /aio/multi/lifecycle
> >> PASS 2 test-aio-multithread /aio/multi/schedule
> >> assertion "QSLIST_EMPTY(&ctx->scheduled_coroutines)" failed: file
> >> "util/async.c", line 279, function "
> >> aio_ctx_finalize"
> >> ERROR - too few tests run (expected 5, got 2)
> >
> > Just saw this QSLIST_EMPTY(&ctx->scheduled_coroutines) assertion
> > from FreeBSD as well, so it's still around as a definite
> > intermittent.
>
> This could be similar to what Stefan fixed in commit 2362a28ea1, only in
> the test infrastructure.

Here's a gdb backtrace from as assertion failure under NetBSD
when running ./tests/test-aio-multithread -p /aio/multi/mutex/contended
(which is pretty replicable for me); does it look like the same
thing as the commit 2362a28ea1 issue? (Apologies for all the
"optimized out" stuff in the backtraces; I'll try to rerun with
a --enable-debug build.)

Thread 7 (LWP 2):
#0  0x00007e4b44499c0a in ___lwp_park60 () from /usr/lib/libc.so.12
#1  0x00007e4b44809ee3 in pthread_cond_timedwait () from
/usr/lib/libpthread.so.1
#2  0x00000000004c0862 in qemu_futex_wait (val=4294967295, ev=0x760c00
<rcu_call_ready_event>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:364
#3  qemu_event_wait (ev=ev@entry=0x760c00 <rcu_call_ready_event>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:459
#4  0x00000000004ce825 in call_rcu_thread (opaque=opaque@entry=0x0)
    at /home/qemu/qemu-test.0WfUNX/src/util/rcu.c:260
#5  0x00000000004bfd17 in qemu_thread_start (args=0x7e4b47b22c00)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:519
#6  0x00007e4b4480b8b5 in ?? () from /usr/lib/libpthread.so.1
#7  0x00007e4b44481e20 in ?? () from /usr/lib/libc.so.12
#8  0x00010102464c457f in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 6 (LWP 3):
#0  0x00007e4b4443e02a in write () from /usr/lib/libc.so.12
#1  0x00007e4b44807561 in write () from /usr/lib/libpthread.so.1
#2  0x00000000004beae0 in event_notifier_set (e=e@entry=0x7e4b46f01214)
    at /home/qemu/qemu-test.0WfUNX/src/util/event_notifier-posix.c:100
#3  0x00000000004bb66c in aio_notify (ctx=0x7e4b46f01120)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:350
#4  0x00000000004bb936 in aio_co_schedule (ctx=0x7e4b46f01120,
co=0x7e4b46b01280)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:473
#5  0x00000000004cf226 in qemu_co_mutex_wake (mutex=0x75fc00
<comutex>, co=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198
#6  qemu_co_mutex_unlock (mutex=mutex@entry=0x75fc00 <comutex>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301
#7  0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208
#8  0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115
#9  0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12
#10 0x00010102464c457f in ?? ()
#11 0x0000000000000000 in ?? ()

Thread 5 (LWP 4):
#0  0x000000000040b34a in test_multi_co_mutex_entry (opaque=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:205
#1  0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115
#2  0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12
#3  0x00010102464c457f in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 4 (LWP 5):
#0  0x00007e4b4443e02a in write () from /usr/lib/libc.so.12
#1  0x00007e4b44807561 in write () from /usr/lib/libpthread.so.1
#2  0x00000000004beae0 in event_notifier_set (e=e@entry=0x7e4b47701384)
    at /home/qemu/qemu-test.0WfUNX/src/util/event_notifier-posix.c:100
#3  0x00000000004bb66c in aio_notify (ctx=0x7e4b47701290)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:350
#4  0x00000000004bb936 in aio_co_schedule (ctx=0x7e4b47701290,
co=0x7e4b46b01420)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:473
#5  0x00000000004cf226 in qemu_co_mutex_wake (mutex=0x75fc00
<comutex>, co=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198
#6  qemu_co_mutex_unlock (mutex=mutex@entry=0x75fc00 <comutex>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301
#7  0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208
#8  0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115
#9  0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12
#10 0x00010102464c457f in ?? ()
#11 0x0000000000000000 in ?? ()

Thread 3 (LWP 7):
#0  0x00000000004cf200 in qemu_co_mutex_unlock
(mutex=mutex@entry=0x75fc00 <comutex>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:291
#1  0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208
#2  0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115
#3  0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12
#4  0x00010102464c457f in ?? ()
#5  0x0000000000000000 in ?? ()

Thread 2 (LWP 6):
#0  0x00007e4b444fe7ea in _lwp_kill () from /usr/lib/libc.so.12
#1  0x00007e4b444fe092 in abort () from /usr/lib/libc.so.12
#2  0x00007e4b4449fd57 in __assert13 () from /usr/lib/libc.so.12
#3  0x00000000004bb3e0 in aio_ctx_finalize (source=0x7e4b47701290)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:279
#4  0x00007e4b46445596 in g_source_unref_internal () from
/usr/pkg/lib/libglib-2.0.so.0
#5  0x00000000004bb93e in aio_context_unref (ctx=0x7e4b47701290)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:516
#6  aio_co_schedule (ctx=0x7e4b47701290, co=0x7e4b46b01420)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:475
#7  0x00000000004cf226 in qemu_co_mutex_wake (mutex=0x75fc00
<comutex>, co=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198
#8  qemu_co_mutex_unlock (mutex=mutex@entry=0x75fc00 <comutex>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301
#9  0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208
#10 0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115
#11 0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12
#12 0x00010102464c457f in ?? ()
#13 0x0000000000000000 in ?? ()

Thread 1 (LWP 1):
#0  0x00007e4b4443e2ea in _sys___nanosleep50 () from /usr/lib/libc.so.12
#1  0x00007e4b44807287 in __nanosleep50 () from /usr/lib/libpthread.so.1
#2  0x00007e4b4646e511 in g_usleep () from /usr/pkg/lib/libglib-2.0.so.0
#3  0x000000000040b72e in test_multi_co_mutex (threads=<optimized
out>, seconds=1)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:237
#4  0x00007e4b4646bd15 in g_test_run_suite_internal () from
/usr/pkg/lib/libglib-2.0.so.0
#5  0x00007e4b4646bc3e in g_test_run_suite_internal () from
/usr/pkg/lib/libglib-2.0.so.0
#6  0x00007e4b4646bc3e in g_test_run_suite_internal () from
/usr/pkg/lib/libglib-2.0.so.0
#7  0x00007e4b4646bc3e in g_test_run_suite_internal () from
/usr/pkg/lib/libglib-2.0.so.0
#8  0x00007e4b4646c1a0 in g_test_run_suite () from /usr/pkg/lib/libglib-2.0.so.0
#9  0x00007e4b4646c1e8 in g_test_run () from /usr/pkg/lib/libglib-2.0.so.0
#10 0x00000000004dbcdc in main (argc=1, argv=0x7f7fffd572f8)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:459

thanks
-- PMM