Re: [PULL v2 00/82] pci,pc,virtio: features, tests, fixes, cleanups

[Stefan Hajnoczi]

On Thu, 3 Nov 2022 at 08:14, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Wed, Nov 02, 2022 at 03:47:43PM -0400, Stefan Hajnoczi wrote:
> > On Wed, Nov 02, 2022 at 12:02:14PM -0400, Michael S. Tsirkin wrote:
> > > Changes from v1:
> > >
> > > Applied and squashed fixes by Igor, Lei He, Hesham Almatary for
> > > bugs that tripped up the pipeline.
> > > Updated expected files for core-count test.
> >
> > Several "make check" CI failures have occurred. They look like they are
> > related. Here is one (see the URLs at the bottom of this email for more
> > details):
> >
> > 17/106 ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child 
> > process 
> > (/arm/virt/virtio-mmio/virtio-bus/virtio-net-device/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
> >  [8609]) failed unexpectedly ERROR
> >  17/106 qemu:qtest+qtest-arm / qtest-arm/qos-test                     ERROR 
> >          31.44s   killed by signal 6 SIGABRT
> > >>> G_TEST_DBUS_DAEMON=/builds/qemu-project/qemu/tests/dbus-vmstate-daemon.sh
> > >>>  MALLOC_PERTURB_=49 QTEST_QEMU_IMG=./qemu-img 
> > >>> QTEST_QEMU_BINARY=./qemu-system-arm 
> > >>> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon 
> > >>> /builds/qemu-project/qemu/build/tests/qtest/qos-test --tap -k
> > ――――――――――――――――――――――――――――――――――――― ✀  
> > ―――――――――――――――――――――――――――――――――――――
> > stderr:
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev 
> > socket,id=chr-reconnect,path=/tmp/vhost-test-6PT2U1/reconnect.sock,server=on:
> >  info: QEMU waiting for connection on: 
> > disconnected:unix:/tmp/vhost-test-6PT2U1/reconnect.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev 
> > socket,id=chr-connect-fail,path=/tmp/vhost-test-H8G7U1/connect-fail.sock,server=on:
> >  info: QEMU waiting for connection on: 
> > disconnected:unix:/tmp/vhost-test-H8G7U1/connect-fail.sock,server=on
> > qemu-system-arm: -netdev 
> > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to read 
> > msg header. Read 0 instead of 12. Original request 1.
> > qemu-system-arm: -netdev 
> > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: 
> > vhost_backend_init failed: Protocol error
> > qemu-system-arm: -netdev 
> > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to init 
> > vhost_net for queue 0
> > qemu-system-arm: -netdev 
> > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU 
> > waiting for connection on: 
> > disconnected:unix:/tmp/vhost-test-H8G7U1/connect-fail.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev 
> > socket,id=chr-flags-mismatch,path=/tmp/vhost-test-94UYU1/flags-mismatch.sock,server=on:
> >  info: QEMU waiting for connection on: 
> > disconnected:unix:/tmp/vhost-test-94UYU1/flags-mismatch.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
> > qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > UndefinedBehaviorSanitizer:DEADLYSIGNAL
> > ==8618==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 
> > 0x000000000000 (pc 0x55e34deccab0 bp 0x000000000000 sp 0x7ffc94894710 T8618)
> > ==8618==The signal is caused by a READ memory access.
> > ==8618==Hint: address points to the zero page.
> >     #0 0x55e34deccab0 in ldl_he_p 
> > /builds/qemu-project/qemu/include/qemu/bswap.h:301:5
> >     #1 0x55e34deccab0 in ldn_he_p 
> > /builds/qemu-project/qemu/include/qemu/bswap.h:440:1
> >     #2 0x55e34deccab0 in flatview_write_continue 
> > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2824:19
> >     #3 0x55e34dec9f21 in flatview_write 
> > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2867:12
> >     #4 0x55e34dec9f21 in address_space_write 
> > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2963:18
> >     #5 0x55e34decace7 in address_space_unmap 
> > /builds/qemu-project/qemu/build/../softmmu/physmem.c:3306:9
> >     #6 0x55e34de6d4ec in vhost_memory_unmap 
> > /builds/qemu-project/qemu/build/../hw/virtio/vhost.c:342:9
> >     #7 0x55e34de6d4ec in vhost_virtqueue_stop 
> > /builds/qemu-project/qemu/build/../hw/virtio/vhost.c:1242:5
> >     #8 0x55e34de72904 in vhost_dev_stop 
> > /builds/qemu-project/qemu/build/../hw/virtio/vhost.c:1882:9
> >     #9 0x55e34d890514 in vhost_net_stop_one 
> > /builds/qemu-project/qemu/build/../hw/net/vhost_net.c:331:5
> >     #10 0x55e34d88fef6 in vhost_net_start 
> > /builds/qemu-project/qemu/build/../hw/net/vhost_net.c:404:13
> >     #11 0x55e34de0bec6 in virtio_net_vhost_status 
> > /builds/qemu-project/qemu/build/../hw/net/virtio-net.c:307:13
> >     #12 0x55e34de0bec6 in virtio_net_set_status 
> > /builds/qemu-project/qemu/build/../hw/net/virtio-net.c:388:5
> >     #13 0x55e34de5e409 in virtio_set_status 
> > /builds/qemu-project/qemu/build/../hw/virtio/virtio.c:2442:9
> >     #14 0x55e34da22a50 in virtio_mmio_write 
> > /builds/qemu-project/qemu/build/../hw/virtio/virtio-mmio.c:428:9
> >     #15 0x55e34deb44a6 in memory_region_write_accessor 
> > /builds/qemu-project/qemu/build/../softmmu/memory.c:493:5
> >     #16 0x55e34deb428a in access_with_adjusted_size 
> > /builds/qemu-project/qemu/build/../softmmu/memory.c:555:18
> >     #17 0x55e34deb402d in memory_region_dispatch_write 
> > /builds/qemu-project/qemu/build/../softmmu/memory.c
> >     #18 0x55e34deccaf1 in flatview_write_continue 
> > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2825:23
> >     #19 0x55e34dec9f21 in flatview_write 
> > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2867:12
> >     #20 0x55e34dec9f21 in address_space_write 
> > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2963:18
> >     #21 0x55e34ded0bf6 in qtest_process_command 
> > /builds/qemu-project/qemu/build/../softmmu/qtest.c
> >     #22 0x55e34ded008d in qtest_process_inbuf 
> > /builds/qemu-project/qemu/build/../softmmu/qtest.c:796:9
> >     #23 0x55e34e109b02 in tcp_chr_read 
> > /builds/qemu-project/qemu/build/../chardev/char-socket.c:508:13
> >     #24 0x7fc6c665d0ae in g_main_context_dispatch 
> > (/lib64/libglib-2.0.so.0+0x550ae)
> >     #25 0x55e34e1fc1bc in glib_pollfds_poll 
> > /builds/qemu-project/qemu/build/../util/main-loop.c:297:9
> >     #26 0x55e34e1fc1bc in os_host_main_loop_wait 
> > /builds/qemu-project/qemu/build/../util/main-loop.c:320:5
> >     #27 0x55e34e1fc1bc in main_loop_wait 
> > /builds/qemu-project/qemu/build/../util/main-loop.c:596:11
> >     #28 0x55e34da52de6 in qemu_main_loop 
> > /builds/qemu-project/qemu/build/../softmmu/runstate.c:739:9
> >     #29 0x55e34d60a4f5 in qemu_default_main 
> > /builds/qemu-project/qemu/build/../softmmu/main.c:37:14
> >     #30 0x7fc6c43a5eaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf)
> >     #31 0x7fc6c43a5f5f in __libc_start_main@GLIBC_2.2.5 
> > (/lib64/libc.so.6+0x3ff5f)
> >     #32 0x55e34d5e1094 in _start 
> > (/builds/qemu-project/qemu/build/qemu-system-arm+0xc17094)
> > UndefinedBehaviorSanitizer can not provide additional info.
> > SUMMARY: UndefinedBehaviorSanitizer: SEGV 
> > /builds/qemu-project/qemu/include/qemu/bswap.h:301:5 in ldl_he_p
> > ==8618==ABORTING
> > Broken pipe
> > ../tests/qtest/libqtest.c:179: kill_qemu() tried to terminate QEMU process 
> > but encountered exit status 1 (expected 0)
> > **
> > ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process 
> > (/arm/virt/virtio-mmio/virtio-bus/virtio-net-device/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
> >  [8609]) failed unexpectedly
> > (test program exited with status code -6)
> >
> > https://gitlab.com/qemu-project/qemu/-/jobs/3265209698
> > https://gitlab.com/qemu-project/qemu/-/pipelines/683909108
> >
> > Stefan
>
>
> Ugh. I need to build with ubsan to reproduce yes? didn't trigger for me
> I am wondering how to bisect on gitlab.

I searched for "clang-system" (the name of the job) in .gitlab-ci.d to
get the job commands. The GitLab job output also contains details of
the commands that were run (unfortunately it doesn't expand
environment variables so some aspects are not visible from the GitLab
output).

That led to the following local command-line:

$ git checkout 645ec851
$ ./configure --enable-werror --disable-docs --target-list=arm-softmmu
--cc=clang --cxx=clang++ --extra-cflags=-fsanitize=undefined
--extra-cflags=-fno-sanitize-recover=undefined && make check-qtest

It reproduces locally on my Fedora 36 machine.

Stefan