Re: [PULL v2 00/82] pci,pc,virtio: features, tests, fixes, cleanups

[Michael S. Tsirkin]

On Thu, Nov 03, 2022 at 09:29:56AM -0400, Stefan Hajnoczi wrote:
> On Thu, 3 Nov 2022 at 08:14, Michael S. Tsirkin <mst@redhat.com> wrote:
> > On Wed, Nov 02, 2022 at 03:47:43PM -0400, Stefan Hajnoczi wrote:
> > > On Wed, Nov 02, 2022 at 12:02:14PM -0400, Michael S. Tsirkin wrote:
> > > > Changes from v1:
> > > >
> > > > Applied and squashed fixes by Igor, Lei He, Hesham Almatary for
> > > > bugs that tripped up the pipeline.
> > > > Updated expected files for core-count test.
> > >
> > > Several "make check" CI failures have occurred. They look like they are
> > > related. Here is one (see the URLs at the bottom of this email for more
> > > details):
> > >
> > > 17/106 ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child 
> > > process 
> > > (/arm/virt/virtio-mmio/virtio-bus/virtio-net-device/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
> > >  [8609]) failed unexpectedly ERROR
> > >  17/106 qemu:qtest+qtest-arm / qtest-arm/qos-test                     
> > > ERROR          31.44s   killed by signal 6 SIGABRT
> > > >>> G_TEST_DBUS_DAEMON=/builds/qemu-project/qemu/tests/dbus-vmstate-daemon.sh
> > > >>>  MALLOC_PERTURB_=49 QTEST_QEMU_IMG=./qemu-img 
> > > >>> QTEST_QEMU_BINARY=./qemu-system-arm 
> > > >>> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon 
> > > >>> /builds/qemu-project/qemu/build/tests/qtest/qos-test --tap -k
> > > ――――――――――――――――――――――――――――――――――――― ✀  
> > > ―――――――――――――――――――――――――――――――――――――
> > > stderr:
> > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument 
> > > (22)
> > > qemu-system-arm: Failed to set msg fds.
> > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument 
> > > (22)
> > > qemu-system-arm: -chardev 
> > > socket,id=chr-reconnect,path=/tmp/vhost-test-6PT2U1/reconnect.sock,server=on:
> > >  info: QEMU waiting for connection on: 
> > > disconnected:unix:/tmp/vhost-test-6PT2U1/reconnect.sock,server=on
> > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument 
> > > (22)
> > > qemu-system-arm: Failed to set msg fds.
> > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument 
> > > (22)
> > > qemu-system-arm: -chardev 
> > > socket,id=chr-connect-fail,path=/tmp/vhost-test-H8G7U1/connect-fail.sock,server=on:
> > >  info: QEMU waiting for connection on: 
> > > disconnected:unix:/tmp/vhost-test-H8G7U1/connect-fail.sock,server=on
> > > qemu-system-arm: -netdev 
> > > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to read 
> > > msg header. Read 0 instead of 12. Original request 1.
> > > qemu-system-arm: -netdev 
> > > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: 
> > > vhost_backend_init failed: Protocol error
> > > qemu-system-arm: -netdev 
> > > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to init 
> > > vhost_net for queue 0
> > > qemu-system-arm: -netdev 
> > > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU 
> > > waiting for connection on: 
> > > disconnected:unix:/tmp/vhost-test-H8G7U1/connect-fail.sock,server=on
> > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument 
> > > (22)
> > > qemu-system-arm: Failed to set msg fds.
> > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument 
> > > (22)
> > > qemu-system-arm: -chardev 
> > > socket,id=chr-flags-mismatch,path=/tmp/vhost-test-94UYU1/flags-mismatch.sock,server=on:
> > >  info: QEMU waiting for connection on: 
> > > disconnected:unix:/tmp/vhost-test-94UYU1/flags-mismatch.sock,server=on
> > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
> > > qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
> > > qemu-system-arm: Failed to set msg fds.
> > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument 
> > > (22)
> > > UndefinedBehaviorSanitizer:DEADLYSIGNAL
> > > ==8618==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 
> > > 0x000000000000 (pc 0x55e34deccab0 bp 0x000000000000 sp 0x7ffc94894710 
> > > T8618)
> > > ==8618==The signal is caused by a READ memory access.
> > > ==8618==Hint: address points to the zero page.
> > >     #0 0x55e34deccab0 in ldl_he_p 
> > > /builds/qemu-project/qemu/include/qemu/bswap.h:301:5
> > >     #1 0x55e34deccab0 in ldn_he_p 
> > > /builds/qemu-project/qemu/include/qemu/bswap.h:440:1
> > >     #2 0x55e34deccab0 in flatview_write_continue 
> > > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2824:19
> > >     #3 0x55e34dec9f21 in flatview_write 
> > > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2867:12
> > >     #4 0x55e34dec9f21 in address_space_write 
> > > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2963:18
> > >     #5 0x55e34decace7 in address_space_unmap 
> > > /builds/qemu-project/qemu/build/../softmmu/physmem.c:3306:9
> > >     #6 0x55e34de6d4ec in vhost_memory_unmap 
> > > /builds/qemu-project/qemu/build/../hw/virtio/vhost.c:342:9
> > >     #7 0x55e34de6d4ec in vhost_virtqueue_stop 
> > > /builds/qemu-project/qemu/build/../hw/virtio/vhost.c:1242:5
> > >     #8 0x55e34de72904 in vhost_dev_stop 
> > > /builds/qemu-project/qemu/build/../hw/virtio/vhost.c:1882:9
> > >     #9 0x55e34d890514 in vhost_net_stop_one 
> > > /builds/qemu-project/qemu/build/../hw/net/vhost_net.c:331:5
> > >     #10 0x55e34d88fef6 in vhost_net_start 
> > > /builds/qemu-project/qemu/build/../hw/net/vhost_net.c:404:13
> > >     #11 0x55e34de0bec6 in virtio_net_vhost_status 
> > > /builds/qemu-project/qemu/build/../hw/net/virtio-net.c:307:13
> > >     #12 0x55e34de0bec6 in virtio_net_set_status 
> > > /builds/qemu-project/qemu/build/../hw/net/virtio-net.c:388:5
> > >     #13 0x55e34de5e409 in virtio_set_status 
> > > /builds/qemu-project/qemu/build/../hw/virtio/virtio.c:2442:9
> > >     #14 0x55e34da22a50 in virtio_mmio_write 
> > > /builds/qemu-project/qemu/build/../hw/virtio/virtio-mmio.c:428:9
> > >     #15 0x55e34deb44a6 in memory_region_write_accessor 
> > > /builds/qemu-project/qemu/build/../softmmu/memory.c:493:5
> > >     #16 0x55e34deb428a in access_with_adjusted_size 
> > > /builds/qemu-project/qemu/build/../softmmu/memory.c:555:18
> > >     #17 0x55e34deb402d in memory_region_dispatch_write 
> > > /builds/qemu-project/qemu/build/../softmmu/memory.c
> > >     #18 0x55e34deccaf1 in flatview_write_continue 
> > > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2825:23
> > >     #19 0x55e34dec9f21 in flatview_write 
> > > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2867:12
> > >     #20 0x55e34dec9f21 in address_space_write 
> > > /builds/qemu-project/qemu/build/../softmmu/physmem.c:2963:18
> > >     #21 0x55e34ded0bf6 in qtest_process_command 
> > > /builds/qemu-project/qemu/build/../softmmu/qtest.c
> > >     #22 0x55e34ded008d in qtest_process_inbuf 
> > > /builds/qemu-project/qemu/build/../softmmu/qtest.c:796:9
> > >     #23 0x55e34e109b02 in tcp_chr_read 
> > > /builds/qemu-project/qemu/build/../chardev/char-socket.c:508:13
> > >     #24 0x7fc6c665d0ae in g_main_context_dispatch 
> > > (/lib64/libglib-2.0.so.0+0x550ae)
> > >     #25 0x55e34e1fc1bc in glib_pollfds_poll 
> > > /builds/qemu-project/qemu/build/../util/main-loop.c:297:9
> > >     #26 0x55e34e1fc1bc in os_host_main_loop_wait 
> > > /builds/qemu-project/qemu/build/../util/main-loop.c:320:5
> > >     #27 0x55e34e1fc1bc in main_loop_wait 
> > > /builds/qemu-project/qemu/build/../util/main-loop.c:596:11
> > >     #28 0x55e34da52de6 in qemu_main_loop 
> > > /builds/qemu-project/qemu/build/../softmmu/runstate.c:739:9
> > >     #29 0x55e34d60a4f5 in qemu_default_main 
> > > /builds/qemu-project/qemu/build/../softmmu/main.c:37:14
> > >     #30 0x7fc6c43a5eaf in __libc_start_call_main 
> > > (/lib64/libc.so.6+0x3feaf)
> > >     #31 0x7fc6c43a5f5f in __libc_start_main@GLIBC_2.2.5 
> > > (/lib64/libc.so.6+0x3ff5f)
> > >     #32 0x55e34d5e1094 in _start 
> > > (/builds/qemu-project/qemu/build/qemu-system-arm+0xc17094)
> > > UndefinedBehaviorSanitizer can not provide additional info.
> > > SUMMARY: UndefinedBehaviorSanitizer: SEGV 
> > > /builds/qemu-project/qemu/include/qemu/bswap.h:301:5 in ldl_he_p
> > > ==8618==ABORTING
> > > Broken pipe
> > > ../tests/qtest/libqtest.c:179: kill_qemu() tried to terminate QEMU 
> > > process but encountered exit status 1 (expected 0)
> > > **
> > > ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child 
> > > process 
> > > (/arm/virt/virtio-mmio/virtio-bus/virtio-net-device/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
> > >  [8609]) failed unexpectedly
> > > (test program exited with status code -6)
> > >
> > > https://gitlab.com/qemu-project/qemu/-/jobs/3265209698
> > > https://gitlab.com/qemu-project/qemu/-/pipelines/683909108
> > >
> > > Stefan
> >
> >
> > Ugh. I need to build with ubsan to reproduce yes? didn't trigger for me
> > I am wondering how to bisect on gitlab.
> 
> I searched for "clang-system" (the name of the job) in .gitlab-ci.d to
> get the job commands. The GitLab job output also contains details of
> the commands that were run (unfortunately it doesn't expand
> environment variables so some aspects are not visible from the GitLab
> output).
> 
> That led to the following local command-line:
> 
> $ git checkout 645ec851
> $ ./configure --enable-werror --disable-docs --target-list=arm-softmmu
> --cc=clang --cxx=clang++ --extra-cflags=-fsanitize=undefined
> --extra-cflags=-fno-sanitize-recover=undefined && make check-qtest
> It reproduces locally on my Fedora 36 machine.
> 
> Stefan

Oh I'm still on 35, that's why.

-- 
MST