Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Michael S. Tsirkin]

On Fri, Nov 04, 2022 at 05:58:17PM +0100, Christian Borntraeger wrote:
> Am 04.11.22 um 17:51 schrieb Christian Borntraeger:
> > 
> > 
> > Am 04.11.22 um 17:14 schrieb Michael S. Tsirkin:
> > > On Fri, Nov 04, 2022 at 04:59:35PM +0100, Christian Borntraeger wrote:
> > > > 
> > > > 
> > > > Am 04.11.22 um 16:56 schrieb Michael S. Tsirkin:
> > > > > On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
> > > > > > During migration the virtio device state can be restored before we
> > > > > > restart the VM. As no devices can be running while the VM is paused 
> > > > > > it
> > > > > > makes sense to bail out early in that case.
> > > > > > 
> > > > > > This returns the order introduced in:
> > > > > > 
> > > > > >    9f6bcfd99f (hw/virtio: move vm_running check to 
> > > > > > virtio_device_started)
> > > > > > 
> > > > > > to what virtio-sock was doing longhand.
> > > > > > 
> > > > > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > > > > > Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> > > > > 
> > > > > 
> > > > > What happens now:
> > > > > 
> > > > > with this applied I get:
> > > > > 
> > > > > https://gitlab.com/mitsirkin/qemu/-/pipelines/685829158/failures
> > > > > 
> > > > > ――――――――――――――――――――――――――――――――――――― ✀  
> > > > > ―――――――――――――――――――――――――――――――――――――
> > > > > stderr:
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: -chardev 
> > > > > socket,id=chr-reconnect,path=/tmp/vhost-test-QLKXU1/reconnect.sock,server=on:
> > > > >  info: QEMU waiting for connection on: 
> > > > > disconnected:unix:/tmp/vhost-test-QLKXU1/reconnect.sock,server=on
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: -chardev 
> > > > > socket,id=chr-connect-fail,path=/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on:
> > > > >  info: QEMU waiting for connection on: 
> > > > > disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> > > > > qemu-system-arm: -netdev 
> > > > > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to 
> > > > > read msg header. Read 0 instead of 12. Original request 1.
> > > > > qemu-system-arm: -netdev 
> > > > > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: 
> > > > > vhost_backend_init failed: Protocol error
> > > > > qemu-system-arm: -netdev 
> > > > > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to 
> > > > > init vhost_net for queue 0
> > > > > qemu-system-arm: -netdev 
> > > > > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU 
> > > > > waiting for connection on: 
> > > > > disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: -chardev 
> > > > > socket,id=chr-flags-mismatch,path=/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on:
> > > > >  info: QEMU waiting for connection on: 
> > > > > disconnected:unix:/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
> > > > > qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
> > > > > qemu-system-arm: unable to start vhost net: 22: falling back on 
> > > > > userspace virtio
> > > > > vhost lacks feature mask 0x40000000 for backend
> > > > > qemu-system-arm: failed to init vhost_net for queue 0
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 2 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 3 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid 
> > > > > argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -5: Input/output 
> > > > > error (5)
> > > > > qemu-system-arm: ../hw/virtio/virtio-bus.c:211: void 
> > > > > virtio_bus_release_ioeventfd(VirtioBusState *): Assertion 
> > > > > `bus->ioeventfd_grabbed != 0' failed.
> > > > > ../tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from 
> > > > > signal 6 (Aborted) (core dumped)
> > > > > **
> > > > > ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child 
> > > > > process 
> > > > > (/arm/virt/virtio-mmio/virtio-bus/vhost-user-gpio-device/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile/subprocess
> > > > >  [8735]) failed unexpectedly
> > > > > (test program exited with status code -6)
> > > > > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > without this it passes:
> > > > > 
> > > > > https://gitlab.com/mitsirkin/qemu/-/jobs/3275949777
> > > > > 
> > > > > 
> > > > > this only triggers under github, clang-system job.
> > > > > trying to decide what to do now. revert just this?
> > > > When we revert this save/restore for vsock is broken. Not sure, maybe 
> > > > we must use a more fine-grained fix as outlined in my initial mail?
> > > 
> > > 
> > > 
> > > Could you take a look here pls?
> > > 
> > > https://gitlab.com/mitsirkin/qemu/-/tree/testrevert1
> > 
> > I kicked of our regression test suite for s390.
> 
> This branch also seems to fix my original problem.
> Regression is still running but looks good so far.

Good to know. And I think that's a better fix in fact it's just a
couple of lines. However, the original gitlab failure is still
with us there :( Trying to figure out what is going on.