[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: qemu crashing when attaching an ISO file to a virtio-scsi CD-ROM dev
|
From: |
Fernando Casas Schössow |
|
Subject: |
Re: qemu crashing when attaching an ISO file to a virtio-scsi CD-ROM device through libvirt |
|
Date: |
Thu, 24 Oct 2019 21:07:16 +0000 |
Today I updated to qemu 4.0.1 since this was the latest version
available for Alpine and I can confirm that I can repro the issue with
this version as well.
Not sure if relevant but I can also confirm that the problem happens
with Windows Server 2012 R2 but also with Linux guests (it doesn't
matter if the guest use uefi or bios firmware). I made this tests just
to discard things.
Also as discussed I compiled qemu with debug symbols, repro the
problem, collected a core dump and run both through gdb. This is the
result:
(gdb) thread apply all bt
Thread 42 (LWP 33704):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fee02380b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 41 (LWP 33837):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1ad5b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 40 (LWP 33719):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fee02266b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 39 (LWP 33696):
#0 0x00007fee04233171 in syscall () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee02be8b64 in ?? ()
#2 0x0000000000000030 in ?? ()
#3 0x00007fee02be2540 in ?? ()
#4 0x00007fee02be2500 in ?? ()
#5 0x00007fee02be2548 in ?? ()
#6 0x000055d7e4987f28 in rcu_gp_event ()
#7 0x0000000000000000 in ?? ()
Thread 38 (LWP 33839):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1a83b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 37 (LWP 33841):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1737b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 36 (LWP 33863):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedb8c83b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 35 (LWP 33842):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc170eb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 34 (LWP 33862):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedb8cacb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 33 (LWP 33843):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc16e5b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 32 (LWP 33861):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedb8cd5b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 31 (LWP 33844):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc16bcb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 30 (LWP 33858):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedb8e83b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 29 (LWP 33845):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1693b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 28 (LWP 33857):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedb8eacb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 27 (LWP 33846):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc166ab64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 26 (LWP 33856):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedb8ed5b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 25 (LWP 33847):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc142ab64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 24 (LWP 33855):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedb8efeb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 23 (LWP 33848):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedbd0feb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 22 (LWP 33854):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedbd031b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 21 (LWP 33849):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedbd0d5b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 20 (LWP 33852):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedbd05ab64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 19 (LWP 33850):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedbd0acb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 18 (LWP 33851):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedbd083b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 17 (LWP 33836):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1afeb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 16 (LWP 33835):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1c5ab64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 15 (LWP 33834):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1c83b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 14 (LWP 33833):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1cacb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 13 (LWP 33677):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x000055d7e516656c in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 12 (LWP 33832):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1cd5b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 11 (LWP 33831):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1cfeb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 10 (LWP 33829):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1e67b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 9 (LWP 33828):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1e90b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 8 (LWP 33827):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fee02a95b64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 7 (LWP 33732):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fee0223db64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 6 (LWP 33706):
#0 0x00007fee0423263d in ioctl () from /lib/ld-musl-x86_64.so.1
#1 0x0000000000000001 in ?? ()
#2 0x00007fee00000010 in ?? ()
#3 0x00007fee02351440 in ?? ()
#4 0x00007fee02351400 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 5 (LWP 33838):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1aacb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 4 (LWP 33860):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedb8cfeb64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 3 (LWP 33859):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedb8e5ab64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 2 (LWP 33840):
#0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1
#1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1
#2 0x00007fedc1a5ab64 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 1 (LWP 33701):
#0 0x00007fee0421b7a1 in abort () from /lib/ld-musl-x86_64.so.1
#1 0x000055d7e6012b70 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb)
I'm not a developer nor skilled with gdb but if you provide me the
debugging commands I can execute them and reply back with the results.
I can also provide the binary and the core dump for analysis if needed.
While waiting for replies I will check if I can upgrade to qemu 4.1.0,
try to repro and provide the results.
Thanks.
Fernando
On mié, oct 23, 2019 at 7:57 PM, Fernando Casas Schössow
<address@hidden> wrote:
> In virsh I would do this while the guest is running:
>
> virsh attach-disk <vmname> <path_to_ISO_file> <guest_device> --type
> cdrom --mode readonly
>
> Following the example for guest from my first email:
>
> virsh attach-disk DCHOMENET01
> /resources/virtio-win-0.1.171-stable.iso sdb --type cdrom --mode
> readonly
>
> Right after executing this, qemu crashes and log the assertion.
> I can repro this also from virt-manager by selecting the cdrom device
> -> Connect -> selecting the ISO file -> Choose volume -> Ok
> (basically the same but in the gui).
>
> I may be able to try 4.1. Will look into it and report back.
>
> On mié, oct 23, 2019 at 7:33 PM, John Snow <address@hidden> wrote:
>>
>>
>> On 10/23/19 1:28 PM, Fernando Casas Schössow wrote:
>>> Hi John,
>>>
>>> Thanks for looking into this.
>>> I can quickly repro the problem with qemu 4.0 binary with debugging
>>> symbols enabled as I have it available already.
>>> Doing the same with qemu 4.1 or development head may be too much
>>> hassle
>>> but if it's really the only way I can give it try.
>>>
>>> Would it worth it to try with 4.0 first and get the strack trace
>>> or it
>>> will not help and the only way to go is with 4.1 (or dev)?
>>>
>>> Thanks,
>>>
>>> Fernando
>>>
>>
>> If 4.0 is what you have access to, having the stack trace for that is
>> better than not, but confirming it happens on the latest release
>> would
>> be nice.
>>
>> Can you share your workflow for virsh that reproduces the failure?
>>
>> --js
>>
>>> On mié, oct 23, 2019 at 5:34 PM, John Snow <address@hidden>
>>> wrote:
>>>> On 10/18/19 5:41 PM, Fernando Casas Schössow wrote:
>>>>
>>>> Hi,
>>>>
>>>> Hi! Thanks for the report.
>>>>
>>>> Today while working with two different Windows Server 2012 R2
>>>> guests I found that when I try to attach an ISO file to a SCSI
>>>> CD-ROM device through libvirt (virsh or virt-manager) while
>>>> the
>>>> guest is running, qemu crashes and the following message is
>>>> logged: Assertion failed: blk_get_aio_context(d->conf.blk) ==
>>>> s->ctx
>>>>
>>>> (/home/buildozer/aports/main/qemu/src/qemu-4.0.0/hw/scsi/virtio-scsi.c:
>>>> virtio_scsi_ctx_check: 246) I can repro this at will. All I
>>>> have
>>>> to do is to try to attach an ISO file to the SCSI CDROM while
>>>> the
>>>> guest is running. The SCSI controller model is virtio-scsi
>>>> with
>>>> iothread enabled. Please find below all the details about my
>>>> setup
>>>> that I considered relevant but I missed something please don't
>>>> hesitate to let me know:
>>>>
>>>> Looks like we got aio_context management wrong with iothread for
>>>> the
>>>> media change events somewhere. Should be easy enough to fix if we
>>>> figure out where the bad assumption is.
>>>>
>>>> Host arch: x86_64 Distro: Alpine Linux 3.10.2 qemu version:
>>>> 4.0
>>>>
>>>> Do you have the ability to try 4.1, or the latest development head
>>>> with debugging symbols enabled?
>>>>
>>>> Linux kernel version: 4.19.67 libvirt: 5.5.0 Emulated SCSI
>>>> controller: virtio-scsi (with iothread enabled) Guest
>>>> firmware:
>>>> OVMF-EFI Guest OS: Window Server 2012 R2 Guest virtio drivers
>>>> version: 171 (current stable) qemu command line:
>>>> /usr/bin/qemu-system-x86_64 -name
>>>> guest=DCHOMENET01,debug-threads=on -S -object
>>>>
>>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-78-DCHOMENET01/master-key.aes
>>>> -machine pc-i440fx-4.0,accel=kvm,usb=off,dump-guest-core=on
>>>> -cpu
>>>>
>>>> IvyBridge,ss=on,vmx=off,pcid=on,hypervisor=on,arat=on,tsc_adjust=on,umip=on,xsaveopt=on,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff
>>>> -drive
>>>>
>>>> file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on
>>>> -drive
>>>>
>>>> file=/var/lib/libvirt/qemu/nvram/DCHOMENET01_VARS.fd,if=pflash,format=raw,unit=1
>>>> -m 1536 -overcommit mem-lock=off -smp
>>>> 1,sockets=1,cores=1,threads=1 -object iothread,id=iothread1
>>>> -uuid
>>>> f06978ad-2734-44ab-a518-5dfcf71d625e -no-user-config
>>>> -nodefaults
>>>> -chardev socket,id=charmonitor,fd=33,server,nowait -mon
>>>> chardev=charmonitor,id=monitor,mode=control -rtc
>>>> base=localtime,driftfix=slew -global
>>>> kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global
>>>> PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot
>>>> strict=on -device qemu-xhci,id=usb,bus=pci.0,addr=0x4 -device
>>>>
>>>> virtio-scsi-pci,iothread=iothread1,id=scsi0,num_queues=1,bus=pci.0,addr=0x5
>>>> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6
>>>> -drive
>>>>
>>>> file=/storage/storage-hdd-vms/virtual_machines_hdd/dchomenet01.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=none,discard=unmap,aio=threads
>>>> -device
>>>>
>>>> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=drive-scsi0-0-0-0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1,write-cache=on
>>>> -drive if=none,id=drive-scsi0-0-0-1,readonly=on -device
>>>>
>>>> scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,device_id=drive-scsi0-0-0-1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1
>>>> -netdev tap,fd=41,id=hostnet0,vhost=on,vhostfd=43 -device
>>>>
>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:99:b5:62,bus=pci.0,addr=0x3
>>>> -chardev
>>>>
>>>> socket,id=charserial0,host=127.0.0.1,port=4900,telnet,server,nowait
>>>> -device
>>>> isa-serial,chardev=charserial0,id=serial0 -chardev
>>>> spicevmc,id=charchannel0,name=vdagent -device
>>>>
>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
>>>> -chardev socket,id=charchannel1,fd=45,server,nowait -device
>>>>
>>>> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
>>>> -chardev
>>>> spiceport,id=charchannel2,name=org.spice-space.webdav.0
>>>> -device
>>>>
>>>> virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=org.spice-space.webdav.0
>>>> -device virtio-tablet-pci,id=input2,bus=pci.0,addr=0x7 -spice
>>>>
>>>> port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on
>>>> -device
>>>>
>>>> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2
>>>> -chardev spicevmc,id=charredir0,name=usbredir -device
>>>> usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2
>>>> -chardev
>>>> spicevmc,id=charredir1,name=usbredir -device
>>>> usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3
>>>> -device
>>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -sandbox
>>>>
>>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
>>>> -msg timestamp=on I can provide a core dump of the process if
>>>> needed for debugging and the guest XML as well.
>>>>
>>>> A backtrace is probably a great starting point (from gdb: `thread
>>>> apply all bt`.) I don't know exactly what codepath is being
>>>> exercised
>>>> when you "attach an ISO file" through libvirt's interface. If you
>>>> don't mind the hassle, trying on the 4.1 (or a development build
>>>> would
>>>> be even more luxurious) and giving a stacktrace would be nice.
>>>>
>>>> Thanks. Fernando
>>>>
>>>> Thanks! --js
>>>
>>>
>>
>
>