Re: [Qemu-devel] [Qemu-stable] Data corruption in Qemu 2.7.1

[Fabian Grünbichler]

On Tue, Jan 17, 2017 at 11:41:44AM +0100, Paolo Bonzini wrote:
> 
> 
> On 17/01/2017 09:03, Fabian Grünbichler wrote:
> > Commit 8fdc7839e40f43a426bc7e858cf1dbfe315a3804 (first included in
> > 2.7.0)[1] changed the behaviour of scsi-block passthrough. Previously
> > this worked with SATA disks, now it doesn't anymore. A bisect run
> > confirmed this, scsi-block with a SATA disk passed through via
> > virtio-scsi-single corrupts on writes since that commit, scsi-hd and
> > scsi-disk work fine (scsi-generic corrupts as well).
> > 
> > PVE's detection logic for passthrough just differentiated between disks
> > and tape drives, and unfortunately the SG_IO ioctl says SATA disks are
> > disks as well.. we probably need to default to scsi-hd or scsi-disk
> > instead of scsi-block, and only when we explicitly detect a "real" SCSI
> > disk we are allowed to use scsi-block?
> > 
> > @Paolo: was the old behaviour just an accident and the new bevaviour
> > intentional? documentation is quite sparse, or maybe I am looking in the
> > wrong places..
> 
> No, it would be a bug (QEMU or kernel).
> 
> Do you have an easy reproducer with dd, as suggested at
> https://forum.proxmox.com/threads/proxmox-4-4-virtio_scsi-regression.31471/page-2?
> 
> Paolo
> 

setup

1) dd 1G of (u)random data to a file on a file system on a SATA disk on
the host (I tested with ext2/3/4, ZFS and btrfs, btrfs produces
corruption the fastest here, but raw writes to the device should already
trigger the kernel error messages)

2) calculate the md5sum of this file, and store it on the same FS

test (with current qemu master, using PVE's 4.4.35 kernel as host kernel

1) start Ubuntu 16.04 VM (see full commandline at the end, I tested with
three disks, one for each of the file systems above) - the same behaviour
can also be observed when booting from an Alpine Linux iso

2) mount FS

3) dd random file from above to new file on same FS

4) sync

5) check md5sum of resulting file

6) repeat 3-5 until md5sum does not match, kernel spews error
messages, or you are convinced that everything is OK

sample kernel message (for ext3):
Jan 17 11:39:32 ubuntu kernel: sd 2:0:0:0: [sda] tag#32 FAILED Result: 
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 17 11:39:32 ubuntu kernel: sd 2:0:0:0: [sda] tag#32 Sense Key : Illegal 
Request [current]
Jan 17 11:39:32 ubuntu kernel: sd 2:0:0:0: [sda] tag#32 Add. Sense: Invalid 
field in cdb
Jan 17 11:39:32 ubuntu kernel: sd 2:0:0:0: [sda] tag#32 CDB: Write(10) 2a 00 0f 
3a 90 00 00 07 d8 00
Jan 17 11:39:32 ubuntu kernel: blk_update_request: critical target error, dev 
sda, sector 255496192
Jan 17 11:39:32 ubuntu kernel: EXT4-fs warning (device sda1): ext4_end_bio:329: 
I/O error -121 writing to inode 125 (offset 0 size 0 starting block 31937280)
Jan 17 11:39:32 ubuntu kernel: buffer_io_error: 246 callbacks suppressed
Jan 17 11:39:32 ubuntu kernel: Buffer I/O error on device sda1, logical block 
31936768
Jan 17 11:39:32 ubuntu kernel: Buffer I/O error on device sda1, logical block 
31936769
Jan 17 11:39:32 ubuntu kernel: Buffer I/O error on device sda1, logical block 
31936770
Jan 17 11:39:32 ubuntu kernel: Buffer I/O error on device sda1, logical block 
31936771
Jan 17 11:39:32 ubuntu kernel: Buffer I/O error on device sda1, logical block 
31936772
Jan 17 11:39:32 ubuntu kernel: Buffer I/O error on device sda1, logical block 
31936773
Jan 17 11:39:32 ubuntu kernel: Buffer I/O error on device sda1, logical block 
31936774
Jan 17 11:39:32 ubuntu kernel: Buffer I/O error on device sda1, logical block 
31936775
Jan 17 11:39:32 ubuntu kernel: Buffer I/O error on device sda1, logical block 
31936776
Jan 17 11:39:32 ubuntu kernel: Buffer I/O error on device sda1, logical block 
31936777
Jan 17 11:39:39 ubuntu kernel: JBD2: Detected IO errors while flushing file 
data on sda1-8
Jan 17 11:39:41 ubuntu kernel: JBD2: Detected IO errors while flushing file 
data on sda1-8
Jan 17 11:39:55 ubuntu kernel: JBD2: Detected IO errors while flushing file 
data on sda1-8
Jan 17 11:39:56 ubuntu kernel: JBD2: Detected IO errors while flushing file 
data on sda1-8
Jan 17 11:40:07 ubuntu kernel: JBD2: Detected IO errors while flushing file 
data on sda1-8
Jan 17 11:40:08 ubuntu kernel: JBD2: Detected IO errors while flushing file 
data on sda1-8
Jan 17 11:40:15 ubuntu kernel: JBD2: Detected IO errors while flushing file 
data on sda1-8
Jan 17 11:40:22 ubuntu kernel: JBD2: Detected IO errors while flushing file 
data on sda1-8

qemu commandline, originally generated by PVE, feel free to adapt or
minimize.. sdb, sdc and sde are the three SATA disks from above,
vm-101-disk-1 is the Ubuntu rootfs on LVM-thin.

/root/qemu/build/x86_64-softmmu/qemu-system-x86_64 \
 -enable-kvm \
 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' \
 -mon 'chardev=qmp,mode=control' \
 -pidfile /var/run/qemu-server/101.pid \
 -smbios 'type=1,uuid=3e550136-9d7f-4a12-9d98-e25a5d2d5806' \
 -name diskpassthroughtest \
 -smp '8,sockets=1,cores=8,maxcpus=8' \
 -nodefaults \
 -boot 
'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg'
 \
 -vga cirrus \
 -vnc unix:/var/run/qemu-server/101.vnc,x509,password \
 -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce \
 -m 4096 \
 -k de \
 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' \
 -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \
 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' \
 -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' \
 -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' \
 -iscsi 'initiator-name=iqn.1993-08.org.debian:01:c6676e1b1f72' \
 -drive 
'file=/mnt/pve/iso/template/iso/ubuntu-16.04.1-server-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads'
 \
 -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
 -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \
 -drive 
'file=/dev/sdb,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on'
 \
 -device 
'scsi-block,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0'
 \
 -device 'virtio-scsi-pci,id=virtioscsi1,bus=pci.3,addr=0x2' \
 -drive 
'file=/dev/sdc,if=none,id=drive-scsi1,format=raw,cache=none,aio=native,detect-zeroes=on'
 \
 -device 
'scsi-block,bus=virtioscsi1.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1'
 \
 -device 'virtio-scsi-pci,id=virtioscsi3,bus=pci.3,addr=0x4' \
 -drive 
'file=/dev/sde,if=none,id=drive-scsi3,format=raw,cache=none,aio=native,detect-zeroes=on'
 \
 -device 
'scsi-block,bus=virtioscsi3.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3'
 \
 -drive 
'file=/dev/pve/vm-101-disk-1,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on'
 \
 -device 
'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=103'
 \
 -netdev 
'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on'
 \
 -device 
'virtio-net-pci,mac=46:83:4B:92:EC:88,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'