Re: [Qemu-devel] nbd: Possible regression in 2.9 RCs

[Ciprian Barbu]

Hi,

Thank you for getting back!

I'm trying to follow you, but I don't understand all the details. I would like 
to ask this question though:

What is the difference between v2.8.0 and this commit? With v2.8.0 the same 
qemu command worked, but I admit it doesn't request sharing.

We also use libvirt v1.3.4, which might be a problem, but at least we want to 
understand if the commit in question introduced an obvious problem or if it's 
all in the details.

Btw, the qemu command generated by libvirt is this one, sorry about that:

2017-03-31 17:40:10.956+0000: starting up libvirt version: 1.3.4, package: 
0+amos3~u16.04 (Enea Armband Devops Team <address@hidden> Fri, 13 Jan 2017 
02:06:05 +0100), qemu version: 2.8.50(Debian 1:2.9+amos2~u16.04), hostname: 
node-2.domain.tld
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 
QEMU_AUDIO_DRV=none /usr/bin/kvm -name instance-00000076,debug-threads=on -S 
-object 
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-14-instance-00000076/master-key.aes
 -machine virt-2.8,accel=kvm,usb=off,gic-version=3 -cpu host -m 256 -realtime 
mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 
2812f3c9-f564-499b-a8c7-e9e7ccf24143 -no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-14-instance-00000076/monitor.sock,server,nowait
 -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew 
-no-shutdown -boot strict=on -kernel 
/var/lib/nova/instances/2812f3c9-f564-499b-a8c7-e9e7ccf24143/kernel -initrd 
/var/lib/nova/instances/2812f3c9-f564-499b-a8c7-e9e7ccf24143/ramdisk -append 
'root=/dev/vda1 rw rootwait console=tty0 console=ttyS0 console=ttyAMA0' -device 
i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device 
pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -usb -drive 
file=/var/lib/nova/instances/2812f3c9-f564-499b-a8c7-e9e7ccf24143/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,aio=native
 -device 
virtio-blk-device,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
-drive 
file=/var/lib/nova/instances/2812f3c9-f564-499b-a8c7-e9e7ccf24143/disk.config,format=raw,if=none,id=drive-virtio-disk1,cache=none,aio=native
 -device virtio-blk-device,scsi=off,drive=drive-virtio-disk1,id=virtio-disk1 
-netdev tap,fd=27,id=hostnet0 -device 
virtio-net-device,netdev=hostnet0,id=net0,mac=fa:16:3e:82:0a:2b -serial 
file:/var/lib/nova/instances/2812f3c9-f564-499b-a8c7-e9e7ccf24143/console.log 
-serial pty -vnc 0.0.0.0:0 -k en-us -device 
VGA,id=video0,vgamem_mb=16,bus=pci.2,addr=0x1 -device 
virtio-balloon-device,id=balloon0 -msg timestamp=on
Domain id=14 is tainted: high-privileges

Regards,
/Ciprian

-----Original Message-----
From: Max Reitz [mailto:address@hidden 
Sent: Friday, March 31, 2017 8:43 PM
To: Ciprian Barbu <address@hidden>; address@hidden; Eric Blake 
<address@hidden>; Alexandru Avadanii <address@hidden>
Cc: Jeff Cody <address@hidden>; Markus Armbruster <address@hidden>; svc-armband 
<address@hidden>; Kevin Wolf <address@hidden>
Subject: Re: [Qemu-devel] nbd: Possible regression in 2.9 RCs

On 31.03.2017 18:03, Ciprian Barbu wrote:
> Hello,
> 
> Similar to the other thread about possible regression with rbd, there might 
> be a regression with nbd.
> This time we are launching an instance from an image (not volume) and try to 
> live migrate it:
> 
> nova live-migration <test_instance>
> 
> The nova-compute service complains with:
> 
> 2017-03-31 15:32:56.179 7806 INFO nova.virt.libvirt.driver 
> [req-15d79cbe-5956-4738-92df-3624e6b993ee 
> d795de59fb9a4ea38776a11d20ae8469 cee03e74881f4ccba3b83345fb652b2c - - 
> -] [instance: 6a04508f-5d79-4582-8e2c-4cc368753f6c] Migration running 
> for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, 
> total=0)
> 2017-03-31 15:32:58.029 7806 WARNING stevedore.named 
> [req-73bc0113-5555-4dd8-8903-d3540cc61b47 
> b9fbceeadd2d4d1bab9c90ae104db1f7 7e7db99b32c6467184701e9a0c2f1de7 - - 
> -] Could not load instance_network_info
> 2017-03-31 15:32:59.038 7806 ERROR nova.virt.libvirt.driver 
> [req-15d79cbe-5956-4738-92df-3624e6b993ee 
> d795de59fb9a4ea38776a11d20ae8469 cee03e74881f4ccba3b83345fb652b2c - - 
> -] [instance: 6a04508f-5d79-4582-8e2c-4cc368753f6c] Live Migration 
> failure: internal error: unable to execute QEMU command 
> 'nbd-server-add': Conflicts with use by drive-virtio-disk0 as 'root', 
> which does not allow 'write' on #block143
> 2017-03-31 15:32:59.190 7806 ERROR nova.virt.libvirt.driver 
> [req-15d79cbe-5956-4738-92df-3624e6b993ee 
> d795de59fb9a4ea38776a11d20ae8469 cee03e74881f4ccba3b83345fb652b2c - - 
> -] [instance: 6a04508f-5d79-4582-8e2c-4cc368753f6c] Migration 
> operation has aborted
> 
> I will try and bisect it myself, but I thought I would paste this here first, 
> just so you know there is this issue too.

Well, I already know the commit in question. It's
8a7ce4f9338c475df1afc12502af704e4300a3e0 ("nbd/server: Use real permissions for 
NBD exports").

Whether this is a bug depends on the standpoint. I would very much consider it 
a bug fix because as of this commit you can no longer create a writable NBD 
server on a block device that is in use by a guest device without the guest 
device being aware of this.

The problem is that the functionality to "make" the guest device "aware"
of it was introduced only a couple of commits before, and it's called 
"share-rw".

So this doesn't work:

$ x86_64-softmmu/qemu-system-x86_64 \
    -blockdev node-name=image,driver=qcow2,\
file.driver=file,file.filename=foo.qcow2 \
    -device virtio-blk,drive=image \
    -qmp stdio
{"QMP": {"version": {"qemu": {"micro": 92, "minor": 8, "major": 2},
"package": " (v2.8.0-2038-g6604c893d0)"}, "capabilities": []}} 
{'execute':'qmp_capabilities'}
{"return": {}}
{'execute':'nbd-server-start','arguments':{'addr':{'type':'inet','data':{'host':'localhost','port':'10809'}}}}
{"return": {}}
{'execute':'nbd-server-add','arguments':{'device':'image','writable':true}}
{"error": {"class": "GenericError", "desc": "Conflicts with use by 
/machine/peripheral-anon/device[0]/virtio-backend as 'root', which does not 
allow 'write' on image"}

But this works:

$ x86_64-softmmu/qemu-system-x86_64 \
    -blockdev node-name=image,driver=qcow2,\
file.driver=file,file.filename=foo.qcow2 \
    -device virtio-blk,drive=image,share-rw=on \
    -qmp stdio
{"QMP": {"version": {"qemu": {"micro": 92, "minor": 8, "major": 2},
"package": " (v2.8.0-2038-g6604c893d0)"}, "capabilities": []}} 
{'execute':'qmp_capabilities'}
{"return": {}}
{'execute':'nbd-server-start','arguments':{'addr':{'type':'inet','data':{'host':'localhost','port':'10809'}}}}
{"return": {}}
{'execute':'nbd-server-add','arguments':{'device':'image','writable':true}}
{"return": {}}

(The difference is the share-rw=on in the -device parameter.)


So in theory all that's necessary is to set share-rw=on for the device in the 
management layer. But I'm not sure whether that's practical.

As for just allowing the NBD server write access to the device... To me that 
appears pretty difficult from an implementation perspective. We assert that 
nobody can write without having requested write access and we make sure that 
nobody can request write access without it being allowed. Making an exception 
for NBD seems very difficult and would probably mean we'd have to drop the 
assertion for write accesses altogether.

Max