Hi all,
I have a namespace question about passthrough disk(multipath device).
In case of enabling namespace and cgroups in qemu.conf, The target(s)
of the
multipath device won't be added into qemu-pr-helper's namespace under
certain
situation, It causes the PERSISTENT RESERVE command failure in guest.
While user starts a vm,
To build namespace, The qemuDomainSetupDisk() will be invoked via
threadA(this
thread id will be the qemu's pid),
To build cgroup, The qemuSetupImagePathCgroup() will be invoked via
threadB.
Both of the functions invoke the virDevMapperGetTargets() trying to
parse a
multipath device to target paths string, Then fill the targetPaths[].
The issue I experienced is:
After libvirtd started, Everything works well for the first booted vm
which has
the passthrough multipath device.
But If I shut it down & start it again, OR keep it running & start
another vm
which has other passthrough multipath device, Then the target(s) of the
fresh
started vm won't be added into the related qemu-pr-helper's namespace
and it
causes PERSISTENT RESERVE command failure in the corresponding guest.
I digged into code, In this situation, The targetPaths[] in
qemuDomainSetupDisk()
won't be filled, it keeps NULL after virDevMapperGetTargets() returns.
The virDevMapperGetTargets doesn't fill targetPaths[] because the
dm_task_run()
of libdevmapper returns 0 with errno 9(Bad file descriptor).
So far, I don't understand why the dm_task_run() return 0 in this
situation.
BTW, The virDevMapperGetTargets() can always successfully fill the
targetPaths[]
in qemuSetupImagePathCgroup().
Please refer to the following 2 tests:
The multipath configuration on host:
host:~ # multipath -l
vm1-data (3600140582d9024bc13f4b8db5ff12de0) dm-11 FreeNAS,lv68
size=6.0G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
`- 2:0:0:2 sdd 8:48 active undef running
vm2-data (36001405fc5f29ace3ec4fb8acd32aae5) dm-8 FreeNAS,lv46
size=4.0G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
`- 2:0:0:1 sde 8:64 active undef running
===================================================================
Test A:
host:~ # systemctl restart libvirtd
host:~ # virsh list
Id Name State
--------------------
host:~ #
host:~ # virsh domblklist vm1
Target Source
------------------------------------------
vda /opt/vms/vm1/disk0.qcow2
sda /dev/mapper/vm1-data
host:~ #
host:~ # virsh start vm1
Domain vm1 started
host:~ # virsh list
Id Name State
---------------------------
1 vm1 running
host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 48 Jul 14 16:30 /dev/sdd
host:~ # exit
exit
host:~ #
vm1:~ # lsscsi
[0:0:0:0] disk FreeNAS lv68 0123 /dev/sda
vm1:~ #
vm1:~ # sg_persist --in -k /dev/sda
FreeNAS lv68 0123
Peripheral device type: disk
PR generation=0x0, there are NO registered reservation keys
vm1:~ #
host:~ # virsh shutdown vm1
Domain vm1 is being shutdown
host:~ # virsh list
Id Name State
--------------------
host:~ #
host:~ # virsh start vm1
Domain vm1 started
host:~ # virsh list
Id Name State
---------------------------
2 vm1 running
host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory
host:~ # exit
exit
host:~ #
vm1:~ # sg_persist --in -k /dev/sda
FreeNAS lv68 0123
Peripheral device type: disk
PR in (Read keys): Aborted command
Aborted command
vm1:~ #
===================================================================
Test B:
host:~ # systemctl restart libvirtd
host:~ # virsh list
Id Name State
--------------------
host:~ #
host:~ # virsh domblklist vm1
Target Source
------------------------------------------
vda /opt/vms/vm1/disk0.qcow2
sda /dev/mapper/vm1-data
host:~ #
host:~ # virsh start vm1
Domain vm1 started
host:~ # virsh list
Id Name State
---------------------------
1 vm1 running
host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 48 Jul 14 17:28 /dev/sdd
host:~ # exit
exit
host:~ #
vm1:~ # lsscsi
[2:0:0:0] disk FreeNAS lv68 0123 /dev/sda
vm1:~ #
vm1:~ # sg_persist --in -k /dev/sda
FreeNAS lv68 0123
Peripheral device type: disk
PR generation=0x0, there are NO registered reservation keys
vm1:~ #
host:~ # virsh list
Id Name State
---------------------------
1 vm1 running
host:~ #
host:~ # virsh domblklist vm2
Target Source
------------------------------------------
vda /opt/vms/vm2/disk0.qcow2
sda /dev/mapper/vm2-data
host:~ #
host:~ # virsh start vm2
Domain vm2 started
host:~ # virsh list
Id Name State
---------------------------
1 vm1 running
2 vm2 running
host:~ # nsenter -t $(qemu-pr-helper pid of vm2) -a bash
host:~ # ls -l /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory
host:~ # exit
exit
host:~ #
vm2:~ # lsscsi
[0:0:0:0] disk FreeNAS lv46 0123 /dev/sda
vm2:~ #
vm2:~ # sg_persist --in -k /dev/sda
FreeNAS lv46 0123
Peripheral device type: disk
PR in (Read keys): Aborted command
Aborted command
vm2:~ #
===================================================================
Any comments will be much appreciated.
Thanks in advance,
Lin