Hello all,
we are using your qemu packages to deploy qemu VMs on our gluster via gfsapi.
Recent upgrade broken our qemu and we are not able to deploy / start VMs
anymore.
Gluster is running OK, mounted with FUSE, everything looks ok, there is
probably some problem with qemu while accessing gluster with gfsapi.
These are our current versions (Qemu is from
https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.7 ):
ii glusterfs-client 3.7.17-ubuntu1~trusty2
amd64 clustered file-system (client
package)
ii glusterfs-common 3.7.17-ubuntu1~trusty2
amd64 GlusterFS common libraries and translator
modules
ii glusterfs-server 3.7.17-ubuntu1~trusty2
amd64 clustered file-system (server package)
ii qemu-keymaps
2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 all QEMU keyboard maps
ii qemu-kvm
2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU Full
virtualization
ii qemu-system-common
2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU full system
emulation binaries (common files)
ii qemu-system-x86
2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU full system
emulation binaries (x86)
ii qemu-utils
2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU utilities
We see error attached lower in mail. Do you have any suggestions what can cause
this problem ?
Thanks in advance for your help.
Regards,
Martin
Volume Name: vmvol
Type: Replicate
Volume ID: a72b5c9e-b8ff-488e-b10f-5ba4b71e62b8
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: node1.storage.internal:/gluster/vmvol/brick1
Brick2: node2.storage.internal:/gluster/vmvol/brick1
Brick3: node3.storage.internal:/gluster/vmvol/brick1
Options Reconfigured:
cluster.self-heal-daemon: on
nfs.disable: on
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: enable
performance.stat-prefetch: on
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
cluster.eager-lock: enable
storage.owner-gid: 9869
storage.owner-uid: 9869
server.allow-insecure: on
performance.readdir-ahead: on
2016-11-25 12:54:06.121+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name one-67 -S -machine
pc-i440fx-trusty,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp
1,sockets=1,cores=1,threads=1 -uuid 3c703b26-3f57-44d0-8d76-bb281fd8902c
-no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-67.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=gluster://node1:24007/vmvol/67/disk.1,if=none,id=drive-ide0-0-0,format=qcow2,cache=none
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2
-drive
file=/var/lib/one//datastores/112/67/disk.0,if=none,id=drive-ide0-0-1,readonly=on,format=raw,cache=none
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=3
-drive
file=/var/lib/one//datastores/112/67/disk.2,if=none,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=02:00:0a:c8:64:1f,bus=pci.0,addr=0x3,bootindex=1
-vnc 0.0.0.0:67 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
[2016-11-25 12:54:06.254217] I [MSGID: 104045] [glfs-master.c:96:notify]
0-gfapi: New graph 6e6f6465-322d-3231-3330-392d32303136 (0) coming up
[2016-11-25 12:54:06.254246] I [MSGID: 114020] [client.c:2113:notify]
0-vmvol-client-0: parent translators are ready, attempting connect on transport
[2016-11-25 12:54:06.254473] I [MSGID: 114020] [client.c:2113:notify]
0-vmvol-client-1: parent translators are ready, attempting connect on transport
[2016-11-25 12:54:06.254672] I [MSGID: 114020] [client.c:2113:notify]
0-vmvol-client-2: parent translators are ready, attempting connect on transport
[2016-11-25 12:54:06.254844] I [rpc-clnt.c:1960:rpc_clnt_reconfig]
0-vmvol-client-0: changing port to 49152 (from 0)
[2016-11-25 12:54:06.255303] I [MSGID: 114057]
[client-handshake.c:1437:select_server_supported_programs] 0-vmvol-client-0:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-11-25 12:54:06.255391] I [rpc-clnt.c:1960:rpc_clnt_reconfig]
0-vmvol-client-2: changing port to 49152 (from 0)
[2016-11-25 12:54:06.255844] I [MSGID: 114057]
[client-handshake.c:1437:select_server_supported_programs] 0-vmvol-client-2:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-11-25 12:54:06.256259] I [MSGID: 114046]
[client-handshake.c:1213:client_setvolume_cbk] 0-vmvol-client-2: Connected to
vmvol-client-2, attached to remote volume '/gluster/vmvol/brick1'.
[2016-11-25 12:54:06.256268] I [MSGID: 114047]
[client-handshake.c:1224:client_setvolume_cbk] 0-vmvol-client-2: Server and
Client lk-version numbers are not same, reopening the fds
[2016-11-25 12:54:06.256288] I [MSGID: 108005] [afr-common.c:4299:afr_notify]
0-vmvol-replicate-0: Subvolume 'vmvol-client-2' came back up; going online.
[2016-11-25 12:54:06.256464] I [MSGID: 114035]
[client-handshake.c:193:client_set_lk_version_cbk] 0-vmvol-client-2: Server lk
version = 1
[2016-11-25 12:54:06.262716] I [MSGID: 114046]
[client-handshake.c:1213:client_setvolume_cbk] 0-vmvol-client-0: Connected to
vmvol-client-0, attached to remote volume '/gluster/vmvol/brick1'.
[2016-11-25 12:54:06.262729] I [MSGID: 114047]
[client-handshake.c:1224:client_setvolume_cbk] 0-vmvol-client-0: Server and
Client lk-version numbers are not same, reopening the fds
[2016-11-25 12:54:06.262863] I [MSGID: 114035]
[client-handshake.c:193:client_set_lk_version_cbk] 0-vmvol-client-0: Server lk
version = 1
[2016-11-25 12:54:06.267906] I [rpc-clnt.c:1960:rpc_clnt_reconfig]
0-vmvol-client-1: changing port to 49152 (from 0)
[2016-11-25 12:54:06.268148] I [MSGID: 114057]
[client-handshake.c:1437:select_server_supported_programs] 0-vmvol-client-1:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-11-25 12:54:06.287724] I [MSGID: 114046]
[client-handshake.c:1213:client_setvolume_cbk] 0-vmvol-client-1: Connected to
vmvol-client-1, attached to remote volume '/gluster/vmvol/brick1'.
[2016-11-25 12:54:06.287734] I [MSGID: 114047]
[client-handshake.c:1224:client_setvolume_cbk] 0-vmvol-client-1: Server and
Client lk-version numbers are not same, reopening the fds
[2016-11-25 12:54:06.313439] I [MSGID: 114035]
[client-handshake.c:193:client_set_lk_version_cbk] 0-vmvol-client-1: Server lk
version = 1
[2016-11-25 12:54:06.314113] I [MSGID: 108031]
[afr-common.c:2071:afr_local_discovery_cbk] 0-vmvol-replicate-0: selecting
local read_child vmvol-client-1
[2016-11-25 12:54:06.314928] I [MSGID: 104041]
[glfs-resolve.c:890:__glfs_active_subvol] 0-vmvol: switched to graph
6e6f6465-322d-3231-3330-392d32303136 (0)
[2016-11-25 12:54:16.331479] I [MSGID: 114021] [client.c:2122:notify]
0-vmvol-client-0: current graph is no longer active, destroying rpc_client
[2016-11-25 12:54:16.331507] I [MSGID: 114021] [client.c:2122:notify]
0-vmvol-client-1: current graph is no longer active, destroying rpc_client
[2016-11-25 12:54:16.331517] I [MSGID: 114018]
[client.c:2037:client_rpc_notify] 0-vmvol-client-0: disconnected from
vmvol-client-0. Client process will keep trying to connect to glusterd until
brick's port is available
[2016-11-25 12:54:16.331531] I [MSGID: 114018]
[client.c:2037:client_rpc_notify] 0-vmvol-client-1: disconnected from
vmvol-client-1. Client process will keep trying to connect to glusterd until
brick's port is available
[2016-11-25 12:54:16.331534] I [MSGID: 114021] [client.c:2122:notify]
0-vmvol-client-2: current graph is no longer active, destroying rpc_client
[2016-11-25 12:54:16.331543] W [MSGID: 108001] [afr-common.c:4379:afr_notify]
0-vmvol-replicate-0: Client-quorum is not met
[2016-11-25 12:54:16.331667] E [rpc-clnt.c:370:saved_frames_unwind] (-->
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f4dcee94502] (-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f4dcec642be] (-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f4dcec643ce] (-->
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f4dcec65aa4]
(--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f4dcec66360] )))))
0-vmvol-client-2: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at
2016-11-25 12:54:16.331128 (xid=0x2f62)
[2016-11-25 12:54:16.331684] W [MSGID: 114031]
[client-rpc-fops.c:907:client3_3_writev_cbk] 0-vmvol-client-2: remote operation
failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331701] I [MSGID: 114018]
[client.c:2037:client_rpc_notify] 0-vmvol-client-2: disconnected from
vmvol-client-2. Client process will keep trying to connect to glusterd until
brick's port is available
[2016-11-25 12:54:16.331710] E [MSGID: 108006] [afr-common.c:4321:afr_notify]
0-vmvol-replicate-0: All subvolumes are down. Going offline until atleast one
of them comes back up.
[2016-11-25 12:54:16.331712] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-0: remote
operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331727] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-1: remote
operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331735] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-2: remote
operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331749] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-0: remote
operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331756] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-1: remote
operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331762] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-2: remote
operation failed [Transport endpoint is not connected]
[2016-11-25 12:54:16.331768] W [inode.c:1814:inode_table_destroy]
(-->/usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_fini+0x3cf) [0x7f4dd57ef49f]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy_all+0x51)
[0x7f4dceebe4d1]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy+0xd4)
[0x7f4dceebe3d4] ) 0-gfapi: Active inode(0x7f4db83101a4) with refcount(1) found during
cleanup
[2016-11-25 12:54:16.331834] E [inode.c:468:__inode_unref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.17/xlator/cluster/replicate.so(afr_local_cleanup+0x128)
[0x7f4dbb5dafd8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x21)
[0x7f4dceebc7d1] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x4a0b6)
[0x7f4dceebc0b6] ) 0-: Assertion failed: inode->ref
[2016-11-25 12:54:16.331876] I [MSGID: 101053]
[mem-pool.c:636:mem_pool_destroy] 0-gfapi: size=84 max=2 total=2
[2016-11-25 12:54:16.331924] I [MSGID: 101053]
[mem-pool.c:636:mem_pool_destroy] 0-gfapi: size=156 max=3 total=3
[2016-11-25 12:54:16.332037] I [MSGID: 101053]
[mem-pool.c:636:mem_pool_destroy] 0-gfapi: size=108 max=2 total=2
[2016-11-25 12:54:16.332046] I [MSGID: 101053]
[mem-pool.c:636:mem_pool_destroy] 0-vmvol-client-0: size=1300 max=2 total=3038
[2016-11-25 12:54:16.332054] I [MSGID: 101053]
[mem-pool.c:636:mem_pool_destroy] 0-vmvol-client-1: size=1300 max=2 total=3039
[2016-11-25 12:54:16.332060] I [MSGID: 101053]
[mem-pool.c:636:mem_pool_destroy] 0-vmvol-client-2: size=1300 max=3 total=3038
[2016-11-25 12:54:16.332068] I [MSGID: 101053]
[mem-pool.c:636:mem_pool_destroy] 0-vmvol-replicate-0: size=10556 max=4
total=3038
[2016-11-25 12:54:16.332107] I [MSGID: 101053]
[mem-pool.c:636:mem_pool_destroy] 0-vmvol-dht: size=1148 max=0 total=0
[2016-11-25 12:54:16.332141] I [MSGID: 101053]
[mem-pool.c:636:mem_pool_destroy] 0-vmvol-dht: size=2316 max=2 total=6
[2016-11-25 12:54:16.332200] I [MSGID: 101053]
[mem-pool.c:636:mem_pool_destroy] 0-vmvol-readdir-ahead: size=60 max=0 total=0
[2016-11-25 12:54:16.332212] I [io-stats.c:2951:fini] 0-vmvol: io-stats
translator unloaded
[2016-11-25 12:54:16.332245] I [MSGID: 101191]
[event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with
index 1
[2016-11-25 12:54:16.332259] I [MSGID: 101191]
[event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with
index 2
qemu-system-x86_64: -drive
file=gluster://node1:24007/vmvol/67/disk.1,if=none,id=drive-ide0-0-0,format=qcow2,cache=none:
could not open disk image gluster://node1:24007/vmvol/67/disk.1: Could not
read L1 table: Bad file descriptor
2016-11-25 12:54:17.199+0000: shutting down