[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [raw] Guest stuck during live live-migration
|
From: |
Quentin Grolleau |
|
Subject: |
RE: [raw] Guest stuck during live live-migration |
|
Date: |
Tue, 24 Nov 2020 12:58:53 +0000 |
Thanks Kevin,
> > Hello,
> >
> > In our company, we are hosting a large number of Vm, hosted behind
> > Openstack (so libvirt/qemu).
> > A large majority of our Vms are runnign with local data only, stored on
> > NVME, and most of them are RAW disks.
> >
> > With Qemu 4.0 (can be even with older version) we see strange
> > live-migration comportement:
> First of all, 4.0 is relatively old. Generally it is worth retrying with
> the most recent code (git master or 5.2.0-rc2) before having a closer
> look at problems, because it is frustrating to spend considerable time
> debugging an issue and then find out it has already been fixed a year
> ago.
> I will try to build it the most recent code
I will try to build with the most recent code, but it will take me some time to
do it
> > - some Vms live migrate at very high speed without issue (> 6 Gbps)
> > - some Vms are running correctly, but migrating at a strange low speed
> > (3Gbps)
> > - some Vms are migrating at a very low speed (1Gbps, sometime less) and
> > during the migration the guest is completely I/O stuck
> >
> > When this issue happen the VM is completly block, iostat in the Vm show us
> > a latency of 30 secs
> Can you get the stack backtraces of all QEMU threads while the VM is
> blocked (e.g. with gdb or pstack)?
(gdb) thread apply all bt
Thread 20 (Thread 0x7f8a0effd700 (LWP 201248)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000056520139878b in qemu_cond_wait_impl (cond=0x5652020f27b0,
mutex=0x5652020f27e8, file=0x5652014e4178
"/root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c", line=214) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:161
#2 0x00005652012a264d in vnc_worker_thread_loop
(queue=queue@entry=0x5652020f27b0) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c:214
#3 0x00005652012a2c18 in vnc_worker_thread (arg=arg@entry=0x5652020f27b0) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c:324
#4 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#5 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a0effd700) at
pthread_create.c:333
#6 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 19 (Thread 0x7f8a0ffff700 (LWP 201222)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x565202354480) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x565202354480)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a0ffff700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 18 (Thread 0x7f8a2cff9700 (LWP 201221)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x565202331320) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x565202331320)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a2cff9700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 17 (Thread 0x7f8a2d7fa700 (LWP 201220)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x56520230e1c0) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x56520230e1c0)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a2d7fa700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 16 (Thread 0x7f8a2dffb700 (LWP 201219)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x5652022ea870) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x5652022ea870)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a2dffb700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 15 (Thread 0x7f8a2e7fc700 (LWP 201218)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x5652022c7710) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x5652022c7710)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a2e7fc700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 14 (Thread 0x7f8a2effd700 (LWP 201217)):
---Type <return> to continue, or q <return> to quit---
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x5652022a45b0) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x5652022a45b0)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a2effd700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 13 (Thread 0x7f8a2f7fe700 (LWP 201216)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x565202281450) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x565202281450)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a2f7fe700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 12 (Thread 0x7f8a2ffff700 (LWP 201214)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x56520225e2f0) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x56520225e2f0)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a2ffff700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 11 (Thread 0x7f8a50ff9700 (LWP 201213)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013c54c0
"/root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c", line=3243) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013c54c0
"/root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c", line=line@entry=3243) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200f3a9a1 in prepare_mmio_access (mr=0x565202db4c10,
mr=0x565202db4c10) at /root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c:3243
#5 0x0000565200f3fe0f in flatview_read_continue (fv=fv@entry=0x7f8a34034250,
addr=addr@entry=49222, attrs=attrs@entry=..., buf=<optimized out>,
buf@entry=0x7f8a6329a000 "", len=len@entry=2, addr1=6, l=2, mr=0x565202db4c10)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c:3338
#6 0x0000565200f40003 in flatview_read (fv=0x7f8a34034250, addr=49222,
attrs=..., buf=0x7f8a6329a000 "", len=2) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c:3378
#7 0x0000565200f4011b in address_space_read_full (as=<optimized out>,
addr=addr@entry=49222, attrs=..., buf=buf@entry=0x7f8a6329a000 "",
len=len@entry=2) at /root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c:3391
#8 0x0000565200f4023a in address_space_rw (as=<optimized out>,
addr=addr@entry=49222, attrs=..., attrs@entry=..., buf=buf@entry=0x7f8a6329a000
"", len=len@entry=2, is_write=is_write@entry=false) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c:3421
#9 0x0000565200fa7d9b in kvm_handle_io (count=1, size=2, direction=<optimized
out>, data=<optimized out>, attrs=..., port=49222) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:1778
#10 kvm_cpu_exec (cpu=cpu@entry=0x56520223b1d0) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2024
#11 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x56520223b1d0)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#12 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#13 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a50ff9700) at
pthread_create.c:333
#14 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 10 (Thread 0x7f8a517fa700 (LWP 201212)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x565202217c30) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x565202217c30)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a517fa700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 9 (Thread 0x7f8a51ffb700 (LWP 201211)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
---Type <return> to continue, or q <return> to quit---
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x5652021f4ad0) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x5652021f4ad0)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a51ffb700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 8 (Thread 0x7f8a527fc700 (LWP 201210)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x5652021d1970) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x5652021d1970)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a527fc700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 7 (Thread 0x7f8a52ffd700 (LWP 201208)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x5652021ae630) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x5652021ae630)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a52ffd700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 6 (Thread 0x7f8a537fe700 (LWP 201207)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x56520218b4d0) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x56520218b4d0)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a537fe700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 5 (Thread 0x7f8a53fff700 (LWP 201206)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013d7c68
"/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c",
line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x565202167a50) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089
#5 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x565202167a50)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#6 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#7 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a53fff700) at
pthread_create.c:333
#8 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 4 (Thread 0x7f8a58fc0700 (LWP 201204)):
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f8a5e24cdbd in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x565201adb680 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x0000565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680
<qemu_global_mutex>, file=0x5652013c54c0
"/root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c", line=3243) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66
#3 0x0000565200f7d00e in qemu_mutex_lock_iothread_impl
(file=file@entry=0x5652013c54c0
"/root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c", line=line@entry=3243) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850
#4 0x0000565200f3a9a1 in prepare_mmio_access (mr=0x56520307d930,
mr=0x56520307d930) at /root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c:3243
#5 0x0000565200f3fe0f in flatview_read_continue (fv=fv@entry=0x7f8a34075e00,
addr=addr@entry=4261433344, attrs=attrs@entry=..., buf=<optimized out>,
buf@entry=0x7f8a633ee028 "\002", len=len@entry=1, addr1=0, l=1,
mr=0x56520307d930)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c:3338
#6 0x0000565200f40003 in flatview_read (fv=0x7f8a34075e00, addr=4261433344,
attrs=..., buf=0x7f8a633ee028 "\002", len=1) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c:3378
#7 0x0000565200f4011b in address_space_read_full (as=<optimized out>,
addr=<optimized out>, attrs=..., buf=buf@entry=0x7f8a633ee028 "\002",
len=<optimized out>) at /root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c:3391
#8 0x0000565200f4023a in address_space_rw (as=<optimized out>, addr=<optimized
out>, attrs=..., attrs@entry=..., buf=buf@entry=0x7f8a633ee028 "\002",
len=<optimized out>, is_write=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/exec.c:3421
#9 0x0000565200fa7cf8 in kvm_cpu_exec (cpu=cpu@entry=0x565202119100) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2034
#10 0x0000565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x565202119100)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281
#11 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#12 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a58fc0700) at
pthread_create.c:333
#13 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
---Type <return> to continue, or q <return> to quit---
Thread 3 (Thread 0x7f8a597c1700 (LWP 201203)):
#0 0x00007f8a5df7474d in poll () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f8a6234138c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00007f8a62341712 in g_main_loop_run () from
/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x00005652010aa3b1 in iothread_run (opaque=opaque@entry=0x565202108420) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/iothread.c:82
#4 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#5 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a597c1700) at
pthread_create.c:333
#6 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 2 (Thread 0x7f8a5a903700 (LWP 201196)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x0000565201398caf in qemu_futex_wait (val=<optimized out>, f=<optimized
out>) at /root/qemu_debug_LSEEK/qemu_debug/qemu/include/qemu/futex.h:29
#2 qemu_event_wait (ev=ev@entry=0x565201b2da88 <rcu_call_ready_event>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:442
#3 0x00005652013a9868 in call_rcu_thread (opaque=opaque@entry=0x0) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/rcu.c:261
#4 0x0000565201398116 in qemu_thread_start (args=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502
#5 0x00007f8a5e24a6ba in start_thread (arg=0x7f8a5a903700) at
pthread_create.c:333
#6 0x00007f8a5df8041d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 1 (Thread 0x7f8a633acbc0 (LWP 201184)):
#0 0x00007f8a5e253b67 in lseek64 () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000565201301278 in find_allocation (bs=0x565202109ca0, hole=<synthetic
pointer>, data=<synthetic pointer>, start=5738856448) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/block/file-posix.c:2427
#2 raw_co_block_status (bs=0x565202109ca0, want_zero=<optimized out>,
offset=5738856448, bytes=15728640, pnum=0x7f7b529e9f20, map=0x7f7b529e9ca0,
file=0x7f7b529e9ca8) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/block/file-posix.c:2490
#3 0x00005652013078b3 in bdrv_co_block_status (bs=0x565202109ca0,
want_zero=want_zero@entry=true, offset=5738856448, bytes=15728640,
pnum=pnum@entry=0x7f7b529e9f20, map=map@entry=0x7f7b529e9d50,
file=0x7f7b529e9d58)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/block/io.c:2110
#4 0x00005652013079c7 in bdrv_co_block_status (bs=bs@entry=0x565202102640,
want_zero=want_zero@entry=true, offset=offset@entry=5738856448,
bytes=bytes@entry=15728640, pnum=pnum@entry=0x7f7b529e9f20, map=map@entry=0x0,
file=0x0)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/block/io.c:2134
#5 0x0000565201307c14 in bdrv_co_block_status_above (file=0x0, map=0x0,
pnum=0x7f7b529e9f20, bytes=15728640, offset=5738856448, want_zero=true,
base=0x0, bs=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/block/io.c:2212
#6 bdrv_block_status_above_co_entry (opaque=0x7f7b529e9e20) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/block/io.c:2242
#7 0x0000565201308198 in bdrv_common_block_status_above
(bs=bs@entry=0x565202102640, base=base@entry=0x0,
want_zero=want_zero@entry=true, offset=offset@entry=5738856448,
bytes=<optimized out>, pnum=pnum@entry=0x7f7b529e9f20, map=0x0, file=0x0)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/block/io.c:2277
#8 0x000056520130918d in bdrv_block_status_above (bs=bs@entry=0x565202102640,
base=base@entry=0x0, offset=offset@entry=5738856448, bytes=<optimized out>,
pnum=pnum@entry=0x7f7b529e9f20, map=map@entry=0x0, file=0x0)
at /root/qemu_debug_LSEEK/qemu_debug/qemu/block/io.c:2290
#9 0x00005652013057c0 in mirror_iteration (s=0x565202fd66a0) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/block/mirror.c:526
#10 mirror_run (job=0x565202fd66a0, errp=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/block/mirror.c:968
#11 0x00005652012b913f in job_co_entry (opaque=0x565202fd66a0) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/job.c:889
#12 0x00005652013aafca in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>) at
/root/qemu_debug_LSEEK/qemu_debug/qemu/util/coroutine-ucontext.c:116
#13 0x00007f8a5dec35d0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#14 0x00007ffff4114730 in ?? ()
#15 0x0000000000000000 in ?? ()
> > First we thought it was related to an hardware issue we check it, we
> > comparing different hardware, but no issue where found there
> >
> > So one of my colleague had the idea to limit with "tc" the bandwidth on the
> > interface the migration was done, and it worked the VM didn't lose any ping
> > nor being I/O stuck
> > Important point : Once the Vm have been migrate (with the limitation ) one
> > time, if we migrate it again right after, the migration will be done at
> > full speed (8-9Gb/s) without freezing the Vm
> Since you say you're using local storage, I assume that you're doing
> both a VM live migration and storage migration at the same time. These
> are separate connections, storage is migrated using a NBD connection.
> Did you limit the bandwith for both connections, or if it was just one
> of them, which one was it?
Yes we are doing a live migration and a storage migration (pre-copy)
We limit the bandwith for both connections but when the disk have finish to
migrate we can remove the bandwith limitation
for the memory transfert and it go up to the maximum speed and with no freeze
> > It only happen on existing VM, we tried to replicate with a fresh instance
> > with exactly the same spec and nothing was happening
> >
> > We tried to replicate the workload inside the VM but there was no way to
> > replicate the case. So it was not related to the workload nor to the server
> > that hosts the Vm
> >
> > So we thought about the disk of the instance : the raw file.
> >
> > We also tried to strace -c the process during the live-migration and it was
> > doing a lot of "lseek"
> >
> > and we found this :
> > https://lists.gnu.org/archive/html/qemu-devel/2017-02/msg00462.html
> This case is different in that it used qcow2 (which should behave much
> better today).
> It also used ZFS, which you didn't mention. Is the problematic image
> stored on ZFS? If not, which filesystem is it?
You are right this is not exactly the same case and we are not storing the
problematic image on ZFS but on Ext4
> So i rebuilt Qemu with this patch and the live-migration went well, at high
> speed and with no VM freeze
> ( https://github.com/qemu/qemu/blob/master/block/file-posix.c#L2601 )
>
> Do you have a way to avoid the "lseek" mechanism as it consumes more
> resources to find the holes in the disk and don't let any for the VM ?
> If you can provide the stack trace during the hang, we might be able to
> tell why we're even trying to find holes.
> Please also provide your QEMU command line.
Here the qemu command line :
/usr/bin/qemu-system-x86_64 \
-name guest=instance-0034d494,debug-threads=on \
-S \
-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-9-instance-0034d494/master-key.aes
\
-machine pc-i440fx-bionic,accel=kvm,usb=off,dump-guest-core=off \
-cpu Broadwell-IBRS,md-clear=on,vmx=on \
-m 60000 \
-overcommit mem-lock=off \
-smp 16,sockets=16,cores=1,threads=1 \
-uuid b959a460-84b0-4280-b851-96755027622b \
-smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack
Nova,version=14.1.1,serial=5b429103-2856-154f-1caf-5ffb5694cdc3,uuid=b959a460-84b0-4280-b851-96755027622b,family=Virtual
Machine' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=28,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-drive
file=/home/instances/b959a460-84b0-4280-b851-96755027622b/disk,format=raw,if=none,id=drive-virtio-disk0,cache=none,discard=unmap,aio=native,throttling.iops-total=80000
\
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on
\
-add-fd set=1,fd=31 \
-chardev file,id=charserial0,path=/dev/fdset/1,append=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev pty,id=charserial1 \
-device isa-serial,chardev=charserial1,id=serial1 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 10.224.27.81:0 \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 \
-s \
-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
> At the moment, my assumption is that this is during a mirror block job
> which is migrating the disk to your destination server. Not looking for
> holes would mean that a sparse source file would become fully allocated
> on the destination, which is usually not wanted (also we would
> potentially transfer a lot more data over the networkj).
The vm disk is already fully allocated, so in this case it doesn't change.
> Can you give us a snippet from your strace that shows the individual
> lseek syscalls? Depending on which ranges are queried, maybe we could
> optimise things by caching the previous result.
Here the strace during the block migration :
strace -c -p 32571
strace: Process 32571 attached
^Cstrace: Process 32571 detached
% time seconds usecs/call calls errors syscall
94.15 13.754159 2503 5495 lseek
3.47 0.507101 91 5549 sendmsg
1.60 0.233516 84 2769 io_submit
0.40 0.057817 11 5496 setsockopt
0.18 0.025747 4 5730 184 recvmsg
0.16 0.023560 6 4259 write
0.02 0.002575 6 408 read
0.02 0.002425 9 266 35 futex
0.01 0.002136 12 184 ppoll
0.00 0.000184 12 16 poll
0.00 0.000038 38 1 clone
0.00 0.000032 16 2 rt_sigprocmask
0.00 0.000013 13 1 ioctl
100.00 14.609303 30176 219 total
> Also, a final remark, I know of some cases (on XFS) where lseeks were
> slow because the image file was heavily fragmented. Defragmenting the
> file resolved the problem, so this may be another thing to try.
> On XFS, newer QEMU versions set an extent size hint on newly created
> image files (during qemu-img create), which can reduce fragmentation
> considerably.
> Kevin
> Server hosting the VM :
> - Bi-Xeon hosts With NVME storage and 10 Go Network card
> - Qemu 4.0 And Libvirt 5.4
> - Kernel 4.18.0.25
>
> Guest having the issue :
> - raw image with Debian 8
>
> Here the qemu img on the disk :
> > qemu-img info disk
> image: disk
> file format: raw
> virtual size: 400G (429496729600 bytes)
> disk size: 400G
>
>
> Quentin GROLLEAU
>