qemu-discuss
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-discuss] Increasing number of threads when using ceph


From: Tyanko Aleksiev
Subject: Re: [Qemu-discuss] Increasing number of threads when using ceph
Date: Fri, 2 Jun 2017 14:45:00 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1

Hi Jan,

On 06/01/2017 06:44 PM, Jan Schermer wrote:
My experience is the same, threads only grow.
The only way to decrease the number of threads is to detach the block device 
completely.

And thank you for the information.

Note that if you need to support VMs with many virtual block devices, then each 
block device will need hundreds of threads and possibly also hundreds of 
connections (depending on how many OSDs you have).
Basically, the qemu process will open new threads and connection(s?) for every 
OSD serving the block device (which is likely all of them).
In practice I think the number was something like:

total = $no_of_osds * $number_of_devices * 3

Thus, in a standard configuration you will then soon run out of PIDs, file 
descriptors, connections, memory...
<rant>... then you'll start loosing sleep, patience and finally data, which is when 
you'll finally start moving to a sane storage backend</rant>

For us this is rather an isolated case than a common issue since it only happens on heavily oversubscribed compute nodes (3 out of 300).

Cheers,
Tyanko


Jan


On 1 Jun 2017, at 15:44, Tyanko Aleksiev <address@hidden> wrote:

Dear QEMU users,

On our OpenStack installation we use QEMU/KVM (v1:2.5 on Ubuntu 14.04, kernel 
3.13) controlled by libvirt (v1.3.1). The storage back-end for the VMs is 
provided via ceph (v0.94.10).

Whenever a new instance is spawned the number of tasks (threads) initially 
started by libvirt-qemu is around 200. Big fraction of those are used to access 
the OSDs of the ceph cluster. Additionally, when some I/O operations are 
performed within the instance the number of threads will increase which is also 
an expected behavior.

However, we noticed is that the number of threads will never decrease over 
time. In particular, I generated some I/O by creating files of different size. 
After the write operation was completed the number of threads did not decrease. 
Then, I deleted the files and reclaimed (discarded/trimed) the space which 
again did not result in any thread being released.

I'm wondering if this is something we should expect? If not, eventually, how 
this issue could be addressed?

Thank you very much for your time and help!

Cheers,
Tyanko

--
Tyanko Aleksiev
System Administrator

Service and Support for Science IT,
University of Zurich
Winterthurerstrasse 190
CH-8057 Zurich Switzerland
Tel: +41 44 635 42 22




--
Tyanko Aleksiev
System Administrator

Service and Support for Science IT,
University of Zurich
Winterthurerstrasse 190
CH-8057 Zurich Switzerland
Tel: +41 44 635 42 22



reply via email to

[Prev in Thread] Current Thread [Next in Thread]