Re: [Qemu-devel] qemu_cond_signal() taking a long time to complete.

[Venkateswararao Jujjuri (JV)]

On 10/4/2010 3:40 AM, Arun R Bharadwaj wrote:
> Hi,
>
> I am working on introducing threading model into Qemu. This introduces
> the Threadlets infrastructure which allows subsystems to offload possibly
> blocking work to a queue to be processed by a pool of threads asynchrnonously.
> Threadlets are useful when there are operations that can be performed
> outside the context of the VCPU/IO threads inorder to free these latter
> to service any other guest requests. As of now we have converted a few
> v9fs calls like v9fs_read, v9fs_write etc to this model to test the
> working nature of this model.

My numbers are little different.

I had a patch with only read() happening through threadlet infrastructure.
Everything else is with old code i.e through vCPU thread.

With SMP=1,
Three threads of  dd of=/dev/null if=<file> bs=4k count=100000
Read on threadlet code gave - 21.6 MB/s
Everything on vCPU              - 24.2 MB/s

Not much difference.

With SMP=4
Read on threadlet code gave - 32 MB/s
Everything on vCPU               - 28 MB/s

Here threadlet code is better... still not by much.

In either case, Threadlet code is giving better system time.

vCPU model is taking around 36 sec system time

Typical everything in vCPU model:
real    0m44.885s
user    0m0.130s
sys     0m36.237s

Typical read in threadlet model:
real    0m37.354s
user    0m0.082s
sys     0m6.006s

Thanks,
JV


> I observed that performance is degrading in the threading model for the
> following reason:
>
> Taking the example of v9fs_read call: We submit the blocking work in
> v9fs_read to a queue and do a qemu_cond_signal(). the work will be picked
> up by a worker thread which is waiting on the condition variable to go
> true. I measured the time taken to execute the v9fs_read call; in the
> case without the threading model, it takes around 15microseconds to
> complete, while in the threading model, it takes around 30microsends
> to complete. Most of this extra time (around 22microsends) is spent in
> completing the qemu_cond_signal() call. I suspect this is the reason why
> I am seeing performance hit with the threading model, because this
> time is much more than the time needed to complete the entire
> v9fs_read call in the non threading model case.
>
> I need advice on how to proceed from this situation. Pasting relevant
> code snippets below.
>
> thanks
> arun.
> ---
>
> /* Code to sumbit work to the queue */
> void submit_threadlet_to_queue(ThreadletQueue *queue, ThreadletWork
> *work)
> {
>      qemu_mutex_lock(&(queue->lock));
>      if (queue->idle_threads == 0&&  queue->cur_threads<
> queue->max_threads) {
>          spawn_threadlet(queue);
>      }
>      QTAILQ_INSERT_TAIL(&(queue->request_list), work, node);
>      qemu_cond_signal(&(queue->cond));
>      qemu_mutex_unlock(&(queue->lock));
> }
>
> /* Worker thread code */
> static void *threadlet_worker(void *data)
> {
>      ThreadletQueue *queue = data;
>
>      while (1) {
>          ThreadletWork *work;
>          int ret = 0;
>          qemu_mutex_lock(&(queue->lock));
>
>          while (QTAILQ_EMPTY(&(queue->request_list))&&
>                 (ret != ETIMEDOUT)) {
>              ret = qemu_cond_timedwait(&(queue->cond),
>                                           &(queue->lock), 10*100000);
>          }
>
>          assert(queue->idle_threads != 0);
>          if (QTAILQ_EMPTY(&(queue->request_list))) {
>              if (queue->cur_threads>  queue->min_threads) {
>                  /* We retain the minimum number of threads */
>                  break;
>              }
>          } else {
>              work = QTAILQ_FIRST(&(queue->request_list));
>              QTAILQ_REMOVE(&(queue->request_list), work, node);
>
>              queue->idle_threads--;
>              qemu_mutex_unlock(&(queue->lock));
>
>              /* execute the work function */
>              work->func(work);
>
>              qemu_mutex_lock(&(queue->lock));
>              queue->idle_threads++;
>          }
>          qemu_mutex_unlock(&(queue->lock));
>      }
>
>      queue->idle_threads--;
>      queue->cur_threads--;
>
>      return NULL;
> }