[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores
From: |
Davide Guerri |
Subject: |
Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores |
Date: |
Thu, 14 Mar 2013 19:15:00 +0100 |
Of course I can do some test but a kernel upgrade is not an option here :(
However I don't think this is only a guest problem since it has been triggered
by something between KVM 1.0 and 1.2.
On 14/mar/2013, at 18:50, Michael S. Tsirkin <address@hidden> wrote:
>
> Could be one of the bugs fixed in guest drivers since 2.6.32.
> For example, 39d321577405e8e269fd238b278aaf2425fa788a ?
> Does it help if you try a more recent guest?
>
> On Thu, Mar 14, 2013 at 11:43:30AM +0100, Alexandre DERUMIER wrote:
>> I don't think it's fixed in 1.3 or 1.4, some proxmox users have reported
>> again this bug with guest kernel 2.6.32. (proxmox host is rhel 6.3 kernel +
>> qemu 1.4)
>>
>>
>>
>> ----- Mail original -----
>>
>> De: "Davide Guerri" <address@hidden>
>> À: "Alexandre DERUMIER" <address@hidden>
>> Cc: "Peter Lieven" <address@hidden>, "Michael S. Tsirkin" <address@hidden>,
>> "Stefan Hajnoczi" <address@hidden>, address@hidden, "Jan Kiszka"
>> <address@hidden>, "Peter Lieven" <address@hidden>, "Dietmar Maurer"
>> <address@hidden>
>> Envoyé: Jeudi 14 Mars 2013 10:22:22
>> Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores
>>
>> I'd like to reopen this thread because this problem is still here and it's
>> really annoying.
>> Another possible work-around is to pin the virtio nic irq to one virtual CPU
>> (with /proc/smp_affinity) but (of course) this is still sub-optimal.
>>
>> Here are some graphs showing the performance of a heavy loaded machine after
>> and before a live migration from KVM-1.0 to KVM-1.2.
>> Host is a Ubuntu 12.10 Kernel 3.5, guest is a Ubuntu 10.04.4 LTS Kernel
>> 2.6.32
>>
>> History
>> -------
>> CPU time:
>> http://s1299.beta.photobucket.com/user/dguerri/media/CPU-10days_zps4a088ff0.png.html
>> CPU Load:
>> http://s1299.beta.photobucket.com/user/dguerri/media/Load-10days_zpsff27f212.png.html
>> NET pps:
>> http://s1299.beta.photobucket.com/user/dguerri/media/pps-10days_zps003dd039.png.html
>> NET Mbps:
>> http://s1299.beta.photobucket.com/user/dguerri/media/Mbps-10days_zpsfc3cba8c.png.html
>>
>> Current
>> -------
>> CPU time:
>> http://s1299.beta.photobucket.com/user/dguerri/media/CPU-2days_zpsd362cac6.png.html
>> CPU Load:
>> http://s1299.beta.photobucket.com/user/dguerri/media/load-2days_zpsd4b7b50d.png.html
>> NET pps:
>> http://s1299.beta.photobucket.com/user/dguerri/media/pps-2days_zps8f5458c9.png.html
>> NET Mbps:
>> http://s1299.beta.photobucket.com/user/dguerri/media/Mbps-2days_zps299338b9.png.html
>>
>> Red arrows indicate the kvm version change.
>> Black arrows indicate when I "pinned" the virtio NIC IRQ to only one CPU
>> (that machine has 2 virtual cores).
>> As can be seen the performance penalty is rather high: that machine was
>> almost unusable!
>>
>> Does version 1.3 fixes this issue?
>>
>> Could someone with the required knowledge look into this, please?
>> Please, this is a very nasty bug because I guess I'm not the only one who is
>> unable to upgrade all the machines with a (not-so) old kernel... :)
>>
>> Thanks!
>>
>> Davide Guerri.
>>
>>
>>
>>
>> On 09/dic/2012, at 19:38, Alexandre DERUMIER <address@hidden> wrote:
>>
>>>>> Have you had any further progress on this regression/problem?
>>>
>>> Hi Peter,
>>> I didn't re-tested myself,
>>> but a proxmox user who's have the problem with qemu-kvm 1.2, with windows
>>> guest and linux guest,
>>> don't have the problem anymore with qemu 1.3.
>>>
>>> http://forum.proxmox.com/threads/12157-Win2003R2-in-KVM-VM-is-slow-in-PVE-2-2-when-multiply-CPU-cores-allowed
>>>
>>> I'll try to redone test myself this week
>>>
>>> Regards,
>>>
>>> Alexandre
>>>
>>> ----- Mail original -----
>>>
>>> De: "Peter Lieven" <address@hidden>
>>> À: "Alexandre DERUMIER" <address@hidden>
>>> Cc: "Dietmar Maurer" <address@hidden>, "Stefan Hajnoczi" <address@hidden>,
>>> "Jan Kiszka" <address@hidden>, address@hidden, "Michael S. Tsirkin"
>>> <address@hidden>, "Peter Lieven" <address@hidden>
>>> Envoyé: Lundi 3 Décembre 2012 12:23:11
>>> Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores
>>>
>>>
>>> Am 16.11.2012 um 12:00 schrieb Alexandre DERUMIER <address@hidden>:
>>>
>>>>>> While trying to reproduce the bug, we just detected that it depends on
>>>>>> the hardware (mainboard) you run on.
>>>>>>
>>>>>> Sigh :-/
>>>>
>>>> Hi,
>>>>
>>>> I can reproduce the bug on all my dell servers,differents generation (R710
>>>> (intel),R815 (amd), 2950 (intel).
>>>>
>>>> They all use broadcom bnx2 network card (don't know if it can be related)
>>>>
>>>> host kernel : rhel 63 with 2.6.32 kernel
>>>>
>>>> guest kernel : 2.6.32 (debian squeeze, ubuntu).
>>>>
>>>> No problem with guest kernel 3.2
>>>
>>> Have you had any further progress on this regression/problem?
>>>
>>> Thanks,
>>> Peter
>>>
>>>>
>>>>
>>>>
>>>>
>>>> ----- Mail original -----
>>>>
>>>> De: "Dietmar Maurer" <address@hidden>
>>>> À: "Peter Lieven" <address@hidden>
>>>> Cc: "Stefan Hajnoczi" <address@hidden>, "Peter Lieven" <address@hidden>,
>>>> "Jan Kiszka" <address@hidden>, address@hidden, "Michael S. Tsirkin"
>>>> <address@hidden>
>>>> Envoyé: Vendredi 16 Novembre 2012 11:44:26
>>>> Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple
>>>> cores
>>>>
>>>>>> I only tested with RHEL6.3 kernel on host.
>>>>>
>>>>> can you check if there is a difference on interrupt delivery between those
>>>>> two?
>>>>>
>>>>> cat /proc/interrupts should be sufficient after some traffic has flown.
>>>>
>>>> While trying to reproduce the bug, we just detected that it depends on the
>>>> hardware (mainboard) you run on.
>>>>
>>>> Sigh :-/
>>>
>>>
>
smime.p7s
Description: S/MIME cryptographic signature
- Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores, Davide Guerri, 2013/03/14
- Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores, Alexandre DERUMIER, 2013/03/14
- Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores, Michael S. Tsirkin, 2013/03/14
- Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores,
Davide Guerri <=
- Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores, Peter Lieven, 2013/03/14
- Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores, Davide Guerri, 2013/03/14
- Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores, Peter Lieven, 2013/03/15
- Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores, Michael S. Tsirkin, 2013/03/17
- Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores, Alexandre DERUMIER, 2013/03/18
- Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores, Michael S. Tsirkin, 2013/03/18
- Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores, Peter Lieven, 2013/03/25