Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores

[Michael S. Tsirkin]

Could be one of the bugs fixed in guest drivers since 2.6.32.
For example, 39d321577405e8e269fd238b278aaf2425fa788a ?
Does it help if you try a more recent guest?

On Thu, Mar 14, 2013 at 11:43:30AM +0100, Alexandre DERUMIER wrote:
> I don't think it's fixed in 1.3 or 1.4, some proxmox users have reported 
> again this bug with guest kernel 2.6.32. (proxmox host is rhel 6.3 kernel + 
> qemu 1.4)
> 
> 
> 
> ----- Mail original -----
> 
> De: "Davide Guerri" <address@hidden>
> À: "Alexandre DERUMIER" <address@hidden>
> Cc: "Peter Lieven" <address@hidden>, "Michael S. Tsirkin" <address@hidden>, 
> "Stefan Hajnoczi" <address@hidden>, address@hidden, "Jan Kiszka" 
> <address@hidden>, "Peter Lieven" <address@hidden>, "Dietmar Maurer" 
> <address@hidden>
> Envoyé: Jeudi 14 Mars 2013 10:22:22
> Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores
> 
> I'd like to reopen this thread because this problem is still here and it's 
> really annoying.
> Another possible work-around is to pin the virtio nic irq to one virtual CPU 
> (with /proc/smp_affinity) but (of course) this is still sub-optimal.
> 
> Here are some graphs showing the performance of a heavy loaded machine after 
> and before a live migration from KVM-1.0 to KVM-1.2.
> Host is a Ubuntu 12.10 Kernel 3.5, guest is a Ubuntu 10.04.4 LTS Kernel 2.6.32
> 
> History
> -------
> CPU time: 
> http://s1299.beta.photobucket.com/user/dguerri/media/CPU-10days_zps4a088ff0.png.html
> CPU Load: 
> http://s1299.beta.photobucket.com/user/dguerri/media/Load-10days_zpsff27f212.png.html
> NET pps: 
> http://s1299.beta.photobucket.com/user/dguerri/media/pps-10days_zps003dd039.png.html
> NET Mbps: 
> http://s1299.beta.photobucket.com/user/dguerri/media/Mbps-10days_zpsfc3cba8c.png.html
> 
> Current
> -------
> CPU time: 
> http://s1299.beta.photobucket.com/user/dguerri/media/CPU-2days_zpsd362cac6.png.html
> CPU Load: 
> http://s1299.beta.photobucket.com/user/dguerri/media/load-2days_zpsd4b7b50d.png.html
> NET pps: 
> http://s1299.beta.photobucket.com/user/dguerri/media/pps-2days_zps8f5458c9.png.html
> NET Mbps: 
> http://s1299.beta.photobucket.com/user/dguerri/media/Mbps-2days_zps299338b9.png.html
> 
> Red arrows indicate the kvm version change.
> Black arrows indicate when I "pinned" the virtio NIC IRQ to only one CPU 
> (that machine has 2 virtual cores).
> As can be seen the performance penalty is rather high: that machine was 
> almost unusable!
> 
> Does version 1.3 fixes this issue?
> 
> Could someone with the required knowledge look into this, please?
> Please, this is a very nasty bug because I guess I'm not the only one who is 
> unable to upgrade all the machines with a (not-so) old kernel... :)
> 
> Thanks!
> 
> Davide Guerri.
> 
> 
> 
> 
> On 09/dic/2012, at 19:38, Alexandre DERUMIER <address@hidden> wrote:
> 
> >>> Have you had any further progress on this regression/problem?
> >
> > Hi Peter,
> > I didn't re-tested myself,
> > but a proxmox user who's have the problem with qemu-kvm 1.2, with windows 
> > guest and linux guest,
> > don't have the problem anymore with qemu 1.3.
> >
> > http://forum.proxmox.com/threads/12157-Win2003R2-in-KVM-VM-is-slow-in-PVE-2-2-when-multiply-CPU-cores-allowed
> >
> > I'll try to redone test myself this week
> >
> > Regards,
> >
> > Alexandre
> >
> > ----- Mail original -----
> >
> > De: "Peter Lieven" <address@hidden>
> > À: "Alexandre DERUMIER" <address@hidden>
> > Cc: "Dietmar Maurer" <address@hidden>, "Stefan Hajnoczi" <address@hidden>, 
> > "Jan Kiszka" <address@hidden>, address@hidden, "Michael S. Tsirkin" 
> > <address@hidden>, "Peter Lieven" <address@hidden>
> > Envoyé: Lundi 3 Décembre 2012 12:23:11
> > Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores
> >
> >
> > Am 16.11.2012 um 12:00 schrieb Alexandre DERUMIER <address@hidden>: 
> >
> >>>> While trying to reproduce the bug, we just detected that it depends on 
> >>>> the hardware (mainboard) you run on.
> >>>>
> >>>> Sigh :-/
> >>
> >> Hi,
> >>
> >> I can reproduce the bug on all my dell servers,differents generation (R710 
> >> (intel),R815 (amd), 2950 (intel).
> >>
> >> They all use broadcom bnx2 network card (don't know if it can be related)
> >>
> >> host kernel : rhel 63 with 2.6.32 kernel
> >>
> >> guest kernel : 2.6.32 (debian squeeze, ubuntu).
> >>
> >> No problem with guest kernel 3.2
> >
> > Have you had any further progress on this regression/problem?
> >
> > Thanks,
> > Peter
> >
> >>
> >>
> >>
> >>
> >> ----- Mail original -----
> >>
> >> De: "Dietmar Maurer" <address@hidden>
> >> À: "Peter Lieven" <address@hidden>
> >> Cc: "Stefan Hajnoczi" <address@hidden>, "Peter Lieven" <address@hidden>, 
> >> "Jan Kiszka" <address@hidden>, address@hidden, "Michael S. Tsirkin" 
> >> <address@hidden>
> >> Envoyé: Vendredi 16 Novembre 2012 11:44:26
> >> Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple 
> >> cores
> >>
> >>>> I only tested with RHEL6.3 kernel on host.
> >>>
> >>> can you check if there is a difference on interrupt delivery between those
> >>> two?
> >>>
> >>> cat /proc/interrupts should be sufficient after some traffic has flown.
> >>
> >> While trying to reproduce the bug, we just detected that it depends on the 
> >> hardware (mainboard) you run on.
> >>
> >> Sigh :-/
> >
> >