Re: [Qemu-devel] Bug#668594: qemu-kvm: Suboptimal virtio/vhost-net performance on Debian KVM hosts compared to others

[Michael Tokarev]

On 13.04.2012 13:46, Hans-Kristian Bakke wrote:
> Package: qemu-kvm
> Version: 1.0+dfsg-9
> Severity: normal
> 
> I cannot get optimal network througput on KVM guest using Debian Wheezy (and 
> stable) as KVM host.
> It is not horribly bad, just not good compared to relevant alternatives.
> 
> I have tried Ubuntu Server 11.10, Proxmox 1.9, Proxmox 2.0 and Fedora 17 
> Alpha as 1:1 replacement for the Debian KVM host using the same guests (just 
> preserving the LVM volumes between installs) and they all manage about 20 
> gbit/s guest to guest using a simple iperf test, while Debian only manages 
> about 2.3 gbit/s with high CPU usage. The CPU usage is generally much higher 
> on all guest network activity on Debian and are in some cases not able to 
> even saturate a gigabit link (when coming from other subnet) without maxing a 
> CPU core where the other KVM hosts barely breaks a sweat.
> 
> Disc IO is very good and the guests feels snappy so it doesn't seem like 
> there is something really wrong, just something suboptimal with the 
> networking.
> The issue follows only the host OS as the guests have been the same in all 
> comparisons (Debian Wheezy)
> 
> 
> To reproduce:
> ------------
> Install Debian Wheezy in guests (minimal with SSH and ntp)
> Install iperf via apt-get
> Configure network
> 
> Run test:
> guest1: iperf -s
> guest2: iperf -c <iperf-server> -i 2 -t 33333
> 
> My results:
> ----------------------------
> - Guest to guest performance via local bridge: ~2.3 gbit/s, very high CPU 
> usage on vhost-$PID and kvm process on host
> - Physical server to guest on same subnet: ~940 mbit/s but with very high CPU 
> usage on vhost-$PID and kvm process on host
> - Physical server to guest via router: ~850 mbit/s with very high CPU usage 
> on vhost-$PID and kvm process on host (why is routed traffic slower than 
> switched on the guest??)
> - Physical server to kvm host via router (just to verify that the router is 
> not the issue): ~940 mbit/s with almost no CPU usage
> 
> Expected results after comparison with other KVM hosts everything else the 
> same:
> -------------------------
> - Guest to guest performance via local bridge: ~20 gbit/s, high CPU usage
> - Physical server to guest on same subnet: ~940 mbit/s with low CPU usage on 
> vhost-$PID and a bit higher on kvm process on host
> - Physical server to guest via router: ~940 mbit/s with low CPU usage on 
> vhost-$PID and a bit higher on kvm process on host 
> - Physical server to kvm host via router (just to verify that the router is 
> not the issue): ~940 mbit/s with almost no CPU usage (the same as my current 
> results)
> 
> Compare results with other OSes on same machine (guest to guest via bridge):
> Ubuntu Server 11.10 (virtualization host): ~19 gbit
> Proxmox VE 2.0: ~20 gbit/s
> Fedora 17 alpha: ~20 gbit/s

Which versions of qemu[-kvm] are used on these?
Did you try older versions of qemu-kvm from snapshot.debian.org?

It really looks like qemu-kvm-specific, not kernel-specific,
and qemu-kvm in Debian is very close to upstream 1.0.1 version,
so I'm not sure where to look at.

I've another bugreport at hand claiming that vhost-net, but
this time with macvtap not bridge, has a speed regression
between 1.0+dfsg-8 and 1.0+dfsg-9 (bridge mode unaffected).
1.0+dfsg-9 is the (debian) revision where I added a 1.0.1
diff (I didn't re-upload the new source).  Maybe this is
related somehow - please try -8 release too, for comparison.

Unfortunately I can't help here at all, as I don't reach any
speeds comparable with what you have, and mine don't change
much.  But I haven't tried installing any other OS (from a
list you mentioned too), -- I don't like to repartition my
only hdd to do so.


> I have tried:
> ----------------
> - Replacing Debian Wheezy with Debian Squeeze (stable, kernel 2.6.32-xx) - 
> even worse results

This kernel does not support vhost-net,
it was 2.6.35 or .38 addition.

> - Replacing kernel 3.2.0-2-amd64 with vanilla kernel 3.4-rc2 and config based 
> on Debians included config - no apparent change
> - Extracted the kernel-config file from Fedora 17 alphas kernel and used this 
> to compile a new kernel based on Debian Wheezys kernel source - slightly 
> worse 
> 
> results
> - Installing Proxmox VE 2.0 kernel in Debian. Results are the same
> - ...in addition to exchanging Debian with Ubuntu Server 11.10, Fedora 17 
> alpha, Proxmox 1.9 and 2.0 and ESXi 5 which all have expected network 
> performance using virtio.
> 
> 
> Please optimize KVM/vhost in Debian so it performs like the other 
> alternatives.

With pleasure, but I need some help ;)

Thank you!

/mjt