[Qemu-devel] Macvtap bug: contractor wanted

[Richard Davies]

Hi. We run a cloud compute provider using qemu-kvm and macvtap and are keen
to find a paid contractor to fix a bug with unusably slow inbound networking
over macvtap.

We originally reported the bug in this thread (report copied below):

  http://marc.info/?t=134511098600002

We have also reproduced using only a Fedora 17 Live CD:

  https://bugzilla.redhat.com/show_bug.cgi?id=855640

This bug is a serious problem for us, since we have built a new version of our
product which suffers from it and did not realise in testing, only once we had
live production installs.

Many thanks to Michael Tsirkin for his initial help. However, we appreciate
that his time is limited and divided among many projects. Given the commercial
time pressure on us to fix this bug, we are keen to hire a contractor to start
work immediately.

If anyone knowledgeable in the area would be interested in being paid to work
on this, or if you know someone who might be, we would be delighted to hear
from you.

Cheers,

Chris and Richard.

P.S. The original report read as follows:

  I'm experiencing a problem with qemu + macvtap which I can reproduce on a
  variety of hardware, with kernels varying from 3.0.4 (the oldest I tried) to
  3.5.1 and with qemu[-kvm] versions 0.14.1, 1.0, and 1.1.

  Large data transfers over TCP into a guest from another machine on the
  network are very slow (often less than 100kB/s) whereas transfers outbound
  from the guest, between two guests on the same host, or between the guest
  and its host run at normal speeds (>= 50MB/s).

  The slow inbound data transfer speeds up substantially when a ping flood is
  aimed either at the host or the guest, or when the qemu process is straced.
  Presumably both of these are ways to wake up something that is otherwise
  sleeping too long?

  For example, I can run

    ip addr add 192.168.1.2/24 dev eth0
    ip link set eth0 up
    ip link add link eth0 name tap0 address 02:02:02:02:02:02 type macvtap mode 
bridge
    ip link set tap0 up
    qemu-kvm -hda debian.img -cpu host -m 512 -vnc :0 \
      -net nic,model=virtio,macaddr=02:02:02:02:02:02 \
      -net tap,fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)

  on one physical host which is otherwise completely idle. From a second
  physical host on the same network, I then scp a large (say 50MB) file onto
  the new guest. On a gigabit LAN, speeds consistently drop to less than
  100kB/s as the transfer progresses, within a second of starting.

  The choice of virtio virtual nic in the above isn't significant: the same 
thing
  happens with e1000 or rtl8139. You can also replace the scp with a straight
  netcat and see the same effect.

  Doing the transfer in the other direction (i.e. copying a large file from the
  guest to an external host) achieves 50MB/s or faster as expected. Copying
  between two guests on the same host (i.e. taking advantage of the 'mode
  bridge') is also fast.

  If I create a macvlan device attached to eth0 and move the host IP address to
  that, I can communicate between the host itself and the guest because of the
  'mode bridge'. Again, this case is fast in both directions.

  Using a bridge and a standard tap interface, transfers in and out are fast
  too:

    ip tuntap add tap0 mode tap
    brctl addbr br0
    brctl addif br0 eth0
    brctl addif br0 tap1
    ip link set eth0 up
    ip link set tap0 up
    ip link set br0 up
    ip addr add 192.168.1.2/24 dev br0
    qemu-kvm -hda debian.img -cpu host -m 512 -vnc :0 \
      -net nic,model=virtio,macaddr=02:02:02:02:02:02 \
      -net tap,script=no,downscript=no,ifname=tap0

  As mentioned in the summary at the beginning of this report, when I strace a
  guest in the original configuration which is receiving data slowly, the data
  rate improves from less than 100kB/s to around 3.1MB/s. Similarly, if I ping
  flood either the guest or the host it is running on from another machine on
  the network, the transfer rate improves to around 1.1MB/s. This seems quite
  suggestive of a problem with delayed wake-up of the guest.

  Two reasonably up-to-date examples of machines I've reproduced this on are
  my laptop with an r8169 gigabit ethernet card, Debian qemu-kvm 1.0 and
  upstream 3.4.8 kernel whose .config and boot dmesg are at

    http://cdw.me.uk/tmp/laptop-config.txt
    http://cdw.me.uk/tmp/laptop-dmesg.txt

  and one of our large servers with an igb gigabit ethernet card, upstream
  qemu-kvm 1.1.1 and upstream 3.5.1 linux:

    http://cdw.me.uk/tmp/server-config.txt
    http://cdw.me.uk/tmp/server-dmesg.txt

  For completeness, I've put the Debian 6 test image I've been using for
  testing at

    http://cdw.me.uk/tmp/test-debian.img.xz

  though I've see the same problem from a variety of guest operating systems.
  (In fact, I've not yet found any combination of host kernel, guest OS and
  hardware which doesn't show these symptoms, so it seems to be very easy to
  reproduce.)

We later found that

  -CONFIG_INTEL_IDLE=y
  +# CONFIG_INTEL_IDLE is not set

helped the problem on my laptop, but none of the obvious similar things made
any difference on AMD hardware.

The bug appears whether or not vhost-net is used, and irrespective of emulated
NIC in qemu, so is very likely to be a kernel issue rather than a qemu issue.