[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-devel] Mysterious RST connection with virtio-net NATting VM
From: |
Neil Skrypuch |
Subject: |
[Qemu-devel] Mysterious RST connection with virtio-net NATting VM |
Date: |
Fri, 21 Oct 2016 16:34:38 -0400 |
User-agent: |
KMail/4.14.10 (Linux/4.1.15-gentoo-r1; KDE/4.14.16; x86_64; ; ) |
I have a NATting VM (let's call this vm/nat) sitting in front of another VM
(let's call this one vm/wget), with vm/wget residing in a private virtual
network, with all network connectivity for vm/wget going through vm/nat.
Additionally, I have a web server running on a physical machine from which
vm/wget is trying to fetch a file. Graphically, it looks something like this:
web server <-> vm/nat <-> vm/wget
Now, when vm/wget tries to wget a file from the web server, it frequently gets
a connection reset by peer network error, wget will retry the connection and
resume the download, after which the cycle will repeat many times before the
file eventually fully downloads. The problem is not limited to wget though,
both scp and curl experience the same connection reset by peer issue quite
consistently.
After some collaborative debugging on IRC, tcpdump revealed that an RST is
being generated which is breaking the network connection. Now, from the web
server's point of view, the RST is coming from vm/wget and from vm/wget's
point of view, the RST is coming from the web server. Further tcpdumping from
both NICs on vm/nat reveals that vm/nat itself is generating the RSTs,
although it is not clear why. Furthermore, the TCP sequence numbers on the
RSTs are totally out of line with the ACKs in the stream, for example:
13:01:19.819882 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack
1914061, win 65535, length 0
13:01:19.819886 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack
1914061, win 65535, length 0
13:01:19.819889 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack
1918441, win 65535, length 0
13:01:19.820127 IP 192.168.123.198.80 > 192.168.100.160.37675: Flags [R], seq
1390832295, win 0, length 0
We ran out of ideas on how to further debug the RST issue, but ideas are
welcome.
One other thing worth noting is that vm/nat is configured with a pair of
virtio-net NICs. If I change the NICs to e1000 models, the RST issue vanishes.
More specifically, only the NIC facing the web server on vm/nat needs to be
changed to e1000 for the issue to go away, the other can remain virtio-net.
Also, running a wget directly on vm/nat works in all cases. I tried a number
of other things to try and narrow down the issue, none of which helped,
including:
- vhost=on
- qemu versions 2.7.0, 2.3.0 and 2.0.0
- disabling kvm accel (ie, using TCG)
The full command line for vm/nat is this:
/usr/bin/qemu-system-x86_64 -machine accel=kvm -vga cirrus -m 512 -smp
4,cores=4,sockets=1,threads=1 -drive
file=/var/lib/kvm/lb.polldev.com.img,format=raw,if=virtio -device virtio-net-
pci,mac=52:54:00:42:45:15,netdev=net1 -netdev type=tap,script=/etc/qemu-ifup-
br0,downscript=no,id=net1 -device virtio-net-
pci,mac=52:54:00:42:45:16,netdev=net2 -netdev type=tap,script=/etc/qemu-ifup-
br2,downscript=no,id=net2 -curses -monitor
unix:/var/lib/kvm/monitor/lb,server,nowait -name
"lb.polldev.com",process=lb.polldev.com
I was able to reproduce this with different web server machines, as well as
different vm/wget machines, as well as when moving vm/nat to a different host
(albeit for the other host, the RST issue presents itself significantly less
frequently).
Since the problem goes away with e1000, my gut feeling is that virtio-net is
somehow slightly mangling the packets which causes iptables to eventually RST
the connection, although it's probably more subtle than that, since I'm sure
I'm not the first person to try NAT in a virtio-net VM.
Any thoughts?
- Neil
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Qemu-devel] Mysterious RST connection with virtio-net NATting VM,
Neil Skrypuch <=