[Qemu-devel] Mysterious RST connection with virtio-net NATting VM

[Neil Skrypuch]

I have a NATting VM (let's call this vm/nat) sitting in front of another VM 
(let's call this one vm/wget), with vm/wget residing in a private virtual 
network, with all network connectivity for vm/wget going through vm/nat. 
Additionally, I have a web server running on a physical machine from which 
vm/wget is trying to fetch a file. Graphically, it looks something like this:

web server <-> vm/nat <-> vm/wget

Now, when vm/wget tries to wget a file from the web server, it frequently gets 
a connection reset by peer network error, wget will retry the connection and 
resume the download, after which the cycle will repeat many times before the 
file eventually fully downloads. The problem is not limited to wget though, 
both scp and curl experience the same connection reset by peer issue quite 
consistently.

After some collaborative debugging on IRC, tcpdump revealed that an RST is 
being generated which is breaking the network connection. Now, from the web 
server's point of view, the RST is coming from vm/wget and from vm/wget's 
point of view, the RST is coming from the web server. Further tcpdumping from 
both NICs on vm/nat reveals that vm/nat itself is generating the RSTs, 
although it is not clear why. Furthermore, the TCP sequence numbers on the 
RSTs are totally out of line with the ACKs in the stream, for example:

13:01:19.819882 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 
1914061, win 65535, length 0
13:01:19.819886 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 
1914061, win 65535, length 0
13:01:19.819889 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 
1918441, win 65535, length 0
13:01:19.820127 IP 192.168.123.198.80 > 192.168.100.160.37675: Flags [R], seq 
1390832295, win 0, length 0

We ran out of ideas on how to further debug the RST issue, but ideas are 
welcome.

One other thing worth noting is that vm/nat is configured with a pair of 
virtio-net NICs. If I change the NICs to e1000 models, the RST issue vanishes. 
More specifically, only the NIC facing the web server on vm/nat needs to be 
changed to e1000 for the issue to go away, the other can remain virtio-net. 
Also, running a wget directly on vm/nat works in all cases. I tried a number 
of other things to try and narrow down the issue, none of which helped, 
including:

- vhost=on
- qemu versions 2.7.0, 2.3.0 and 2.0.0
- disabling kvm accel (ie, using TCG)

The full command line for vm/nat is this:

/usr/bin/qemu-system-x86_64 -machine accel=kvm -vga cirrus -m 512 -smp 
4,cores=4,sockets=1,threads=1 -drive 
file=/var/lib/kvm/lb.polldev.com.img,format=raw,if=virtio -device virtio-net-
pci,mac=52:54:00:42:45:15,netdev=net1 -netdev type=tap,script=/etc/qemu-ifup-
br0,downscript=no,id=net1 -device virtio-net-
pci,mac=52:54:00:42:45:16,netdev=net2 -netdev type=tap,script=/etc/qemu-ifup-
br2,downscript=no,id=net2 -curses -monitor 
unix:/var/lib/kvm/monitor/lb,server,nowait -name 
"lb.polldev.com",process=lb.polldev.com

I was able to reproduce this with different web server machines, as well as 
different vm/wget machines, as well as when moving vm/nat to a different host 
(albeit for the other host, the RST issue presents itself significantly less 
frequently).

Since the problem goes away with e1000, my gut feeling is that virtio-net is 
somehow slightly mangling the packets which causes iptables to eventually RST 
the connection, although it's probably more subtle than that, since I'm sure 
I'm not the first person to try NAT in a virtio-net VM.

Any thoughts?

- Neil