[Qemu-discuss] Question regarding QEMU Networking with MACVLAN/MACVTAP and File Descriptor (fd=) parameter

[Zir Blazer]

At the moment, I'm a Xen user, but I'm planning to migrate to standalone QEMU so I can use its latest features without having to wait for them to be enabled Xen-side. To ease the transition, I'm testing absolutely everything I can inside a VM enviroment (Only exception would be PCI/VGA Passthrough, since I would need to test that in bare metal) to not have surprises down the road. I'm doing the testing in a Nested VM (Xen Host, Nested KVM Host, VM).

Previously, I successfully managed to set up a small Network with a Bridge that had attached the NIC and a TAP Device using Arch Linux netctl Network Manager. It covered the basics: Both the Nested KVM Host and the VM had Internet connectivity, and received an IP from the DHCP Server in the Router, so they could be directly seen from other machines in the network. Then suddently, I hear about that thing called MACVLAN and how it had better in performance, bla bla bla. Since I'm going to start mostly from scratch, I wanted to use the latest and greatest (Including trying systemd-networkd instead of netctl), so why not?

...Then, I spend the last three days googling around trying to understand a few concepts of how to use MACVTAP with QEMU. However, nearly all the info I found is around 4-5 years old, and while most of it seems to still be valid, there are some things that I still didn't managed to get a explanation of. This is even worse since I barely have any type of Networking knowledge, nearly everything I know about the matter is what I learned recently while toying with this, and is related to virtualization enviroments.

My current setup involves the following:

1- An ADSL Modem+Router. Should also be the DHCP Server. Dynamically assigns IPs, since I never really touched its config (Like, for reserving IP for a specific MAC Address)

2- A computer with two physical NICs. Only one NIC is in use, wired to the ADSL Router

3- A Xen Host running Xen 4.5 with Arch Linux as Dom0 using Kernel 4.0, and three VMs (One main everyday VM, another for some work which I want isolated, and the Nested KVM Host). I'm intending to replace this host with QEMU-KVM-VFIO

It has a Bridge called xenbr0 made with netctl, following Arch Linux Wiki Xen installation instructions. Xen automatically creates a vif1.0 and vif1.0-emu (Or another number instead of 1) virtual network devices for each VM launched, and destroys them when you close it

4- A Nested KVM Host running Arch Linux using Kernel 4.4 and QEMU 2.5 (I'm not using libvirt, just standalone QEMU). Since I'm using Nested Virtualization, this host can create KVM capable VMs

5- A test VM spawned by the Nested KVM Host. I just use it to boot ArchISO (Arch Linux LiveCD) and check if connectivity is working

I think I managed to get MACVLAN and MACVTAP working inside the Nested KVM Host, but I prefer if someone else can confirm that I didn't missed anything, and explain some things about the QEMU parameters regarding File Descriptors that I don't understand. I suppose is better if I write all the reelevant systemd-networkd config files first:

/etc/systemd/network/macvlan0.netdev

[NetDev]

Name=macvlan0

Kind=macvlan

MACAddress=52:54:ff:00:00:00

[MACVLAN]

Mode=bridge

/etc/systemd/network/macvtap0.netdev

[NetDev]

Name=macvtap0

Kind=macvtap

MACAddress=52:54:1f:00:00:00

[MACVTAP]

Mode=bridge

/etc/systemd/network/eth0.network

[Match]

name=eth0

[Network]

MACVLAN=macvlan0

MACVTAP=macvtap0

/etc/systemd/network/macvlan0.network

[Match]

name=macvlan0

[Network]

DHCP=yes

This gives me connectivity in the Nested KVM Host, and I can ping google and other machines in the Network (Actually, I don't need macvtap0.netdev and the MACVTAP reference in eth0.network for this part to work).

ip addr output on the Nested KVM Host looks like this:

1: lo / MAC: 00:00:00:00:00:00 / IP: 127.0.0.1

2: eth0 / MAC: 00:16:3e:ff:ff:ff / IP: - (This MAC Address comes from the Xen config file)

3: address@hidden / MAC: 52:54:1f:00:00:00 / IP: 192.168.1.x

4: address@hidden / MAC: 52:54:ff:00:00:00 / IP: 192.168.1.x

Up to this, everything seems ok. Checking my Router config, I can see that it sees as DHCP Clients the macvtap0 and macvlan0 MAC Address and assigns an IP Address to each. Both are named xentest (The hostname of the Nested KVM Host).

Now comes the QEMU part, which is the reason why I'm mailing this to qemu-users...

As far that I managed to google, QEMU does NOT support MACVTAP interfaces natively, at least that was the case 5 years ago, but couldn't find info stating if it currently does. Instead, to use MACVTAP you have to trick QEMU when you launch it via a script-like annex when using a -netdev tap interface. The syntax I'm using is similar to the one I got from here:

https://lists.nongnu.org/archive/html/qemu-devel/2011-09/msg03778.html

-netdev tap,id=tap,vhost=on,fd=3 3<>/dev/tap$(< /sys/class/net/macvtap0/ifindex)

-device virtio-net-pci,netdev=tap,mac=$(< /sys/class/net/macvtap0/address)

With this, I have connectivity in the VM. I can also ping from it to the Nested KVM Host macvlan0 and macvtap0 IPs and from the Nested KVM Host to the guest NIC IP, which I suppose is thanks to the MACVLAN in Bridge Mode, since according to all info I read, the host is isolated in pure MACVTAP setups and adding MACVLAN to the host is supposed to workaround that.

Now, its about decoding what the above two lines does...

The second line is easy for me to understand. It is a standard virtio-net-pci Device which reads the MAC Address from macvtap0 (Since < /sys/class/net/macvtap0/address on a terminal returns 52:54:1f:00:00:00) and uses it as its own. It seems that the guest NIC REQUIRES to have the same MAC Address than the macvtap0 Device, since I tried manually setting mac= to something else and I had no connectivity in the VM, but it worked with either mac=$(< /sys/class/net/macvtap0/address) or a straight input of mac=52:54:1f:00:00:00.

The curious thing here, is that the Router can see both macvtap0 and also the guest VM NIC. They both share the same MAC Address, but it assigns a different IP to each. The hostname of both is also replace by archiso, the default ArchISO LiveCD hostname (Previously it was only one, and called xentest). I don't know if two devices with the same MAC Address and different IP Address can cause conflict between them, or if this behavior with MACVTAP is intended.

Understanding the first line is much worse, because I really get very few things out of it. < /sys/class/net/macvtap0/ifindex returns the value 3, which is also the number at the left of the Interface when you use ip addr. As far that I got info, this value is the File Descriptor, and <>/dev/tap$ is intended to open it in read/write mode, the $ being effectively the returned 3. This is supposed to be the workaround to use MACVTAP in QEMU. What I don't get is the fd=3 3 part. I managed to change it to other numbers (fd=5 5, fd=7 7, fd=22 22), and the VM still has network connectivity. I found no hard info about what value it should have, or if it can be a totally arbitrary number. The same applies to vhostfd= (which I'm not using), which I saw in some examples where it has a different number that fd (Like, fd=21, vhostfd=22), but no one tells you where those values originates from, since they don't seem to be the File Descriptor I got earlier.

info network in QEMU Monitor returns the following:

virtio-net-pci.0: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:1f:00:00:00

 \ tap: index=0,type=tap,fd=3

fd can be whatever, if I launch QEMU with fd=22 22 then it shows fd=22. The only out of ordinary behavior is with fd=0 0, the guest VM works, but using info network hangs the QEMU Monitor (-monitor stdio)

This is the part that after tons of googling, I failed to find a response for. So, what fd= should be and why?

Should I don't get a reply, as least someone will have clean basic templates to try out MACVTAP in standalone QEMU with systemd-networkd.

As a sidenote, the vhost_net Kernel Module by default is not loaded until the first time I launch the VM with vhost=on, and seems that it is automatically loaded thanks to udev. Based on using lsmod | grep vhost, the "Used by" number for vhost_net is 0 when the VM is not open or isn't using vhost=on, and 1 when I open the VM with vhost=on. It seems to be irrelevant if I'm using as guest NIC the virtio-net-pci or another one like the e1000, the vhost_net module seems to get in use in either case. All the mentions I read about vhost_net mentions that it only works specifically with virtio-net-pci NIC, not as general purpose. Did it gained such capability, or it does nothing if not paired with a virtio-net-pci even when lsmod says that it is in use?