Networking on Mac host

[Phil Endecott]

Dear Experts,

I have set up qemu on an ARM Mac, running a Debian guest. I've done 
this in order to use the Mac as part of a distcc "compiler farm" - 
which is very worthwhile, as it is much faster than any of my other 
ARM systems. It is functional, but I have some issues with networking.

I'm using the Homebrew package of qemu. I believe that this is just 
your qemu-6.1.0 with v12 of Alexander Graf's hvf patches applied [1].
See below [2] for my qemu invocation. The guest is a fairly minimal 
install of Debian 11. The host is running MacOS 11.4. I'm using user 
networking.

The first issue is that DNS doesn't work. It appears that the qemu 
DHCP server reports the 10.0.2.3 DNS server address to the guest, but 
the guest gets no reply when it sends requests to that address.

I've worked around this by overriding the DHCP-supplied DNS server 
address with the address of my local network's existing DNS server, 
and this now works as expected.

Is there any known issue with the qemu built-in DNS server on 
MacOS? What could I be doing wrong to make this happen? How can I 
debug it?

The second issue is that I get some connection failures when 
distcc tries to connect over ssh. On the other machine I run something 
similar to this:

$ DISTCC_HOSTS="@gabon/10 localhost/1" CC="distcc gcc" make -B -j11

@gabon/10 means connect to gabon over ssh and run up to 10 jobs there. 
gabon is the name of the Linux qemu guest. I have an ssh config file that 
tells to it connect to the Mac instead and use port 2222, which qemu 
forwards to port 22 on the guest.

The failures look like this:

Connection reset by 192.168.1.53 port 2222
distcc[21710] (dcc_writex) ERROR: failed to write: Broken pipe
ssh: connect to host finland port 2222: Connection reset by peer
kex_exchange_identification: read: Connection reset by peer

distcc retries and/or compiles locally and actually it's easy to 
overlook the errors. I think it may occur mainly when the build 
starts, perhaps because distcc opens 10 ssh connections almost 
simultaneously? So is there something in the qemu user-mode 
networking implementation that could cause connections to fail in 
that case? Or maybe something in MacOS that doesn't like lots of 
connections starting simultaneously?

(It's quite possible that the cause is something in the guest. 
I've increased sshd_config's MaxStartup parameter, without any 
effect.)

Third issue: I have tried to make bridged networking work, and failed. 
I've seen reports that this doesn't work in recent versions of MacOS 
- is this true? I've seen a patch that adds "vmnet" support to qemu 
- is this what I need? What benefits would I get from bridged / vmnet 
networking compared to user mode? I think that my current distcc speed 
is limited either by the build host or by networking, so if vmnet 
would improve this I'd like to try it.

One final question: I've seen suggestions that the speed of qemu 
guests on MacOS may depend on whether MacOS thinks the process is 
interactive and needs the fast CPU cores, or not. Does anyone know 
if there is any truth in that?

Thanks in advance for any advice.


Regards, Phil.


[1] https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/qemu.rb

[2] My complete qemu invocation:

qemu-system-aarch64 \
  -machine virt,accel=hvf,highmem=off \
  -cpu host \
  -smp 7 \
  -m 8G \
  -device qemu-xhci,id=usb-bus \
  -device usb-tablet,bus=usb-bus.0 \
  -device usb-mouse,bus=usb-bus.0 \
  -device usb-kbd,bus=usb-bus.0 \
  -device virtio-gpu-pci \
  -display none \
  -netdev 
user,id=net0,hostname=gabon,domainname=chezphil.org,dnssearch=chezphil.org,hostfwd=tcp::2222-:22
 \
  -device virtio-net-pci,netdev=net0 \
  -drive 
format=raw,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd,if=pflash,readonly=on
 \
  -drive format=raw,file=$D/ovmf_vars.fd,if=pflash \
  -device nvme,drive=root,serial=drive0,bootindex=0 \
  -drive if=none,media=disk,id=root,format=qcow2,file=$D/rootimg.qcow2 \
  -daemonize