Re: [PATCH v5 0/6] Add vmnet.framework based network backend

[Jason Wang]

On Fri, Nov 19, 2021 at 1:42 AM Vladislav Yaroshchuk
<yaroshchuk2000@gmail.com> wrote:
>
>
>
> ср, 17 нояб. 2021 г. в 09:10, Jason Wang <jasowang@redhat.com>:
>>
>> On Tue, Nov 16, 2021 at 11:30 PM Vladislav Yaroshchuk
>> <yaroshchuk2000@gmail.com> wrote:
>> >
>> >
>> >
>> > вт, 16 нояб. 2021 г. в 10:23, Jason Wang <jasowang@redhat.com>:
>> >>
>> >> On Mon, Nov 15, 2021 at 6:45 PM Vladislav Yaroshchuk
>> >> <yaroshchuk2000@gmail.com> wrote:
>> >> >
>> >> >
>> >> >
>> >> > пн, 15 нояб. 2021 г. в 07:53, Jason Wang <jasowang@redhat.com>:
>> >> >>
>> >> >> On Fri, Nov 12, 2021 at 5:14 PM Vladislav Yaroshchuk
>> >> >> <yaroshchuk2000@gmail.com> wrote:
>> >> >> >
>> >> >> > macOS provides networking API for VMs called 'vmnet.framework':
>> >> >> > https://developer.apple.com/documentation/vmnet
>> >> >> >
>> >> >> > We can provide its support as the new QEMU network backends which
>> >> >> > represent three different vmnet.framework interface usage modes:
>> >> >> >
>> >> >> >   * `vmnet-shared`:
>> >> >> >     allows the guest to communicate with other guests in shared mode 
>> >> >> > and
>> >> >> >     also with external network (Internet) via NAT. Has 
>> >> >> > (macOS-provided)
>> >> >> >     DHCP server; subnet mask and IP range can be configured;
>> >> >> >
>> >> >> >   * `vmnet-host`:
>> >> >> >     allows the guest to communicate with other guests in host mode.
>> >> >> >     By default has enabled DHCP as `vmnet-shared`, but providing
>> >> >> >     network unique id (uuid) can make `vmnet-host` interfaces 
>> >> >> > isolated
>> >> >> >     from each other and also disables DHCP.
>> >> >> >
>> >> >> >   * `vmnet-bridged`:
>> >> >> >     bridges the guest with a physical network interface.
>> >> >> >
>> >> >> > This backends cannot work on macOS Catalina 10.15 cause we use
>> >> >> > vmnet.framework API provided only with macOS 11 and newer. Seems
>> >> >> > that it is not a problem, because QEMU guarantees to work on two most
>> >> >> > recent versions of macOS which now are Big Sur (11) and Monterey 
>> >> >> > (12).
>> >> >> >
>> >> >> > Also, we have one inconvenient restriction: vmnet.framework 
>> >> >> > interfaces
>> >> >> > can create only privileged user:
>> >> >> > `$ sudo qemu-system-x86_64 -nic vmnet-shared`
>> >> >> >
>> >> >> > Attempt of `vmnet-*` netdev creation being unprivileged user fails 
>> >> >> > with
>> >> >> > vmnet's 'general failure'.
>> >> >> >
>> >> >> > This happens because vmnet.framework requires 
>> >> >> > `com.apple.vm.networking`
>> >> >> > entitlement which is: "restricted to developers of virtualization 
>> >> >> > software.
>> >> >> > To request this entitlement, contact your Apple representative." as 
>> >> >> > Apple
>> >> >> > documentation says:
>> >> >> > https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_vm_networking
>> >> >>
>> >> >> Do you know how multipass work? Looks like it uses vmnet without 
>> >> >> privileges.
>> >> >>
>> >> >
>> >> > I've just checked this, and they still need root privileges. They have a
>> >> > `multipassd` daemon which is launched as root by launchd by default.
>> >> >
>> >> > ```
>> >> > bash-5.1$ ps axo ruser,ruid,comm | grep multipass
>> >> > root                 0 /Library/Application 
>> >> > Support/com.canonical.multipass/bin/multipassd
>> >> > root                 0 /Library/Application 
>> >> > Support/com.canonical.multipass/bin/hyperkit
>> >> > ```
>> >> >
>> >> > That's the reason why it's required to 'enter a password' while 
>> >> > multipass installation:
>> >> > it creates launchd plist (kinda launch rule) and places it to 
>> >> > /Library/LaunchDaemons/,
>> >> > which can be done only with root privileges.
>> >> >
>> >> > ```
>> >> > bash-5.1$ ll /Library/LaunchDaemons | grep multipass
>> >> > -rw-r--r--   1 root  wheel   1.1K 15 Nov 12:47 
>> >> > com.canonical.multipassd.plist
>> >> > ```
>> >> >
>> >> > And after this launchd launches this multipassd daemon as root every 
>> >> > boot.
>> >> >
>> >> > So an unprivileged user can launch a multipass VM instance, but 
>> >> > actually the
>> >> > `hyperkit` process which interacts with vmnet is gonna be launched by 
>> >> > multipassd
>> >> > running as root.
>> >>
>> >> I wonder how it passes the vmnet object to qemu? Nothing obvious from
>> >> the qemu command line that multipass launched:
>> >>
>> >> -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:52:e8:e4
>> >>
>> >> (But I haven't had time to check their qemu codes).
>> >>
>> >
>> > It seems they just use QEMU with patch by Phillip Tennen:
>> > https://patchew.org/QEMU/20210218134947.1860-1-phillip.ennen@gmail.com/
>> >
>> > In that patch he does quite the same as we in this series, the
>> > difference remains in foreground: he creates one new 'vmnet-macos'
>> > netdev, and uses 'mode=shared' property to choose vmnet
>> > operating mode. I decided to create three different netdevs instead
>> > (vmnet-shared, vmnet-host, vmnet-bridged). Also I've added some
>> > features related to isolation and ipv6.
>>
>> Ok.
>>
>> >
>> > > I wonder how it passes the vmnet object to qemu
>> > I hope I clearly described this.
>>
>> A silly question, how did the 'hyperkit' process pass the vmnet object to 
>> qemu?
>>
> I think I didn't describe it very well in the previous mail.
> The 'hyperkit' and QEMU are two multipass 'drivers'.
>
> hyperkit is an independent hypervisor based on xhyve (bhyve):
> https://github.com/moby/hyperkit
>
> So there are many ways to launch multipass VMs:
> * VM <-> hyperkit <-> host
> * VM <-> QEMU <-> host
> * VM <-> virtualbox <-> host
> * etc.
>
> https://discourse.ubuntu.com/t/announcing-the-first-release-candidate-for-apple-m1-support/24445
>
> So hyperkit doesn't pass anything to QEMU, it's a separate
> hypervisor and multipass driver. But it uses vmnet backend
> too, the same way as we do:
> https://github.com/moby/hyperkit/blob/2f061e447e1435cdf1b9eda364cea6414f2c606b/src/lib/pci_virtio_net_vmnet.c#L250

Ok, I think I get it.

>
>> >
>> >> >
>> >> > tl;dr sadly, we can't interact with vmnet.framework without having our 
>> >> > binary correctly
>> >> > signed and being an unprivileged user. Root privileges or special 
>> >> > signature with
>> >> > entitlement is required.
>> >>
>> >> This is something similar to what happens in other OS. E.g for the tap
>> >> backend, it can't be created without privileges. So qemu allows to:
>> >>
>> >> 1) the TAP to be created by privileged program like libvirt, and its
>> >> fd could be passed to qemu via SCM_RIGHTS
>> >> 2) run a set-uid helper to create and config TAP
>> >>
>> >> This is something we need to consider now or in the future probably.
>> >>
>> >
>> > Seems we can do nothing with this if we have qemu-bundled &
>> > direct vmnet.framework interaction, it always requires privileges
>> > or entitlement.
>> > The workaround can be moving vmnet-related things to
>> > another helper binary running with privileges, and usage of
>> > this helper somewhere between qemu and vmnet.
>>
>> Yes, that's my point.
>>
>> >
>> > I think for now it's applicable to leave it as is, having qemu
>> > that requires privileges for vmnet.framework usage.
>>
>> Right, but we need to consider it for the future.
>>
>> Btw, if you wish, you can list yourself as the maintainer for this backend.
>>
>
> Ok, thank you! Let me do this in the next patch version, after
> we finish our discussion and I fix all the issues you pointed
> in your review.

Sure.

Thanks

>
>>
>> Thanks
>>
>> >
>> >>
>> >> Thanks
>> >>
>> >> >
>> >> >
>> >> >> Thanks
>> >> >>
>> >> >
>> >> > Thank you for your review, I will check it this week and reply as soon 
>> >> > as possible.
>> >> >
>> >> >>
>> >> >> >
>> >> >> > One more note: we still have quite useful but not supported
>> >> >> > 'vmnet.framework' features as creating port forwarding rules, IPv6
>> >> >> > NAT prefix specifying and so on.
>> >> >> >
>> >> >> > Nevertheless, new backends work fine and tested within 
>> >> >> > `qemu-system-x86-64`
>> >> >> > on macOS Bir Sur 11.5.2 host with such nic models:
>> >> >> >   * e1000-82545em
>> >> >> >   * virtio-net-pci
>> >> >> >   * vmxnet3
>> >> >> >
>> >> >> > The guests were:
>> >> >> >   * macOS 10.15.7
>> >> >> >   * Ubuntu Bionic (server cloudimg)
>> >> >> >
>> >> >> >
>> >> >> > This series partially reuses patches by Phillip Tennen:
>> >> >> > https://patchew.org/QEMU/20210218134947.1860-1-phillip.ennen@gmail.com/
>> >> >> > So I included his signed-off line into one of the commit messages and
>> >> >> > also here.
>> >> >> >
>> >> >> > v1 -> v2:
>> >> >> >  Since v1 minor typos were fixed, patches rebased onto latest master,
>> >> >> >  redundant changes removed (small commits squashed)
>> >> >> >
>> >> >> > v2 -> v3:
>> >> >> >  - QAPI style fixes
>> >> >> >  - Typos fixes in comments
>> >> >> >  - `#include`'s updated to be in sync with recent master
>> >> >> > v3 -> v4:
>> >> >> >  - Support vmnet interfaces isolation feature
>> >> >> >  - Support vmnet-host network uuid setting feature
>> >> >> >  - Refactored sources a bit
>> >> >> > v4 -> v5:
>> >> >> >  - Missed 6.2 boat, now 7.0 candidate
>> >> >> >  - Fix qapi netdev descriptions and styles
>> >> >> >    (@subnetmask -> @subnet-mask)
>> >> >> >  - Support vmnet-shared IPv6 prefix setting feature
>> >> >> >
>> >> >> > Signed-off-by: Phillip Tennen <phillip@axleos.com>
>> >> >> > Signed-off-by: Vladislav Yaroshchuk <yaroshchuk2000@gmail.com>
>> >> >> >
>> >> >> > Vladislav Yaroshchuk (6):
>> >> >> >   net/vmnet: add vmnet dependency and customizable option
>> >> >> >   net/vmnet: add vmnet backends to qapi/net
>> >> >> >   net/vmnet: implement shared mode (vmnet-shared)
>> >> >> >   net/vmnet: implement host mode (vmnet-host)
>> >> >> >   net/vmnet: implement bridged mode (vmnet-bridged)
>> >> >> >   net/vmnet: update qemu-options.hx
>> >> >> >
>> >> >> >  meson.build                   |   4 +
>> >> >> >  meson_options.txt             |   2 +
>> >> >> >  net/clients.h                 |  11 ++
>> >> >> >  net/meson.build               |   7 +
>> >> >> >  net/net.c                     |  10 ++
>> >> >> >  net/vmnet-bridged.m           | 111 ++++++++++++
>> >> >> >  net/vmnet-common.m            | 325 
>> >> >> > ++++++++++++++++++++++++++++++++++
>> >> >> >  net/vmnet-host.c              | 111 ++++++++++++
>> >> >> >  net/vmnet-shared.c            |  92 ++++++++++
>> >> >> >  net/vmnet_int.h               |  48 +++++
>> >> >> >  qapi/net.json                 | 127 ++++++++++++-
>> >> >> >  qemu-options.hx               |  25 +++
>> >> >> >  scripts/meson-buildoptions.sh |   3 +
>> >> >> >  13 files changed, 874 insertions(+), 2 deletions(-)
>> >> >> >  create mode 100644 net/vmnet-bridged.m
>> >> >> >  create mode 100644 net/vmnet-common.m
>> >> >> >  create mode 100644 net/vmnet-host.c
>> >> >> >  create mode 100644 net/vmnet-shared.c
>> >> >> >  create mode 100644 net/vmnet_int.h
>> >> >> >
>> >> >> > --
>> >> >> > 2.23.0
>> >> >> >
>> >> >>
>> >> >
>> >> >
>> >> > --
>> >> > Best Regards,
>> >> >
>> >> > Vladislav Yaroshchuk
>> >>
>> >
>> >
>> > --
>> > Best Regards,
>> >
>> > Vladislav Yaroshchuk
>>