Re: How to make a VM guest look like just another metal machine on my LAN?

[Alex Bennée]

Alok Prasad <prasadalok77@gmail.com> writes:

> Just a simple Bridge configuration is too daunting to configure with QMEU.
> serious some helper script /config is need to make this simpler for user of
> qemu to use this.

Generally we advise using a higher layer VM manager based on libvirt to
do these sorts of things. They make the process of configuring networks
a lot easier than doing it all by hand.

>
> On Sun, Feb 21, 2021 at 2:16 PM Steve Litt <slitt@troubleshooters.com>
> wrote:
>
>> Hi all,
>>
>> My LAN at home is on 192.168.0.0/24, connected to the Internet via a
>> cablemodem/firewall/router/gateway at 192.168.0.1. My Daily Driver
>> Desktop (DDD), which will after this be referred to as the "host" or
>> "metal host" is at 192.168.0.2. I have a printer with an http interface
>> at 192.168.0.13. Throughout this post I'm careful to discriminate
>> between the metal host and the VM guest, which is created on the metal
>> host, for all config options.
>>
>> What I'm trying to accomplish is to launch a VM guest (Devuan) on my
>> metal host (Void Linux), such that the VM guest performs as if it were
>> just another physical computer on my LAN.
>>
>> I've been reading and experimenting for four days and still don't have
>> what I need. Here are some of the documents I've used trying to get
>> this done:
>>
>> https://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
>>
>>
>> https://ahelpme.com/linux/howto-do-qemu-full-virtualization-with-bridged-networking/
>>
>> http://www.mpaoli.net/~root/bin/TEMPLATE
>>
>>
>> https://www.debian.org/doc/manuals/debian-handbook/sect.virtualization.en.html#sect.lxc.network
>>
>> I'm trying to do it purely with ip commands, although I could use brctl
>> if necessary. I'm staying away from virt-manager and aqemu because they
>> don't work on my Void Linux metal host, and would just add even more
>> variables and ambiguity.
>>
>> Speaking of ambiguity, every document I've read (and I've read dozens)
>> has the following ambiguities:
>>
>> 1) When discussing a setting, they don't indicate whether that setting
>>    should be on the metal host or the VM guest. Perhaps to a person who
>>    thoroughly understands virtual machines, such a distinction would be
>>    obvious via context, but it's not obvious to me.
>>
>> 2) When specifying an "id=whatever", they don't indicate how the id
>>    would be used, or what other references to that id need to be made.
>>
>> 3) They include no realistic steps for troubleshooting a "warning:
>>    netdev mybridge0 has no peer" type warning, nor even explain what it
>>    means more than a few guesses and "have you tried...".
>>
>> Based on the previously listed links, I deduce that the TAP is created
>> by the guest VM, in such a way that it attaches to the bridge created on
>> the metal host, and therefore I have no need to create a TAP on the
>> metal host.
>>
>> Here's my progress so far, based on the links listed above and my
>> other readings and experimentation:
>>
>> ***
>>
>> I build the bridge purely with ip commands. Also, I don't mess
>> with the firewall (which perhaps has been my problem all along). I'll
>> investigate the firewall aspect tomorrow.
>>
>> Below are some scripts and stuff I'm using. The following is
>> upnet.sh, which I use to set up networking on the metal host, which
>> happens to run Void Linux, which has no /etc/network/interfaces:
>>
>> =========================================
>> #!/bin/sh
>>
>> use_bridge=1
>> use_tap=0
>>
>> dev="enp40s0"
>> ipaddr_major="192.168.0.2"
>> ipaddr_minor="192.168.0.102"
>> gateway="192.168.0.1"
>>
>> error_tap_without_bridge(){
>>    echo -n "ERROR: Can\'t set TAP without "
>>    echo -n "BRIDGE! "
>>    echo Aborting...
>>    exit 1
>> }
>>
>>
>> enable_ip_forwarding(){
>>    echo 1 > /proc/sys/net/ipv4/ip_forward
>> }
>>
>> unset_everything(){
>>    dev=$1
>>    ip_maj=$2
>>    ip_min=$3
>>    gateway=$4
>>    echo "Unsetting everything for $dev, $ip_maj and $ip_min"
>>    ip link set dev tap0 down
>>    brctl delif br0 tap0
>>    ip link del tap0
>>    ip link set dev br0 down
>>    ip addr del $ip_min/24 dev br0
>>    ip addr del $ip_maj/24 dev br0
>>    brctl delbr br0
>>    ip link set dev $dev down
>>    ip addr del $ip_min/24 dev $dev
>>    ip addr del $ip_maj/24 dev $dev
>>    echo ""
>> }
>>
>> set_hostname_and_localhost(){
>>    echo "Setting hostname and localhost"
>>    hostname=`grep -v "^\s*#"  /etc/hostname | head -n1`
>>    ip link set dev lo up
>>    echo ""
>> }
>>
>> create_phys_device_link(){
>>    dev=$1
>>    echo Creating device link for $dev
>>    ip link set dev $dev up
>>    echo ""
>> }
>>
>> set_phys_device_addr(){
>>    dev=$1
>>    ip_maj=$2
>>    ip_min=$3
>>    gateway=$4
>>    echo -n "Setting physical device addresses "
>>    echo -n "$ip_maj "
>>    echo -n "and $ip_min "
>>    echo -n "for $physdev "
>>    echo "with gateway $gateway"
>>    ip link set dev $dev down
>>    ip addr add $ip_maj/24 dev $dev
>>    ip addr add $ip_min/24 dev $dev
>>    ip link set dev $dev up
>>    ip route add default via $gateway
>>    echo ""
>> }
>>
>> set_bridge(){
>>    dev=$1
>>    ip_maj=$2
>>    ip_min=$3
>>    gateway=$4
>>    echo Setting bridge for $dev
>>    echo -n "Creating and setting bridge addresses "
>>    echo -n "$ip_maj "
>>    echo -n "and $ip_min "
>>    echo -n "for $physdev "
>>    echo "with gateway $gateway"
>>
>>    ip link add name br0 type bridge
>>    ip link set dev $dev master br0
>>    ip addr add $ip_maj/24 dev br0
>>    ip addr add $ip_min/24 dev br0
>>    ip link set dev br0 up
>>    ip route add default via $gateway
>>    echo ""
>> }
>>
>> set_tap(){
>>    echo Setting tap
>>    ip tuntap add tap0 mode tap
>>    brctl addif br0 tap0
>>    #ip addr add 192.168.0.66/24 dev tap0
>>    ip link set dev tap0 up
>>    echo ""
>> }
>>
>> show_networking(){
>>    echo -n "Networking follows in 3 seconds..."
>>    sleep 3
>>    echo "\n"
>>    echo "========================================"
>>    echo "========================================"
>>    ip -4 link
>>    echo "......................"
>>    ip -4 addr
>>    echo "......................"
>>    ip -4 route
>>    echo "========================================"
>>    echo "========================================"
>> }
>>
>> echo "\nBegin upnet.sh"
>>
>> [ "$use_tap" = "1" ] && [ "$use_bridge" != "1" ] && \
>>    error_tap_without_bridge
>>
>> unset_everything $dev $ipaddr_major $ipaddr_minor $gateway
>>
>> enable_ip_forwarding
>>
>> set_hostname_and_localhost
>>
>> create_phys_device_link $dev $ipaddr_major $ipaddr_minor $gateway
>>
>> [ "$use_bridge" = "1" ] || \
>>    set_phys_device_addr $dev $ipaddr_major $ipaddr_minor $gateway
>>
>> [ "$use_bridge" = "1" ] && set_bridge $dev \
>>    $ipaddr_major $ipaddr_minor $gateway
>>
>> [ "$use_tap" = "1" ] && \
>>    set_tap $dev $ipaddr_major $ipaddr_minor $gateway
>>
>> show_networking
>> =========================================
>>
>> The preceding script just builds br0 with ip addresses 192.168.0.2 and
>> 192.168.0.102, default route (gateway) 192.168.0.1, for my metal host,
>> and runs every time my metal host is rebooted (or it can be run any
>> time). It has provisions to build a tap, or to not build a bridge and
>> instead assign the IP addresses and default route to enp40s0 itself.
>>
>> The next shellscript runs on my metal host (Void Linux) to launch a
>> (Devuan) VM guest:
>>
>> =========================================
>> #!/bin/sh
>>
>> dvddir=/scratch/linuxinst/devuan/devuan_beowulf/installer-iso
>>
>> qemu-system-x86_64 --enable-kvm \
>> -cdrom $dvddir/devuan_beowulf_3.0.0_amd64-desktop.iso \
>> -hda /scratch/qemu_images/beowulf.disk \
>> -m 4G \
>> -boot c \
>> -netdev bridge,id=mybridge0,br=br0 \
>> -netdev user,id=mynet0,restrict=no,net=192.168.0.0/24 -device
>> e1000,netdev=mynet0 \
>>
>>
>> =========================================
>>
>> In the preceding, notice that the blank line at the end is necessary to
>> end the series of backslashed lines. I backslash ALL the lines so
>> experimentally I can move things in and out and rearrange. Like I said,
>> I've tried hundreds of combinations.
>>
>> In the preceding, I didn't declare a TAP. As best I could read the
>> various conflicting documentation, the VM guest *creates* the tap and
>> connects it to the metal host created bridge, so no need to create the
>> TAP on the metal host. But then again, I've tried it both ways.
>>
>> One hint I've gotten, a hint which I cannot bring to fruition because
>> of sparse and contradictory documentation, is that the preceding script
>> brings up a "no peer for mybridge0" warning on the terminal from which
>> it was run. I've tried substituting "eth0", enp40s0, and "junk" for
>> "mybridge0", and as I remember (hundreds of experiments, didn't write
>> them all down), doing so didn't affect the symptom (the no peer
>> warning). I chose the preceding script due to the following
>> documentation, which seemed better than most:
>>
>>
>> https://www.debian.org/doc/manuals/debian-handbook/sect.virtualization.en.html#sect.lxc.network
>>
>> On my Devuan *guest* VM, I have the following /etc/network/interfaces,
>> as suggested by a careful reading of
>>
>> https://www.debian.org/doc/manuals/debian-handbook/sect.virtualization.en.html#sect.lxc.network
>>
>> =========================================
>> source /etc/network/interfaces.d
>>
>> auto lo
>> iface lo inet loopback
>>
>> auto eth0
>> allow-hotplug eth0
>>
>> auto tap0
>> iface tap0 ifacem manual
>>   vde-switch -t tap0
>>
>> auto br0 inet static
>>   bridge-ports tap0
>>   address 192.168.0.60
>>   netmask 255.255.255.0
>> =========================================
>>
>> When, on the *host*, I run my instantiations script to create a guest
>> VM, the following is output to the terminal:
>>
>> =========================================
>> [root@mydesk qemu_images]# ./runbeowulf.sh
>> WARNING: Image format was not specified for
>> '/scratch/qemu_images/beowulf.disk' and probing guessed raw.
>> Automatically detecting the format is dangerous for raw images, write
>> operations on block 0 will be restricted. Specify the 'raw' format
>> explicitly to remove the restrictions. qemu-system-x86_64: warning:
>> netdev mybridge0 has no peer
>> =========================================
>>
>> After that, my Devuan guest VM appears, to which I log in and run a
>> terminal. IP addresses are 192.168.0.15 for the guest VM itself, and
>> 192.168.0.2 (my metal host) for the default route.
>>
>> The VM guest can lynx to my nginx server on 192.168.0.2, and to any
>> HTML page on the Internet, but cannot lynx to my printer at
>> 192.168.0.13 or my metal cable modem at 192.168.0.1. From my metal
>> desktop (which runs the guest VM) at 192.168.0.2 I cannot ssh to
>> slitt@192.168.0.15:
>>
>> =========================================
>> [slitt@mydesk qemu]$ ssh slitt@192.168.0.15
>> ssh: connect to host 192.168.0.15 port 22: No route to host
>> [slitt@mydesk qemu]$
>> =========================================
>>
>> I'm pretty sure "no route to host" means this isn't caused by a
>> firewall problem, although once I fix the routing thing, that might
>> unmask a further firewall problem.
>>
>> In other words, my VM guest is in no way a peer of the various metal
>> hosts on my 192.168.0.0/24 physical Ethernet network.
>>
>> If anybody has any words of wisdom, and can identify whether each
>> wisdom word applies to the metal host or the guest VM, I'd love to hear
>> them. I could also use info on what the "id=" should be, and how to
>> narrow down and diagnose a "has no peer" warning.
>>
>> Thanks,
>>
>> SteveT
>>
>> Steve Litt
>> Autumn 2020 featured book: Thriving in Tough Times
>> http://www.troubleshooters.com/thrive
>>
>>


-- 
Alex Bennée