SoCruel.NU

The domain that loves BSD

Home About Me
FreeBSD Security Miscellaneous
Archive Contact ☰

vm-bhyve and VLANs

Introduction

VLANs have not been used in the SoCruel.NU homelab. Using (just) a seperate layer 2 switch per network has worked out so far! Up to now! FreeBSD has support for VLANs and it is also well documented in the FreeBSD Handbook!

But I am also running virtual machines on my physical FreeBSD hosts. bhyve is used as the hypervisor and I use vm-bhyve to setup and manage my virtual machines. And both have VLAN support (see links in the resources section below). And all my virtual machines run (also) FreeBSD!

So I had some idea how this should work based on bits and pieces of online information, but I did not come across an article which describes this end to end. So I thought why not document it myself (as it has been a long time since my last blog post!)?

The first step is to get 2 physical FreeBSD hosts communicating with each other with both hosts connected to the same VLAN. And the second step is to get 2 FreeBSD virtual machines running on the same 2 physical hosts communicating with each other with both virtual machines connected to the same VLAN.

Physical hosts connected to one VLAN

The following technical prerequisites have to be in place for this implementation:

  • 2 up to date and supported physical FreeBSD systems running FreeBSD version 13.2
  • both above mentioned systems have an available unused network interface (which is in this case em2)
  • the available unused network interface of both systems is connected to a switch which supports VLANs
  • the ports of the switch used to connect the 2 physical FreeBSD are configured for VLAN 21 (It is out of the scope of this post how to actually do this)
  • the IP address range used for VLAN 21 is 172.16.21.0/24

The above described setup is shown in the basic diagram below.

2 physical FreeBSD hosts connected to one VLAN

We assume that the em2 interface of both physical FreeBSD are not enabled yet, so we perform the following command on both hosts:

$ sudo ifconfig em2 up

Then to configure interface em2 on Host A for VLAN 21 we use the command:

$ sudo ifconfig em2.21 create vlan 21 vlandev em2 inet 172.16.21.2/24

We do the same for interface em2 on Host B:

$ sudo ifconfig em2.21 create vlan 21 vlandev em2 inet 172.16.21.3/24

With the above configuration we can ping IP address 172.16.21.3 from Host A and we can ping IP address 172.16.21.2 from Host B.

So a VLAN setup with a physical interface on both hosts works as planned.

vm-bhyve managed VMs connected to one VLAN

Now we want to do the same with running a virtual machine on both hosts. We call the virtual machine running on physical host Host A vln-02. And the virtual machine on physical host Host B vln-03.

The above described setup is shown in the basic diagram below.

2 FreeBSD VMs connected to one VLAN

First we need to destroy the VLAN interface em2.21 on both physical hosts (Host A and Host B):

$ sudo ifconfig em2.21 destroy

Now we create a virtual switch, connect it to the interface em2 and assign the VLAN id 21. We do this on both physical hosts Host A and Host B:

$ sudo vm switch create test_vlan21
$ sudo vm switch add test_vlan21 em2
$ sudo vm switch vlan test_vlan21 21

We can check the result of the above with the command sudo vm switch list and the output should be like:

NAME         TYPE      IFACE           ADDRESS  PRIVATE  MTU  VLAN  PORTS
test_vlan21  standard  vm-test_vlan21  -        no       -    21    em2

The vm-bhyve configuration of the 2 virtual machines vln-02 and vln-03 is:

guest="freebsd"
loader="bhyveload"
cpu=1
memory=512M
network0_type="virtio-net"
network0_switch="test_vlan21"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"

Both virtual machines have a vtnet0 network interface. And we configure this interface for VLAN 21 in the /etc/rc.conf file. For vln-02 this configuration looks like:

ifconfig_vtnet0="up"
vlans_vtnet0="21"
ifconfig_vtnet0_21="inet 172.16.21.2/24"

And to check all is ok:

root@vln-03:~ # ifconfig vtnet0.21
vtnet0.21: flags=8843 metric 0 mtu 1500
        options=80000
        ether 58:9c:fc:05:90:57
        inet 172.16.21.3 netmask 0xffffff00 broadcast 172.16.21.255
        groups: vlan
        vlan: 21 vlanproto: 802.1q vlanpcp: 0 parent interface: vtnet0
        media: Ethernet autoselect (10Gbase-T )
        status: active
        nd6 options=29

And we do the same for vln-03:

ifconfig_vtnet0="up"
vlans_vtnet0="21"
ifconfig_vtnet0_21="inet 172.16.21.3/24"

And to check all is ok for vln-03 as well:

root@vln-03:~ # ifconfig vtnet0.21
vtnet0.21: flags=8843 metric 0 mtu 1500
        options=80000
        ether 58:9c:fc:05:90:57
        inet 172.16.21.3 netmask 0xffffff00 broadcast 172.16.21.255
        groups: vlan
        vlan: 21 vlanproto: 802.1q vlanpcp: 0 parent interface: vtnet0
        media: Ethernet autoselect (10Gbase-T )
        status: active
        nd6 options=29

And the work is done! We can now ping the IP address 172.16.21.3 from vln-02 and the IP address 172.16.21.2 from vln-03!

Resources

Some (other) resources about this subject:

Updated: October 10, 2023