VXLAN routing does not work

Hello all!
We have the Pritunl Enterprise subscription and configured two Pritunl hosts for redundancy. Also we enabled the option that provides usage vxlan for client-to-client connections. But when two clients sit on the different hosts, vxlan does not work and clients can not see each other. If these clients sit on the same host - everything works well (client is able to reach other client)
I have checked the vxlan interfaces and found that both have the same MAC address. Is this a root cause of this issue?
Also, I checked the MongoDB collection vxlan and found the same MACs there

The Pritunl code does not set a MAC address for the VXLan interfaces and leaves the system to generate one. Two hosts should not have the same MAC address. I did find the information below from ChatGPT on the methods used to generate that address. It’s possible you cloned the system and caused one of the identifiers to be duplicated.

In Linux, when a VXLan (Virtual eXtensible LAN) interface is created, the MAC address for the interface is generated using the following mechanism:

The MAC address for a VXLan interface consists of three parts:

Fixed prefix: The first 24 bits (3 bytes) of the MAC address are usually set as a fixed prefix. The most common prefix used is 00:00:5E (which corresponds to the “locally administered” range of MAC addresses). This prefix helps identify that the MAC address belongs to a VXLan interface.

Virtual Network Identifier (VNI): The next 24 bits (3 bytes) represent the VNI associated with the VXLan interface. The VNI is a unique identifier that distinguishes different virtual networks within the overlay network. It allows different VXLan segments to operate independently.

Instance identifier: The remaining 16 bits (2 bytes) are used as an instance identifier. This identifier is assigned by the Linux kernel and ensures that multiple VXLan interfaces with the same VNI can coexist on the same host without MAC address conflicts.

To generate the MAC address, the Linux kernel combines the fixed prefix, VNI, and instance identifier using bitwise operations and assigns the resulting MAC address to the VXLan interface.

For example, let’s say we have a VXLan interface with a VNI of 1234 and an instance identifier of 1. The resulting MAC address would be 00:00:5E:04:D2:01, where 00:00:5E is the fixed prefix, 04:D2 is the VNI (1234 in hexadecimal), and 01 is the instance identifier (1 in hexadecimal).

It’s important to note that this MAC address generation mechanism is specific to Linux’s implementation of VXLan interfaces and may vary slightly in other systems or network virtualization technologies.

Thank you for your answer.
I have delved into this issue and found out the following - if we start a Pritunl Server, then Pritunl codebase executes the Linux command for creating a vxlan interface. In my case, I ran into the issue when OS Ubuntu generates the same MAC addresses each time for a specific VXLAN_ID, and since the EC2 instances are created from an ASG, all instances have the same MAC address for the same vxlan interface. It is weird, but that is true

To solve this issue I have created a udev rule which helps me to set a random MAC address on an interface after a new vxlan interface is created

cat /etc/udev/rules.d/99-vxlan-custom.rules
ACTION=="add", SUBSYSTEM=="net", KERNEL=="pxlan*", RUN+="/99-vxlan-custom.sh %k"

the script is

cat /99-vxlan-custom.sh

/usr/bin/ip link set dev $1 address `echo 00:00:1F:$[RANDOM%10]$[RANDOM%10]:$[RANDOM%10]$[RANDOM%10]:$[RANDOM%10]$[RANDOM%10]`

BUT the udev rule is not atomic and applies over time.

The Pritunl code is very fast and we need to make a delay for applying the udev rule
I have submitted a PR for adding this delay