LFCE 31 🐧 VXLAN — Configuration and Use - KandZ

⏱️ Estimated read: 12 min | |

The previous chapter established the concepts: VXLAN encapsulates Layer 2 frames inside UDP/IP packets, extends the identifier space to 24 bits, and allows a Layer 2 segment to span a Layer 3 underlay. This chapter is the configuration sequence. It covers the ip link commands that create a VXLAN interface, the two peer-discovery models (unicast and multicast), the forwarding database that maps MAC addresses to remote VTEPs, the MTU adjustment that prevents fragmentation, and the persistence mechanisms for the three major network managers.

The distinction between a VXLAN tunnel that works in a running kernel and one that survives a reboot is the central concern. ip link add type vxlan creates an interface that disappears on restart. nmcli, systemd-networkd, and Netplan write the same configuration to files that are applied at boot. The running configuration is verified with ip -d link show and bridge fdb show; the persistent configuration is verified by rebooting.

Key point: VXLAN adds 50 bytes of overhead to every packet: 14 bytes for the inner Ethernet header, 20 bytes for the outer IP header, 8 bytes for the UDP header, and 8 bytes for the VXLAN header . On a standard 1500-byte underlay, the overlay MTU must be reduced to 1450. Without this adjustment, large packets are silently dropped or fragmented, producing TCP stalls that are difficult to diagnose.

Why VXLAN configuration requires explicit steps

The peer-discovery problem.

A VXLAN tunnel endpoint (VTEP) must know the IP address of every other VTEP that participates in the same VNI. There are two mechanisms: unicast and multicast. In unicast mode, the remote VTEP is specified directly with the remote parameter. This is simple and works on any IP underlay, but it scales linearly because the source VTEP must send a copy of broadcast traffic to each peer . In multicast mode, the VTEP joins a multicast group and sends broadcast, unknown unicast, and multicast (BUM) traffic to the group address. This provides automatic peer discovery, but it requires multicast routing in the underlay, which many networks do not support .

The MTU problem.

The 50-byte encapsulation overhead reduces the effective payload from 1500 bytes to 1450 bytes. If the overlay MTU is left at 1500, the underlay sees packets that exceed its MTU. Fragmentation may occur, or the packets may be dropped entirely. The VXLAN interface and any bridge it is attached to must have their MTU reduced to 1450, or the underlay must support jumbo frames large enough to absorb the overhead . TCP MSS clamping is also recommended to prevent TCP connections from negotiating a segment size that exceeds the tunnel capacity.

The FDB problem.

The forwarding database maps MAC addresses to remote VTEP IP addresses. It is populated dynamically when traffic arrives from a remote VTEP, or statically when the administrator adds entries. If the FDB is empty and no flood entry exists, the VTEP has no way to reach a destination MAC. The flood entry (all-zero MAC) tells the VTEP where to send BUM traffic. In unicast mode without a flood entry, the VTEP cannot discover remote MACs automatically .

a. Creating a VXLAN interface with unicast peers

The simplest VXLAN configuration is a point-to-point tunnel between two VTEPs. The remote parameter specifies the peer’s underlay IP address, and the local parameter specifies the source address for the outer IP header.

# Create a VXLAN interface with VNI 100
# The underlay is eth0; local VTEP is 10.0.0.1; remote VTEP is 10.0.0.2
ip link add vxlan100 type vxlan \
    id 100 \
    local 10.0.0.1 \
    remote 10.0.0.2 \
    dev eth0 \
    dstport 4789

# Bring the interface up
ip link set vxlan100 up

# Assign an overlay IP address
ip addr add 192.168.100.1/24 dev vxlan100

# Verify the interface
ip -d link show vxlan100
# Output includes: vxlan id 100 ... dstport 4789

The dstport is set to 4789, the IANA-assigned VXLAN port . Linux historically defaulted to 8472 for backward compatibility, but 4789 is the standard that interoperates with other VXLAN implementations . The local and remote parameters define the endpoints of the tunnel. The dev parameter specifies the physical interface that carries the encapsulated packets.

On the remote host, the configuration is mirrored: local and remote addresses are swapped, and the overlay IP is on the same subnet but a different address.

# On the remote host (10.0.0.2)
ip link add vxlan100 type vxlan \
    id 100 \
    local 10.0.0.2 \
    remote 10.0.0.1 \
    dev eth0 \
    dstport 4789
ip link set vxlan100 up
ip addr add 192.168.100.2/24 dev vxlan100

With this configuration, the two overlay IPs can reach each other through the tunnel. The underlay routes the encapsulated packets between 10.0.0.1 and 10.0.0.2, and the VTEPs decapsulate them to deliver the inner frames.

b. Using multicast for peer discovery

When more than two VTEPs participate in the same VNI, a multicast group provides automatic discovery. The group parameter specifies the multicast address, and the dev parameter specifies the underlay interface.

# Create a VXLAN interface with multicast peer discovery
ip link add vxlan100 type vxlan \
    id 100 \
    group 239.1.1.1 \
    dev eth0 \
    dstport 4789
ip link set vxlan100 up
ip addr add 192.168.100.1/24 dev vxlan100

The VTEP joins the multicast group 239.1.1.1. When it needs to send BUM traffic, it sends the encapsulated frame to the group address. Every VTEP that has joined the group receives the frame and decapsulates it. The underlay must be configured to route the multicast group between the VTEPs, which may require PIM or IGMP configuration on the physical routers .

For environments where multicast routing is unavailable, head-end replication is the alternative. The VXLAN interface is created without a remote or group parameter, and the flood entry in the FDB is populated manually with each remote VTEP address .

# Create VXLAN without remote or group
ip link add vxlan100 type vxlan id 100 dev eth0 dstport 4789
ip link set vxlan100 up

# Add flood entries for each remote VTEP
bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.0.2
bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.0.3

The all-zero MAC address is the flood entry. When a frame needs to be sent to an unknown destination, the VTEP duplicates it and sends a copy to each address in the flood list. This is head-end replication: the source VTEP performs the replication that multicast would otherwise handle .

c. Configuring the MTU and MSS

The VXLAN interface and any bridge it is attached to must have their MTU reduced to account for the encapsulation overhead. On a standard 1500-byte underlay, the overlay MTU is 1450.

# Set the VXLAN interface MTU
ip link set vxlan100 mtu 1450

# If the VXLAN is attached to a bridge, set the bridge MTU too
ip link set br0 mtu 1450

# Verify
ip link show vxlan100 | grep mtu
# Output: mtu 1450

The 50-byte overhead consists of the outer IP header (20 bytes), the outer UDP header (8 bytes), the VXLAN header (8 bytes), and the inner Ethernet header (14 bytes) . If the underlay supports jumbo frames, the overlay MTU can be larger. For a 9000-byte underlay, the overlay MTU is 8950.

TCP MSS clamping prevents TCP connections from negotiating a segment size that exceeds the tunnel capacity. The MSS is the VXLAN MTU minus 40 bytes (the IP and TCP headers).

# Clamp MSS for TCP traffic traversing the VXLAN interface
iptables -t mangle -A FORWARD -o vxlan100 -p tcp --tcp-flags SYN,RST SYN \
    -j TCPMSS --set-mss 1410

The value 1410 is 1450 minus 40. Without clamping, a TCP connection might negotiate an MSS based on the physical MTU, producing segments that the VXLAN interface cannot forward without fragmentation .

Complete Example Session

# ============================================
# PART 1: CREATE THE VXLAN INTERFACE
# ============================================
# VNI 100, local VTEP 10.0.0.1, remote VTEP 10.0.0.2
ip link add vxlan100 type vxlan \
    id 100 \
    local 10.0.0.1 \
    remote 10.0.0.2 \
    dev eth0 \
    dstport 4789

# ============================================
# PART 2: SET THE MTU
# ============================================
# Account for 50 bytes of encapsulation overhead
ip link set vxlan100 mtu 1450

# ============================================
# PART 3: BRING UP AND ASSIGN OVERLAY IP
# ============================================
ip link set vxlan100 up
ip addr add 192.168.100.1/24 dev vxlan100

# ============================================
# PART 4: VERIFY THE INTERFACE
# ============================================
ip -d link show vxlan100
# Output confirms VNI 100, dstport 4789, and MTU 1450

# ============================================
# PART 5: ADD A STATIC FDB ENTRY
# ============================================
# Map a known remote MAC to the remote VTEP
bridge fdb add 00:11:22:33:44:55 dst 10.0.0.2 dev vxlan100

# ============================================
# PART 6: INSPECT THE FDB
# ============================================
bridge fdb show dev vxlan100
# Output includes the static entry and any learned entries

# ============================================
# PART 7: ATTACH TO A BRIDGE
# ============================================
ip link add br0 type bridge
ip link set br0 mtu 1450
ip link set vxlan100 master br0
ip link set br0 up

# ============================================
# PART 8: VERIFY ENCAPSULATION ON THE UNDERLAY
# ============================================
tcpdump -i eth0 -n udp port 4789
# Output shows outer IP and UDP headers with VXLAN payload

# ============================================
# PART 9: TEST CONNECTIVITY
# ============================================
# From the overlay interface, ping the remote overlay IP
ping -c 2 192.168.100.2

# ============================================
# PART 10: PERSIST WITH NMCLI
# ============================================
nmcli connection add type vxlan \
    ifname vxlan100 \
    con-name vxlan100 \
    vxlan.id 100 \
    vxlan.local 10.0.0.1 \
    vxlan.remote 10.0.0.2 \
    vxlan.destination-port 4789 \
    ipv4.method manual \
    ipv4.addresses "192.168.100.1/24"
nmcli connection up vxlan100

The ten parts demonstrate the complete VXLAN lifecycle: interface creation, MTU adjustment, overlay addressing, FDB configuration, bridge attachment, encapsulation verification, connectivity testing, and persistence. The critical steps are Part 2 (MTU) and Part 4 (verification with ip -d).

Quick Reference

VXLAN Configuration Parameters

Parameter	Purpose	Example
`id`	VNI (24-bit)	`id 100`
`local`	Local VTEP IP	`local 10.0.0.1`
`remote`	Remote VTEP IP (unicast)	`remote 10.0.0.2`
`group`	Multicast group (multicast mode)	`group 239.1.1.1`
`dev`	Underlay interface	`dev eth0`
`dstport`	UDP destination port	`dstport 4789`

FDB Commands

Task	Command
Show FDB	`bridge fdb show dev vxlan100`
Add static entry	`bridge fdb add <mac> dst <vtep> dev vxlan100`
Add flood entry	`bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst <vtep>`
Delete entry	`bridge fdb delete <mac> dev vxlan100`

MTU Calculation

Underlay MTU	Overlay MTU	TCP MSS
1500	1450	1410
9000	8950	8910

Persistence Methods

System	Method
RHEL/NetworkManager	`nmcli connection add type vxlan`
systemd-networkd	`.netdev` with `Kind=vxlan`
Netplan	YAML `vlans` or `tunnels` section

Best Practices

✅ Do This:

# Set the IANA-standard port explicitly
ip link add vxlan100 type vxlan id 100 dstport 4789 ...      # ✅

# Reduce overlay MTU to underlay minus 50
ip link set vxlan100 mtu 1450                                 # ✅

# Set the bridge MTU to match the VXLAN MTU
ip link set br0 mtu 1450                                      # ✅

# Clamp MSS for TCP traffic
iptables -A FORWARD -o vxlan100 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1410 # ✅

# Add flood entries when multicast is unavailable
bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst <vtep>   # ✅

# Verify with ip -d and bridge fdb show
ip -d link show vxlan100; bridge fdb show dev vxlan100         # ✅

❌ Don’t Do This:

# Leave the default Linux port (8472) when interoperating
# Other vendors expect 4789                                     # ❌

# Ignore MTU overhead
# 1500-byte overlay packets become 1550 bytes on the wire        # ❌

# Assume multicast works everywhere
# Many underlays do not route multicast                          # ❌

# Forget to set the bridge MTU
# The bridge forwards frames larger than the VXLAN can carry     # ❌

# Use VXLAN when a simple bridge or VLAN is sufficient
# VXLAN adds complexity that is unnecessary for small networks   # ❌

# Forget to open UDP 4789 in the firewall
# Encapsulated packets are dropped silently                      # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
VXLAN interface has no traffic	Underlay cannot reach remote VTEP	Verify `ip route get <vtep-ip>`
Fragmentation on underlay	Overlay MTU exceeds underlay capacity	Set MTU to 1450
No FDB entries	No traffic has flowed	Generate traffic or add static entries
Multicast discovery fails	Underlay does not route multicast	Use unicast with static peers
Port mismatch	Vendor uses different UDP port	Set `dstport 4789` explicitly
TCP stalls	MSS too large for tunnel	Apply MSS clamping

Real-World Examples

1. Point-to-point VXLAN

ip link add vxlan100 type vxlan id 100 local 10.0.0.1 remote 10.0.0.2 dev eth0 dstport 4789

2. Multicast VXLAN

ip link add vxlan100 type vxlan id 100 group 239.1.1.1 dev eth0 dstport 4789

3. VXLAN with MTU

ip link set vxlan100 mtu 1450

4. Static FDB entry

bridge fdb add 00:11:22:33:44:55 dst 10.0.0.2 dev vxlan100

5. Flood entry for head-end replication

bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.0.3

6. Attach VXLAN to bridge

ip link set vxlan100 master br0

7. MSS clamping

iptables -A FORWARD -o vxlan100 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1410

8. Verify with tcpdump

tcpdump -i eth0 -n udp port 4789

9. Persist with nmcli

nmcli connection add type vxlan ifname vxlan100 vxlan.id 100 vxlan.local 10.0.0.1 vxlan.remote 10.0.0.2

10. Persist with systemd-networkd

[NetDev]
Name=vxlan100
Kind=vxlan
[VXLAN]
VNI=100
Remote=10.0.0.2
Local=10.0.0.1
DestinationPort=4789

Visual

Unicast vs Multicast VXLAN

┌──────────────────────────────────────────────────────────────┐
│  UNICAST: EXPLICIT PEER                                       │
│                                                              │
│  VTEP A (10.0.0.1) ──────▶ VTEP B (10.0.0.2)                │
│  ip link add ... remote 10.0.0.2                             │
│                                                              │
│  Simple, works on any IP underlay.                           │
│  BUM traffic requires head-end replication.                  │
│                                                              │
│  MULTICAST: GROUP-BASED DISCOVERY                            │
│                                                              │
│  VTEP A (10.0.0.1) ──┐                                       │
│  VTEP B (10.0.0.2) ──┼──▶ Multicast Group 239.1.1.1         │
│  VTEP C (10.0.0.3) ──┘                                       │
│                                                              │
│  ip link add ... group 239.1.1.1                             │
│                                                              │
│  Automatic discovery.                                        │
│  Requires multicast routing in the underlay.                 │
└──────────────────────────────────────────────────────────────┘

MTU and Encapsulation

┌──────────────────────────────────────────────────────────────┐
│  VXLAN OVERHEAD REDUCES EFFECTIVE MTU                        │
│                                                              │
│  Underlay frame (1500 bytes):                                │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Outer IP (20) │ UDP (8) │ VXLAN (8) │ Inner Eth (14) │  │
│  │               │         │           │ + Payload      │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                              │
│  Overhead: 20 + 8 + 8 + 14 = 50 bytes                        │
│                                                              │
│  Overlay MTU = 1500 - 50 = 1450 bytes                        │
│                                                              │
│  If overlay MTU is 1500:                                     │
│  → Underlay frame is 1550 bytes                              │
│  → Exceeds 1500-byte underlay MTU                            │
│  → Fragmentation or drop                                     │
└──────────────────────────────────────────────────────────────┘

FDB Entry Types

┌──────────────────────────────────────────────────────────────┐
│  FDB MAPS MAC TO VTEP                                        │
│                                                              │
│  bridge fdb show dev vxlan100                                │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ 00:00:00:00:00:00 dst 10.0.0.2  ← Flood entry          │  │
│  │ 00:00:00:00:00:00 dst 10.0.0.3  ← Flood entry          │  │
│  │ aa:bb:cc:dd:ee:01 dst 10.0.0.2  ← Learned/static       │  │
│  │ aa:bb:cc:dd:ee:02 dst 10.0.0.3  ← Learned/static       │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                              │
│  All-zero MAC = BUM traffic destination                      │
│  Real MAC = unicast forwarding destination                   │
└──────────────────────────────────────────────────────────────┘

Summary

Item	Value
Create VXLAN	`ip link add type vxlan id <vni>`
Unicast peer	`remote <vtep-ip>`
Multicast group	`group <multicast-addr>`
UDP port	`dstport 4789` (IANA)
MTU adjustment	Underlay MTU minus 50
FDB show	`bridge fdb show dev <vxlan>`
Static FDB entry	`bridge fdb add <mac> dst <vtep>`
Flood entry	`bridge fdb append 00:00:00:00:00:00 ...`
MSS clamping	`--set-mss` (MTU minus 40)
Persistence	nmcli, systemd-networkd, Netplan

Key takeaways:

VXLAN adds 50 bytes of overhead. The overlay MTU must be reduced from 1500 to 1450 on a standard underlay. Without this adjustment, packets are fragmented or dropped, and the resulting TCP stalls are difficult to diagnose .
Unicast is simpler; multicast discovers automatically. Unicast requires explicit peer addresses but works on any IP underlay. Multicast provides automatic discovery but requires multicast routing, which many networks do not support .
The FDB is the forwarding table. It maps MAC addresses to VTEP IPs. The all-zero MAC entry is the flood entry for BUM traffic. Static entries eliminate the need for dynamic learning .
The IANA port is 4789. Linux historically defaulted to 8472. Setting dstport 4789 explicitly ensures interoperability with other VXLAN implementations .
MSS clamping prevents TCP stalls. A TCP connection that negotiates an MSS based on the physical MTU will produce segments that the VXLAN interface cannot forward. Clamping the MSS to the tunnel MTU minus 40 bytes prevents this .
Persistence depends on the network manager. nmcli for RHEL, systemd-networkd .netdev files for Arch and others, and Netplan for Ubuntu. Each writes the configuration to disk and applies it at boot .

Remember: VXLAN configuration is a sequence of dependencies. The underlay must route between VTEPs. The VXLAN interface must be created with the correct VNI and endpoints. The MTU must be reduced to account for encapsulation. The FDB must contain the remote VTEPs, either through multicast discovery or static entries. And the configuration must be persisted through the network manager, or it disappears on reboot. Verify each step with ip -d link show, bridge fdb show, and tcpdump on the underlay. The overlay is invisible to the underlay routers, but it depends entirely on their correct configuration.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!