LFCE 30 🐧 Overlay Networks — Concepts
An overlay network is a virtual network built on top of an existing physical network. The physical network, called the underlay, provides basic IP connectivity between hosts. The overlay creates a separate logical topology on top of it, encapsulating one type of packet inside another so that the underlay only sees the outer headers. The most common overlay protocol in Linux environments is VXLAN (Virtual Extensible LAN), which tunnels Layer 2 Ethernet frames inside UDP/IP packets. This allows a Layer 2 broadcast domain to span multiple physical racks, data centers, or cloud regions without requiring the underlay to be a single Layer 2 segment.
The previous chapters covered bridges, VLANs, and virtual interfaces. VLANs extend Layer 2 within a single physical infrastructure, limited to 4,094 usable VLAN IDs. VXLAN extends Layer 2 across a Layer 3 boundary and expands the identifier to 24 bits, allowing roughly 16 million segments . This is the mechanism that makes large-scale container and virtual machine networking possible.
Key point: The overlay is transparent to the workloads. A virtual machine or container sends a normal Ethernet frame. The VXLAN tunnel endpoint (VTEP) encapsulates that frame in a UDP/IP packet addressed to the remote VTEP’s underlay IP. The underlay routes the packet to the destination host, where the remote VTEP decapsulates it and delivers the original Ethernet frame to the destination workload. The workloads never see the encapsulation.
Why overlay networks exist
The VLAN scale problem.
IEEE 802.1Q provides 12 bits for the VLAN identifier, with 0 and 4095 reserved, leaving 4,094 usable VLANs . In a data center with thousands of tenants, each requiring multiple isolated networks, this limit is reached quickly. VXLAN expands the identifier to 24 bits, producing 16,777,216 possible VXLAN Network Identifiers (VNIs) . The overlay can support a scale that the underlay’s VLAN space cannot.
The Layer 2 stretch problem.
A virtual machine needs to keep its IP address when it migrates from one host to another. If the hosts are on different Layer 3 subnets, the VM’s IP address would be invalid after migration. An overlay network solves this by presenting a single Layer 2 segment to the VMs, regardless of where they are physically located. The VMs believe they are on the same switch, even though the underlay is routing packets between data centers . The overlay provides the Layer 2 continuity that live migration requires.
The underlay independence problem.
The physical network does not need to know about the overlay’s topology. The underlay only needs to route IP packets between the VTEP addresses. The overlay can be reconfigured, expanded, or segmented without touching the physical switches. This separation allows the network to be managed at two levels: the underlay for physical connectivity, and the overlay for tenant isolation and logical topology.
a. VXLAN encapsulation and the VTEP
A VXLAN Tunnel Endpoint (VTEP) is the entity that performs encapsulation and decapsulation. On Linux, a VTEP is a virtual network interface created with ip link add type vxlan. It has its own VNI and is associated with an underlay interface and a local IP address.
# Create a VXLAN interface with VNI 100
# The underlay is eth0; the local VTEP address is 10.0.0.1
ip link add vxlan100 type vxlan \
id 100 \
dstport 4789 \
local 10.0.0.1 \
dev eth0
# Assign an overlay IP and bring up the interface
ip addr add 192.168.100.1/24 dev vxlan100
ip link set vxlan100 upThe id parameter is the VNI. The dstport is the UDP port; the IANA-assigned value is 4789, but Linux historically defaulted to 8472 for backward compatibility . The local parameter specifies the source IP address used in the outer IP header. The dev parameter specifies the underlay interface.
When a frame is sent on vxlan100, the kernel wraps it in a VXLAN header (which contains the VNI), then a UDP header, then an IP header, and then an Ethernet header for the underlay. The outer packet is addressed to the remote VTEP’s IP. The encapsulation adds approximately 50 bytes of overhead: the inner Ethernet header, the VXLAN header, the UDP header, and the outer IP header . This overhead must be accounted for in the underlay MTU.
b. Peer discovery: multicast and unicast
A VTEP must know the IP address of every other VTEP that participates in the same VNI. There are two mechanisms for this: multicast and unicast.
# Multicast-based VTEP discovery
# The VTEP joins a multicast group; frames to unknown destinations
# are sent to the group, and all VTEPs in the group receive them
ip link add vxlan100 type vxlan \
id 100 \
group 239.1.1.1 \
dev eth0 \
dstport 4789With multicast, the VTEP sends broadcast, unknown unicast, and multicast (BUM) traffic to a multicast group address. Every VTEP that has joined that group receives the frame. The underlay must be configured to route the multicast group between the VTEPs, which can be complex at scale . Multicast provides automatic discovery and a decentralized design, but it requires multicast routing in the underlay.
# Unicast-based VTEP discovery with a static peer list
# The remote VTEP is specified explicitly
ip link add vxlan100 type vxlan \
id 100 \
remote 10.0.0.2 \
local 10.0.0.1 \
dev eth0 \
dstport 4789With unicast, the remote VTEP address is specified directly. For BUM traffic that must reach multiple VTEPs, head-end replication is used: the source VTEP duplicates the frame and sends a copy to each configured peer. This is simpler than multicast because it does not require multicast routing, but it scales linearly with the number of peers. The Linux kernel supports this through the forwarding database (FDB), where multiple remote VTEPs are associated with the all-zero MAC address .
c. The forwarding database and MAC learning
A VXLAN device maintains a forwarding database (FDB) that maps MAC addresses to remote VTEP IP addresses. When a frame needs to be sent to a MAC address, the VTEP looks up the address in the FDB. If an entry exists, the frame is encapsulated and sent to the associated VTEP. If no entry exists, the frame is flooded to all known VTEPs (through multicast or head-end replication).
# Inspect the VXLAN forwarding database
bridge fdb show dev vxlan100
# Output includes entries like:
# 00:00:00:00:00:00 dst 239.1.1.1 via eth0 self permanent
# 50:54:33:00:00:0b dst 10.0.0.2 selfThe all-zero MAC address entry represents the flood destination. In a multicast configuration, it points to the multicast group. In a unicast configuration, it points to a remote VTEP. The other entries are learned dynamically: when a frame arrives from a remote VTEP, the source MAC is recorded and associated with the VTEP’s IP address. Future frames to that MAC are sent directly to that VTEP .
# Add a static FDB entry
bridge fdb add 00:17:42:8a:b4:05 dst 192.19.0.2 dev vxlan100
# Delete a static FDB entry
bridge fdb delete 00:17:42:8a:b4:05 dev vxlan100Static entries are useful when the remote MAC is known in advance and dynamic learning is disabled. They eliminate the flooding step for known destinations, reducing the load on the underlay.
Complete Example Session
# ============================================
# PART 1: CREATE A VXLAN INTERFACE
# ============================================
# VNI 100, underlay eth0, local VTEP 10.0.0.1
ip link add vxlan100 type vxlan id 100 dev eth0 dstport 4789 local 10.0.0.1
ip link set vxlan100 up
# ============================================
# PART 2: ASSIGN AN OVERLAY IP
# ============================================
ip addr add 192.168.100.1/24 dev vxlan100
# ============================================
# PART 3: VERIFY THE VXLAN INTERFACE
# ============================================
ip -d link show vxlan100
# Output shows: vxlan id 100 ... dstport 4789
# ============================================
# PART 4: ADD A STATIC FDB ENTRY
# ============================================
# Send frames for this MAC to the remote VTEP at 10.0.0.2
bridge fdb add 00:11:22:33:44:55 dst 10.0.0.2 dev vxlan100
# ============================================
# PART 5: INSPECT THE FDB
# ============================================
bridge fdb show dev vxlan100
# Output includes the static entry and the flood entry
# ============================================
# PART 6: CREATE A BRIDGE FOR LOCAL INTERFACES
# ============================================
# Attach the VXLAN interface and a local veth to a bridge
ip link add br0 type bridge
ip link set vxlan100 master br0
ip link set br0 up
# ============================================
# PART 7: VERIFY ENCAPSULATION ON THE UNDERLAY
# ============================================
# Capture traffic on eth0 to see the VXLAN encapsulation
tcpdump -i eth0 -n udp port 4789
# Output shows outer IP, UDP, and VXLAN headers
# ============================================
# PART 8: CLEAN UP
# ============================================
ip link delete vxlan100
ip link delete br0The eight parts demonstrate the VXLAN lifecycle: interface creation, overlay addressing, FDB inspection, static entry configuration, bridge integration, and packet capture on the underlay. The critical verification is Part 7, where tcpdump shows the outer UDP/IP headers that the underlay sees.
Quick Reference
VXLAN Parameters
| Parameter | Purpose | Default |
|---|---|---|
id | VNI (24-bit identifier) | Required |
dstport | UDP destination port | 8472 (Linux), 4789 (IANA) |
local | Source IP for outer header | Chosen by kernel |
dev | Underlay interface | Required for multicast |
group | Multicast group for peer discovery | None |
remote | Unicast peer VTEP address | None |
VXLAN vs VLAN
| Property | VLAN (802.1Q) | VXLAN |
|---|---|---|
| Identifier size | 12 bits | 24 bits |
| Usable segments | 4,094 | 16,777,216 |
| Underlay | Layer 2 | Layer 3 (IP) |
| Encapsulation | 4-byte tag | ~50-byte header |
| MTU impact | 4 bytes | ~50 bytes |
Key Commands
| Task | Command |
|---|---|
| Create VTEP | ip link add vxlan100 type vxlan id 100 ... |
| Verify VTEP | ip -d link show vxlan100 |
| Show FDB | bridge fdb show dev vxlan100 |
| Add static entry | bridge fdb add <mac> dst <vtep-ip> dev vxlan100 |
| Capture underlay | tcpdump -i eth0 udp port 4789 |
Best Practices
✅ Do This:
# Set the IANA-standard UDP port explicitly
ip link add vxlan100 type vxlan id 100 dstport 4789 ... # ✅
# Account for encapsulation overhead in MTU
# Reduce overlay MTU to underlay MTU minus 50 # ✅
# Use static FDB entries for known remote MACs
bridge fdb add <mac> dst <vtep> dev vxlan100 # ✅
# Verify the VNI with ip -d
ip -d link show vxlan100 | grep "id 100" # ✅
# Use head-end replication when multicast is unavailable
# Specify multiple peers with the all-zero MAC entry # ✅❌ Don’t Do This:
# Forget to set the destination port
# Linux defaults to 8472, which may not match other vendors # ❌
# Ignore MTU overhead
# 1500-byte overlay frames become 1550 bytes on the wire # ❌
# Assume multicast works everywhere
# Many underlay networks do not route multicast # ❌
# Use VXLAN when VLANs are sufficient
# VXLAN adds complexity that is unnecessary for small networks # ❌
# Forget to open UDP port 4789 in the firewall
iptables -A INPUT -p udp --dport 4789 -j ACCEPT # ✅ (do this)Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| VXLAN interface has no traffic | Underlay cannot reach remote VTEP IP | Verify ip route get <vtep-ip> |
| Fragmentation on underlay | Overlay MTU exceeds underlay capacity | Reduce VXLAN MTU by 50 |
| No FDB entries | No traffic has flowed | Generate traffic or add static entries |
| Multicast peer discovery fails | Underlay does not route multicast | Use unicast with static peers |
| Port mismatch | Vendor uses different UDP port | Set dstport 4789 explicitly |
Real-World Examples
1. Create a basic VXLAN interface
ip link add vxlan100 type vxlan id 100 dev eth0 dstport 47892. VXLAN with local VTEP address
ip link add vxlan100 type vxlan id 100 local 10.0.0.1 dev eth0 dstport 47893. VXLAN with unicast peer
ip link add vxlan100 type vxlan id 100 remote 10.0.0.2 local 10.0.0.1 dev eth04. VXLAN with multicast group
ip link add vxlan100 type vxlan id 100 group 239.1.1.1 dev eth0 dstport 47895. Assign overlay IP
ip addr add 192.168.100.1/24 dev vxlan1006. Show FDB entries
bridge fdb show dev vxlan1007. Add static FDB entry
bridge fdb add 00:11:22:33:44:55 dst 10.0.0.2 dev vxlan1008. Capture VXLAN traffic
tcpdump -i eth0 -n udp port 47899. Set overlay MTU
ip link set vxlan100 mtu 145010. Delete VXLAN interface
ip link delete vxlan100Visual
VXLAN Encapsulation
┌──────────────────────────────────────────────────────────────┐
│ THE OVERLAY FRAME IS WRAPPED IN AN UNDERLAY PACKET │
│ │
│ Overlay frame (what the VM sees): │
│ ┌──────────┬──────────┬──────────┬───────────────────────┐ │
│ │ Dst MAC │ Src MAC │ EtherType│ Payload │ │
│ └──────────┴──────────┴──────────┴───────────────────────┘ │
│ │ │
│ ▼ │
│ VXLAN encapsulation: │
│ ┌──────────┬──────────┬──────────┬──────────┬────────────┐ │
│ │ Outer IP │ Outer UDP│ VXLAN │ Inner │ Inner │ │
│ │ Header │ Header │ Header │ Ethernet │ Payload │ │
│ │ (VTEPs) │ (4789) │ (VNI) │ Frame │ │ │
│ └──────────┴──────────┴──────────┴──────────┴────────────┘ │
│ │
│ Overhead: ~50 bytes (outer IP + UDP + VXLAN + inner Eth) │
│ Underlay sees only the outer IP/UDP headers. │
└──────────────────────────────────────────────────────────────┘VTEP and Underlay Routing
┌──────────────────────────────────────────────────────────────┐
│ VTEPS COMMUNICATE THROUGH THE UNDERLAY IP NETWORK │
│ │
│ Host A (VTEP 10.0.0.1) Host B (VTEP 10.0.0.2) │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ vxlan100 (VNI 100) │ │ vxlan100 (VNI 100) │ │
│ │ 192.168.100.1 │ │ 192.168.100.2 │ │
│ └──────────┬──────────┘ └──────────┬──────────┘ │
│ │ │ │
│ │ Underlay IP │ │
│ ▼ ▼ │
│ ┌────────────────────────────────────────────────────────┐ │
│ │ Physical IP Network (Underlay) │ │
│ │ │ │
│ │ Routes packets between 10.0.0.1 and 10.0.0.2 │ │
│ │ Does not see the VXLAN encapsulation │ │
│ │ Does not know about VNI 100 │ │
│ └────────────────────────────────────────────────────────┘ │
│ │
│ The overlay is invisible to the underlay routers. │
└──────────────────────────────────────────────────────────────┘FDB Lookup
┌──────────────────────────────────────────────────────────────┐
│ VXLAN FORWARDING DATABASE MAPS MAC TO REMOTE VTEP │
│ │
│ Frame arrives with destination MAC 00:11:22:33:44:55 │
│ │ │
│ ▼ │
│ bridge fdb show dev vxlan100 │
│ ┌────────────────────────────────────────────────────────┐ │
│ │ 00:11:22:33:44:55 dst 10.0.0.2 │ │
│ │ 00:00:00:00:00:00 dst 239.1.1.1 (flood) │ │
│ └────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Entry found → encapsulate and send to 10.0.0.2 │
│ │
│ If no entry: flood to all VTEPs via multicast group │
└──────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Overlay protocol | VXLAN (RFC 7348) |
| Identifier | VNI, 24 bits, ~16 million segments |
| Encapsulation | Ethernet-in-UDP-in-IP |
| Overhead | ~50 bytes |
| VTEP | VXLAN Tunnel Endpoint |
| Peer discovery | Multicast group or unicast peers |
| FDB | MAC-to-VTEP mapping |
| Default UDP port | 8472 (Linux), 4789 (IANA) |
| MTU consideration | Reduce overlay MTU by 50 |
Key takeaways:
- VXLAN solves the VLAN scale limit. 802.1Q provides 4,094 usable VLANs; VXLAN provides 16,777,216 VNIs. This is the difference between a single data center and a multi-tenant cloud .
- The overlay is transparent to workloads. A VM or container sends a normal Ethernet frame. The VTEP encapsulates it, the underlay routes it, and the remote VTEP decapsulates it. The workload never sees the VXLAN header.
- Encapsulation adds ~50 bytes of overhead. The underlay MTU must be at least 50 bytes larger than the overlay MTU, or the overlay MTU must be reduced to compensate .
- Peer discovery is either multicast or unicast. Multicast provides automatic discovery but requires multicast routing in the underlay. Unicast requires explicit peer configuration but works on any IP network.
- The FDB maps MAC addresses to VTEP IPs. Learned entries are populated from incoming traffic. Static entries can be added for known destinations, eliminating the flooding step.
- The IANA-standard UDP port is 4789. Linux historically defaulted to 8472. Setting
dstport 4789explicitly ensures interoperability with other VXLAN implementations .
Remember: An overlay network is a logical topology built on top of a physical underlay. VXLAN is the Linux implementation: it encapsulates Layer 2 frames in UDP/IP packets, allowing Layer 2 segments to span Layer 3 boundaries. The VTEP is the endpoint that performs encapsulation and decapsulation. The FDB maps MAC addresses to remote VTEPs. The underlay routes the encapsulated packets without knowing about the overlay. VXLAN is the mechanism that makes large-scale container and VM networking possible, and it is the foundation of overlay networking in Kubernetes, OpenStack, and Docker.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!