Docker 65 🐳 Swarm Overlay Networks: VXLAN Encapsulation and Ingress Routing Mesh
Docker Swarm overlay networks solve a fundamental problem: how do containers running on different physical hosts communicate as if they were on the same machine? The answer has two layers. The first is the overlay network itself—a virtual Layer 2 network that spans all Swarm nodes, with container traffic encapsulated in UDP datagrams through the VXLAN protocol . The second is the ingress routing mesh—Swarm listens on published ports on every node and automatically forwards external requests to any container running the service, regardless of which node that container is on.
These two layers solve the hardest networking problems in container orchestration. Without overlay networks, containers are trapped in their host’s bridge network and cannot communicate across hosts. Without the ingress routing mesh, you would have to know which node runs which replica and manually configure an external load balancer with that knowledge. Swarm automates both: after creating an overlay network, services discover each other by DNS name; after publishing a port, any node in the cluster can accept requests and route them to the correct container.
This chapter goes deep into how overlay networks are implemented and how the routing mesh works. You will understand how VXLAN encapsulates Layer 2 frames inside UDP to cross Layer 3 networks, what roles the ingress network and docker_gwbridge each play, why the routing mesh lets every node respond on a published port, and how host mode bypasses it to preserve client source IPs. Finally, we will discuss the performance trade-offs of encrypted overlay networks and their known security limitations.
Key point: Swarm overlay networks use VXLAN to encapsulate Ethernet frames from containers inside UDP datagrams that traverse the underlying physical network; the ingress routing mesh lets every node in the Swarm accept connections on a published port and automatically route them to a healthy container on any node, even one that runs no task for that service .
Why overlay networks and the routing mesh exist
The cross-host communication problem. Every Docker host has its own bridge network, typically using the private 172.17.0.0/16 range. This means containers on two different hosts can have identical IP addresses, and those addresses are not routable outside their host . An overlay network solves this by creating a virtual, separately addressed network layer. Each container gets a unique IP on this virtual network regardless of which physical host it runs on. VXLAN encapsulates frames from the virtual network into UDP packets, sends them across the physical network to the target host, and decapsulates them there .
The service discovery problem. When you run multiple replicas of a web service across three nodes, frontend containers need to know how to reach the backend. If the backend replicas live at three different IP addresses, the frontend must know all of them and load-balance on its own. Swarm overlay networks include built-in DNS resolution: each service gets a virtual IP (VIP), a DNS query for the service name returns that VIP, and an internal IPVS load balancer distributes connections to healthy replicas . Frontend code only needs to connect to http://api:8080—no IP addresses required.
The external access problem. Even if containers can talk to each other inside the cluster, external users still need a way in. The traditional approach is to run a reverse proxy container, place it where it can reach all nodes, and configure it to forward to each replica. The ingress routing mesh removes that requirement entirely: you publish a port on any node, every node listens on that port, and requests are automatically routed to running containers .
Avoiding port conflicts. If every node bound the published port directly on the host, and some nodes ran multiple replicas, port conflicts would occur. The routing mesh avoids this by not actually binding the port on the host. It uses iptables rules to intercept incoming packets, direct them into the ingress overlay network, and let an internal load balancer distribute them to containers .
Encryption is off by default. Swarm’s management and control-plane traffic is always encrypted, but application data is not encrypted by default. VXLAN packets travel between nodes in plaintext, and anyone capturing network traffic can read inter-container communication . The --opt encrypted option encrypts the VXLAN payload with IPsec, but it carries a non-trivial performance cost and has had a patched security vulnerability (CVE-2023-28842).
a. VXLAN encapsulation and overlay network implementation
The overlay network driver is an implementation of VXLAN. VXLAN stands for Virtual Extensible LAN, and the problem it solves is: how do you carry Layer 2 Ethernet frames across a Layer 3 IP network? The key to understanding this is the relationship between Layer 2 and Layer 3. Ethernet frames (Layer 2) carry source and destination MAC addresses, but they can only travel within the same physical network segment. IP packets (Layer 3) carry source and destination IP addresses and can be routed across the entire Internet. VXLAN places Layer 2 frames inside Layer 3 IP packets, letting a Layer 2 network cross Layer 3 boundaries.
VXLAN encapsulation works like this: when a container sends an Ethernet frame, the VXLAN driver wraps that frame in a VXLAN header, which contains a 24-bit VXLAN Network Identifier (VNI) used to distinguish overlay networks . The entire VXLAN packet is then placed inside a UDP datagram, and the UDP datagram is placed inside an IP packet. The source address of this IP packet is the sending node’s physical IP, and the destination address is the receiving node’s physical IP. After the packet crosses the physical network and arrives at the receiving node, the VXLAN driver decapsulates it, extracts the original Ethernet frame, and delivers it to the target container .
The VNI is the key to overlay network isolation. Each overlay network gets a unique VNI. When a node receives a VXLAN packet, it checks the VNI to decide which overlay network the packet belongs to. Different overlay networks have different VNIs, so containers cannot communicate across networks even if they run on the same physical host . This provides network-level isolation that is lower-level and more reliable than firewall rules.
Docker’s overlay driver configures the VXLAN network on each node dynamically and lazily. When a container is attached to the network, the driver creates the VXLAN interface and routing rules on that node. When the container is removed, the configuration is cleaned up . Routing and encryption parameters are defined only for nodes participating in the network; non-participating nodes require no configuration.
The default overlay subnet is /24, providing 256 IP addresses. Docker explicitly recommends against creating overlay networks larger than /24 . If you need more addresses, do not increase the IP block size—use DNS round-robin (DNSRR) mode with an external load balancer, or create several smaller overlay networks . This limitation comes from a known Swarm mode issue (moby/moby#30820) . Another limit is that the number of containers on the same overlay network per host should not exceed 1000; beyond that, Linux kernel limits can cause network instability .
b. The ingress network and the routing mesh
When you run docker swarm init, Swarm automatically creates two networks: ingress and docker_gwbridge . ingress is an overlay network used for the ingress routing mesh. docker_gwbridge is a bridge network that connects each node’s Docker daemon to the Swarm gateway port .
The ingress routing mesh works like this: when you create a service and publish a port, Swarm assigns the service a virtual IP (VIP) and configures that VIP on the ingress overlay network . Swarm then configures iptables rules on every node to listen on the published port. When a request arrives at the published port on any node, the iptables rules direct the traffic into the ingress network, and the ingress network’s internal load balancer (based on IPVS) forwards the request to one of the containers running the service .
There is a key architectural fact here: the node that runs the request-forwarding logic is not necessarily the node that runs the container. If you have a 5-node Swarm with a service that has 3 replicas, all 5 nodes accept connections on the published port. When a request arrives at node 1 (which runs no replicas), node 1’s ingress load balancer forwards the request over the ingress overlay network to node 3 (which runs a replica), and node 3’s container responds . The user never needs to know where the replicas are.
Port publishing has two syntaxes. The short form is -p 8080:80, and the long form is --publish published=8080,target=80 . The long form is clearer and allows more flexibility. target is the port inside the container, and published is the port the routing mesh listens on. If you omit published, Swarm assigns a random high port .
The routing mesh listens on the published port on whatever IP addresses are assigned to the node. For externally routable IPs, the port is reachable from outside the host. For other IP addresses, it is only reachable from inside the host . This means that if your node has multiple network interfaces, the routing mesh listens on all of them.
c. Host mode and bypassing the routing mesh
The routing mesh has an important limitation: it performs source NAT (SNAT), which means the container sees the Swarm node’s IP address as the client IP, not the original client’s IP . For most applications this is fine, but for applications that need to log client IPs, perform IP-based access control, or see the true source address, it is a problem.
Host mode publishing solves this. When you use --publish mode=host,published=80,target=8080, the port is bound directly on the node running the container, bypassing the ingress overlay network . This means only the node running the container listens on that port, there is no internal load balancing, requests go directly to the local container, and the client source IP is preserved .
The cost of host mode is losing the convenience of the routing mesh. You cannot publish the same host port on more replicas than you have nodes, because each host port can only be bound once. If you have 5 nodes, you can run at most 5 replicas, one per node . When using a fixed host port, it is common to pair it with a global service (--mode global) or placement constraints to ensure exactly one replica per node .
Host mode is appropriate for scenarios such as: applications that need to preserve client IPs, external load balancers that need to health-check individual nodes, and latency-sensitive applications that want to avoid the overlay network hop . It is also appropriate when you want to use your own reverse proxy (such as Traefik or Nginx) for TLS termination and Layer 7 routing, because the routing mesh only does Layer 4 port forwarding and does not support hostname or path routing .
Complete Example Session
The following session demonstrates creating a custom overlay network, deploying a service that uses it, and comparing the port-publishing behavior of ingress mode and host mode.
# ============================================
# PART 1: VIEW DEFAULT NETWORKS
# ============================================
docker network ls
# Output includes:
# ingress overlay swarm
# docker_gwbridge bridge local
# bridge bridge local
# host host local
# none null local
# ============================================
# PART 2: CREATE A CUSTOM OVERLAY NETWORK
# ============================================
docker network create \
--driver overlay \
--attachable \
kodekloudnet
# --attachable allows manually started containers to join the network
# By default only Swarm services can attach
# ============================================
# PART 3: INSPECT THE NETWORK CONFIGURATION
# ============================================
docker network inspect kodekloudnet
# Output shows:
# "Scope": "swarm"
# "Driver": "overlay"
# IPAM subnet (e.g., 10.0.2.0/24)
# Gateway address
# ============================================
# PART 4: DEPLOY AN INGRESS-MODE SERVICE
# ============================================
docker service create \
--name ingressservice \
--publish published=80,target=80 \
--replicas=2 \
--network=kodekloudnet \
nginx:latest
# Accessing port 80 on any node reaches the container
# Even on nodes that run no replicas
# ============================================
# PART 5: DEPLOY A HOST-MODE SERVICE
# ============================================
docker service create \
--name hostservice \
--publish mode=host,published=80,target=80 \
--replicas=2 \
--network=kodekloudnet \
nginx:latest
# Only the nodes running containers respond on port 80
# Other nodes do not respond
# ============================================
# PART 6: VERIFY SERVICE DISCOVERY
# ============================================
# Enter a container attached to kodekloudnet
docker run -it --rm \
--network=kodekloudnet \
busybox sh
# Inside the container, query the service name
nslookup ingressservice
# Returns the service's VIP address
# ============================================
# PART 7: TEST CROSS-SERVICE COMMUNICATION
# ============================================
# ingressservice is reachable by service name
wget -qO- http://ingressservice
# ============================================
# PART 8: CLEAN UP RESOURCES
# ============================================
docker service rm ingressservice hostservice
docker network rm kodekloudnetThe eight parts cover viewing default networks, creating a custom overlay network, inspecting network configuration, deploying in ingress mode, deploying in host mode, verifying service discovery, testing cross-service communication, and cleanup. The session demonstrates the complete workflow for overlay networks and service publishing.
Quick Reference
Overlay Network vs Routing Mesh
| Feature | Ingress Routing Mesh | Host Mode |
|---|---|---|
| Listening nodes | All nodes | Only nodes running containers |
| Internal load balancing | Yes (IPVS) | No |
| Client source IP | Hidden by SNAT | Preserved |
| Port conflicts | None | Possible |
| Replica limit | None | Limited by node count |
| Use case | General publishing | Source IP preservation, low latency |
Overlay Network Commands
| Operation | Command |
|---|---|
| Create | docker network create --driver overlay mynet |
| Attachable | docker network create --driver overlay --attachable mynet |
| Encrypted | docker network create --opt encrypted --driver overlay mynet |
| Specify subnet | docker network create --subnet 10.10.0.0/24 --driver overlay mynet |
| Inspect | docker network inspect mynet |
| Remove | docker network rm mynet |
Port Publishing Syntax
| Syntax | Example | Meaning |
|---|---|---|
| Short | -p 8080:80 | Host 8080 → container 80 |
| Long | --publish published=8080,target=80 | Same, clearer |
| Host mode | --publish mode=host,published=80,target=80 | Direct bind, no mesh |
Required Swarm Ports
| Port | Protocol | Purpose |
|---|---|---|
| 2377 | TCP | Cluster management communication |
| 7946 | TCP/UDP | Container network discovery |
| 4789 | UDP | VXLAN overlay network traffic |
Best Practices
✅ Do This:
docker network create --driver overlay --attachable mynet # ✅ attachable for debugging
docker service create --publish published=80,target=80 app # ✅ ingress mode general
docker service create --publish mode=host,published=80,target=80 app # ✅ preserve source IP
docker network create --opt encrypted --driver overlay secure # ✅ encrypt on untrusted networks❌ Don’t Do This:
docker network create --driver overlay --subnet 10.0.0.0/16 big # ❌ exceeds /24 limit
docker service create --publish mode=host --replicas 10 app # ❌ replicas exceed nodes
# ignore port 4789 UDP # ❌ overlay won't work
# use encrypted overlay on Windows containers # ❌ not supportedCommon Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Overlay network cannot cross hosts | Firewall blocks UDP 4789 | Open the port |
| Routing mesh not responding | Ingress network deleted or corrupted | Recreate the ingress network |
| Host mode port conflict | Multiple replicas on one node | Use global service or placement constraints |
| Encrypted network slower | IPsec overhead | Benchmark, enable only for sensitive data |
| Unstable beyond 1000 containers | Linux kernel limits | Split across multiple networks |
| Service not discoverable | Not on the same network | Attach to the same overlay network |
Real-World Examples
1. Create an encrypted overlay network
docker network create --driver overlay --opt encrypted secure-net2. Attach a service to an overlay network
docker service create --network mynet --name api api:latest3. Ingress-mode port publishing
docker service create --publish published=80,target=8080 --name web web:latest4. Host mode preserves source IP
docker service create --publish mode=host,published=80,target=8080 web5. Inspect overlay subnet
docker network inspect mynet --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}'6. View a service’s VIP
docker service inspect mynet --format '{{.Endpoint.VirtualIPs}}'7. Global service with host mode
docker service create --mode global --publish mode=host,published=80,target=80 agent8. Multi-subnet overlay network
docker network create -d overlay --subnet=10.0.1.0/24 --subnet=10.0.2.0/24 multi9. Clean up unused networks
docker network prune10. Inspect the ingress network
docker network inspect ingressVisual
Overlay Network and VXLAN Encapsulation
┌──────────────────────────────────────────────┐
│ NODE 1 (Physical IP: 192.168.1.10) │
│ ┌────────────────────────────────────────┐ │
│ │ Container A (overlay IP: 10.0.1.5) │ │
│ │ sends an Ethernet frame │ │
│ └────────────────┬───────────────────────┘ │
│ │ │
│ ┌─────────▼─────────┐ │
│ │ VXLAN encapsulation│ │
│ │ + VNI identifies net│ │
│ │ + UDP header │ │
│ │ + IP header (dst node2)│ │
│ └─────────┬─────────┘ │
│ │ │
└───────────────────┼──────────────────────────┘
│
Physical network (UDP 4789)
│
┌───────────────────┼──────────────────────────┐
│ ▼ │
│ ┌─────────────────┐ │
│ │ VXLAN decapsulation│ │
│ │ extract original frame│ │
│ └─────────┬───────┘ │
│ │ │
│ ┌────────────────▼───────────────────────┐ │
│ │ Container B (overlay IP: 10.0.1.6) │ │
│ │ receives the original Ethernet frame │ │
│ └────────────────────────────────────────┘ │
│ NODE 2 (Physical IP: 192.168.1.11) │
└──────────────────────────────────────────────┘Ingress Routing Mesh
┌──────────────────────────────────────────────┐
│ EXTERNAL CLIENT │
│ Request to any node:80 │
└───────────────┬──────────────────────────────┘
│
┌───────┴───────┬───────────────┐
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ NODE 1 │ │ NODE 2 │ │ NODE 3 │
│ no replicas │ │ runs replica │ │ no replicas │
│ │ │ │ │ │
│ iptables │ │ iptables │ │ iptables │
│ listens :80 │ │ listens :80 │ │ listens :80 │
│ │ │ │ │ │ │ │ │
│ ▼ │ │ ▼ │ │ ▼ │
│ ingress net │ │ ingress net │ │ ingress net │
│ │ │ │ │ │ │ │ │
│ └───────┼─┼──────┘ │ │ │ │
│ │ │ │ │ │ │
│ │ ▼ │ │ │ │
│ │ container │ │ │ │
│ │ handles request│ │ │ │
│ │ │ │ │ │
│ └──────────────┘ └──────┘ │
│ All nodes accept requests; routed to the node│
│ running the container │
└──────────────────────────────────────────────┘Ingress Mode vs Host Mode
┌──────────────────────────────────────────────┐
│ INGRESS MODE (default) │
│ ┌─────────┐ │
│ │ NODE 1 │◄── request arrives at any node │
│ │ :80 │ │
│ └────┬────┘ │
│ │ routing mesh forwards │
│ ▼ │
│ ┌─────────┐ │
│ │ NODE 2 │ │
│ │ container│ │
│ └─────────┘ │
│ Client IP hidden by SNAT │
│ │
│ HOST MODE │
│ ┌─────────┐ │
│ │ NODE 1 │◄── request arrives at the port │
│ │ container│ reaches the local container │
│ │ :80 │ directly │
│ └─────────┘ │
│ Client IP preserved │
│ Only nodes running containers respond │
└──────────────────────────────────────────────┘Swarm Default Networks
┌──────────────────────────────────────────────┐
│ ingress (overlay, swarm scope) │
│ - Foundation of the ingress routing mesh │
│ - Auto-created, cannot be deleted while in │
│ use │
│ - Used for service port publishing │
│ │
│ docker_gwbridge (bridge, local scope) │
│ - Connects the node to the Swarm gateway │
│ - Provides node-level external connectivity │
│ - Auto-created on each node │
│ │
│ Custom overlay networks │
│ - Inter-service communication │
│ - Not encrypted by default │
│ - Enable IPsec with --opt encrypted │
└──────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Overlay driver | overlay, based on VXLAN |
| VXLAN encapsulation | Ethernet frame → UDP datagram |
| VNI | 24-bit ID identifying the overlay network |
| Default subnet | /24 (256 addresses) |
| Ingress network | Auto-created overlay network |
| Routing mesh | All nodes listen on published ports |
| Internal load balancing | IPVS, based on VIP |
| Host mode | Bypasses routing mesh, preserves source IP |
| Required ports | 2377 TCP, 7946 TCP/UDP, 4789 UDP |
| Encryption | --opt encrypted, IPsec, performance cost |
Key takeaways:
- Overlay networks use VXLAN encapsulation for cross-host communication. Ethernet frames are placed inside UDP datagrams, traverse the physical network, and are decapsulated on the target node .
- The VNI provides network isolation. Each overlay network has a unique 24-bit identifier, and containers on different networks cannot reach each other .
- The ingress routing mesh involves every node. Every node listens on published ports and routes requests to whichever node runs a container .
- The routing mesh uses SNAT and hides client source IPs. Use host mode publishing when source IP preservation is required .
- Host mode bypasses the mesh. The port binds directly on the node running the container, there is no internal load balancing, and the replica count is limited by the node count .
- Overlay networks are not encrypted by default. Application data travels in plaintext VXLAN;
--opt encryptedenables IPsec at a performance cost . /24is the recommended overlay subnet size. When you need more addresses, use DNSRR mode or multiple networks—do not increase the IP block size .- Service discovery uses built-in DNS. A service name resolves to a VIP, and IPVS distributes connections to healthy replicas .
Remember: The heart of Swarm overlay networking is VXLAN encapsulation—it extends a container’s Layer 2 network across the entire cluster, making cross-host communication as simple as local communication. The ingress routing mesh solves external access: you publish a port on any node, every node responds, and requests are automatically forwarded to the node running the container. Understanding these two mechanisms is understanding all of Swarm networking. When you need to preserve client source IPs, use host mode. When you run sensitive workloads on untrusted networks, enable encryption—but benchmark the performance impact. When you need more than 256 addresses, do not enlarge the subnet; split the network or use DNSRR.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!