Berik Ashimov
Senior IT Engineer

VXLAN: Scalable L2 Over L3 Overlay

· 7 min read
#vyos #networking #overlay

VLANs scale to 4094. That’s not enough for large datacenters with thousands of tenants. VLAN tags are local to Layer 2 domains. Extending VLANs across L3 boundaries requires complex tricks.

VXLAN (Virtual Extensible LAN) encapsulates Ethernet frames in UDP. 24-bit VNI supports 16 million segments. Runs over any IP network. Decouples overlay from underlay.

VXLAN enables scalable Layer 2 networks over any IP infrastructure.

VXLAN Concepts

How VXLAN Works

[Host A]──[VTEP1]═══ IP Network ═══[VTEP2]──[Host B]
           │                          │
    Encapsulate in UDP         Decapsulate
    (add VXLAN header)         (remove header)

VXLAN Header

Outer Ethernet │ Outer IP │ Outer UDP │ VXLAN │ Inner Ethernet │ Inner IP │ Payload
               │          │  dst 4789 │ VNI   │                │          │

Key Terms

TermDescription
VNIVXLAN Network Identifier (24-bit, up to 16M)
VTEPVXLAN Tunnel Endpoint
NVENetwork Virtualization Edge
BUMBroadcast, Unknown unicast, Multicast

Basic VXLAN Configuration

Static VXLAN (Point-to-Point)

Terminal window
configure


# Create VXLAN interface
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 10.0.0.1
set interfaces vxlan vxlan100 remote 10.0.0.2
set interfaces vxlan vxlan100 port 4789


# Bridge VXLAN with local interface
set interfaces bridge br100 member interface vxlan100
set interfaces bridge br100 member interface eth1


commit

Remote Side

Terminal window
configure


# Mirror configuration, swap source/remote
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 10.0.0.2
set interfaces vxlan vxlan100 remote 10.0.0.1
set interfaces vxlan vxlan100 port 4789


set interfaces bridge br100 member interface vxlan100
set interfaces bridge br100 member interface eth1


commit

Multicast VXLAN

For multi-point VXLAN using multicast for BUM traffic:

Terminal window
configure


# VXLAN with multicast group
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 10.0.0.1
set interfaces vxlan vxlan100 group 239.1.1.100
set interfaces vxlan vxlan100 port 4789


# Bridge configuration
set interfaces bridge br100 member interface vxlan100
set interfaces bridge br100 member interface eth1


commit

Multicast Requirements

Terminal window
# Underlay must support multicast routing
# Enable PIM on underlay interfaces
set protocols pim interface eth0


# Or use static IGMP membership

Head-End Replication (Unicast Mode)

No multicast required — VTEP replicates BUM to all remote VTEPs:

Terminal window
configure


# VXLAN with multiple remote VTEPs
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 10.0.0.1
set interfaces vxlan vxlan100 remote 10.0.0.2
set interfaces vxlan vxlan100 remote 10.0.0.3
set interfaces vxlan vxlan100 remote 10.0.0.4
set interfaces vxlan vxlan100 port 4789


# BUM traffic is replicated to all remotes


commit

Scaling Consideration

Multicast: Efficient BUM delivery, requires multicast underlay
Unicast:   Simple, but BUM traffic multiplied by VTEP count


Small scale (few VTEPs): Unicast fine
Large scale: Multicast or EVPN control plane

VXLAN with EVPN

Best practice for production: EVPN control plane handles:

  • MAC learning (no data plane flooding)
  • Remote VTEP discovery (no manual configuration)
  • BUM optimization
Terminal window
configure


# VXLAN interface
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan100 source-address 10.0.0.1
set interfaces vxlan vxlan100 parameters nolearning


# nolearning: Disable data plane MAC learning (EVPN handles it)


# BGP EVPN configuration
set protocols bgp address-family l2vpn-evpn advertise-all-vni


commit

MTU Considerations

VXLAN Overhead

Outer Ethernet:  14 bytes
Outer IP:        20 bytes
Outer UDP:        8 bytes
VXLAN header:     8 bytes
Total:           50 bytes


Standard 1500 MTU - 50 = 1450 inner MTU

Configure MTU

Terminal window
configure


# Option 1: Increase underlay MTU
set interfaces ethernet eth0 mtu 1550


# Option 2: Reduce overlay MTU
set interfaces bridge br100 mtu 1450


commit

Jumbo Frames

Terminal window
# Better option: Use jumbo frames on underlay
set interfaces ethernet eth0 mtu 9000


# VXLAN inner MTU: 9000 - 50 = 8950
# Standard 1500 MTU traffic fits easily

VXLAN Gateway

L2 Gateway (Bridging Only)

Terminal window
# VXLAN bridges to local VLAN
set interfaces bridge br100 member interface vxlan100
set interfaces bridge br100 member interface eth1.100


# Local VLAN 100 traffic bridged to VXLAN 100

L3 Gateway (Routing)

Terminal window
configure


# Add IP to bridge for routing
set interfaces bridge br100 address 192.168.100.1/24


# VMs/hosts in VXLAN use this as gateway
# Router handles inter-VXLAN routing


commit

VXLAN Routing Between VNIs

Terminal window
configure


# Two VXLANs
set interfaces vxlan vxlan100 vni 100
set interfaces vxlan vxlan200 vni 200


# Two bridges with IPs
set interfaces bridge br100 member interface vxlan100
set interfaces bridge br100 address 192.168.100.1/24


set interfaces bridge br200 member interface vxlan200
set interfaces bridge br200 address 192.168.200.1/24


# Router routes between 192.168.100.0/24 and 192.168.200.0/24


commit

Viewing VXLAN State

Check Interface

Terminal window
# Show VXLAN interface
show interfaces vxlan


# Show VXLAN details
show interfaces vxlan vxlan100


# Show bridge MAC table
show bridge fdb interface br100

Check Forwarding Database

Terminal window
# View learned MACs
bridge fdb show dev vxlan100


# Output:
# aa:bb:cc:dd:ee:ff dev vxlan100 dst 10.0.0.2 self permanent
# 11:22:33:44:55:66 dev vxlan100 master br100

Troubleshooting VXLAN

Tunnel Not Working

Terminal window
# Check VXLAN interface is up
show interfaces vxlan vxlan100


# Verify underlay connectivity
ping 10.0.0.2  # Remote VTEP


# Check UDP port 4789 is not filtered
nc -vzu 10.0.0.2 4789


# Capture VXLAN traffic
sudo tcpdump -i eth0 udp port 4789

No MAC Learning

Terminal window
# Check bridge FDB
bridge fdb show dev vxlan100


# If empty, check:
# - ARP traffic flowing
# - VXLAN interface in bridge
# - nolearning not set (unless using EVPN)


# Generate traffic to trigger learning
arping -I br100 192.168.100.100

MTU Issues

Terminal window
# Symptoms: Small packets work, large fail


# Test with large ping
ping -s 1400 192.168.100.100


# If fails, check MTU
ip link show vxlan100
ip link show br100


# Verify underlay MTU is sufficient
ping -s 1500 -M do 10.0.0.2

Security Considerations

VXLAN Has No Encryption

Traffic visible to underlay network:
- Inner Ethernet frames
- All payload content


For sensitive data:
- Encrypt at application layer
- Use IPsec on underlay
- Consider alternative (WireGuard overlay)

Firewall VXLAN Traffic

Terminal window
# Allow VXLAN only from known VTEPs
set firewall ipv4 name UNDERLAY-IN rule 100 action accept
set firewall ipv4 name UNDERLAY-IN rule 100 protocol udp
set firewall ipv4 name UNDERLAY-IN rule 100 destination port 4789
set firewall ipv4 name UNDERLAY-IN rule 100 source group network-group VTEPS

VXLAN Design Patterns

Datacenter Fabric

         [Spine1]     [Spine2]
            │            │
    ────────┼────────────┼────────
    │       │       │    │       │
 [Leaf1] [Leaf2] [Leaf3] ...


Underlay: IP routing (OSPF/BGP) on spine-leaf
Overlay:  VXLAN between leaves
Control:  EVPN for MAC learning

DCI (Datacenter Interconnect)

[DC1]══════VXLAN══════[DC2]
       over WAN


Extended L2 between datacenters
Watch out for: Latency, BUM flooding, split-brain

The Lesson

VXLAN enables scalable Layer 2 networks over any IP infrastructure.

VXLAN advantages:

  • 16 million segments (vs 4K VLANs)
  • Works over any IP network
  • Decouples overlay from underlay
  • Foundation for modern DC fabrics

VXLAN considerations:

  • 50-byte overhead (needs MTU planning)
  • BUM handling (multicast, unicast, or EVPN)
  • No encryption (plan accordingly)
  • Control plane important at scale

For small deployments: Static VXLAN with head-end replication works. For scale: EVPN control plane is the answer.

VXLAN is infrastructure. The real magic is in the control plane (EVPN) and how you design the overlay-underlay interaction.