OSPF on VyOS: When Details Break Everything

OSPF is deceptively simple to configure. Two routers, same area, same subnet — they should just work. And then they don’t. The adjacency sticks at EXSTART, or neighbors appear and disappear, or routes mysteriously vanish.

The problem is always in the details. OSPF has strict requirements that must match between neighbors: MTU, hello/dead timers, area type, authentication. Miss one, and the adjacency fails — often silently.

OSPF Fundamentals

OSPF (Open Shortest Path First) is a link-state protocol. Each router maintains a complete map of the network topology and calculates shortest paths independently.

Key concepts:

• Area: Logical grouping of routers. Area 0 is the backbone — all other areas must connect to it
• Router ID: Unique identifier, usually an IP address
• Adjacency: Full neighbor relationship where routers exchange LSAs
• LSA: Link State Advertisement — the building blocks of the topology database

Basic OSPF Configuration

configure

# Set router ID (use a loopback IP if you have one)
set protocols ospf parameters router-id '10.255.0.1'

# Enable OSPF on interfaces
set protocols ospf area 0 network '10.0.0.0/24'
set protocols ospf area 0 network '10.0.1.0/24'
set protocols ospf area 0 network '10.255.0.1/32'

commit

This enables OSPF on all interfaces matching those networks in area 0.

Interface-Based Configuration

More explicit approach — configure OSPF per interface:

configure

set protocols ospf parameters router-id '10.255.0.1'

# Enable on specific interfaces
set protocols ospf interface eth0 area '0'
set protocols ospf interface eth1 area '0'
set protocols ospf interface lo area '0'

commit

Interface-based is clearer and preferred for complex setups.

Passive Interfaces: The Silent Killer

Passive interfaces don’t send or receive OSPF hello packets. Use them on:

• LAN segments with no OSPF neighbors
• Internet-facing interfaces
• Management networks

# Mark interface as passive
set protocols ospf passive-interface 'eth2'
set protocols ospf passive-interface 'default'  # All interfaces passive by default

# Then explicitly enable OSPF interfaces
set protocols ospf passive-interface-exclude 'eth0'
set protocols ospf passive-interface-exclude 'eth1'

The trap: Forgetting to exclude an interface means no neighbors form. OSPF just sits there, advertising the network but never receiving hellos. No errors, no warnings — just silence.

Debugging Passive Issues

show ip ospf neighbor

# Empty? Check if interface is passive
show ip ospf interface eth0
# Look for "Passive interface" in output

MTU Mismatch: The Classic OSPF Failure

OSPF includes MTU in Database Description packets. If MTU doesn’t match between neighbors, adjacency sticks at EXSTART/EXCHANGE state.

# Check current MTU
show interfaces ethernet eth0

# Symptoms of MTU mismatch
show ip ospf neighbor
# Neighbor stuck in EXSTART or EXCHANGE state

Fixing MTU Issues

Option 1: Match MTU on both sides (preferred)

set interfaces ethernet eth0 mtu '1500'

Option 2: Ignore MTU check (workaround)

set protocols ospf interface eth0 mtu-ignore

Use mtu-ignore only when you can’t control the other side’s MTU. It hides the problem rather than fixing it.

Common MTU Scenarios

Tunnel interfaces are the usual suspects. Always check MTU when OSPF over tunnels fails.

Hello and Dead Timers

OSPF sends hello packets at regular intervals. Miss too many, and the neighbor is declared dead.

• Hello interval: How often to send hellos (default: 10 seconds)
• Dead interval: How long to wait before declaring neighbor dead (default: 40 seconds)

These must match between neighbors.

# Check current timers
show ip ospf interface eth0

# Modify timers (both sides must match)
set protocols ospf interface eth0 hello-interval '10'
set protocols ospf interface eth0 dead-interval '40'

Fast Failure Detection

For faster convergence, reduce timers:

# Aggressive timers (1 second hello, 4 second dead)
set protocols ospf interface eth0 hello-interval '1'
set protocols ospf interface eth0 dead-interval '4'

Trade-off: Faster detection but more CPU and more sensitive to packet loss. A single dropped hello could trigger failover.

BFD for Sub-Second Failover

For true fast failover, use BFD (Bidirectional Forwarding Detection) instead of aggressive OSPF timers:

# Enable BFD on interface
set protocols ospf interface eth0 bfd

# Configure BFD parameters
set protocols bfd peer 10.0.0.2 source address '10.0.0.1'
set protocols bfd peer 10.0.0.2 interval transmit '300'
set protocols bfd peer 10.0.0.2 interval receive '300'
set protocols bfd peer 10.0.0.2 interval multiplier '3'

BFD provides ~1 second detection without the overhead of fast OSPF hellos.

OSPF Areas

Large OSPF networks need multiple areas to:

• Reduce SPF calculations (changes in one area don’t affect others)
• Limit LSA flooding
• Summarize routes at area boundaries

Multi-Area Setup

configure

# Backbone area (always area 0)
set protocols ospf interface eth0 area '0'

# Other areas connect through ABR (Area Border Router)
set protocols ospf interface eth1 area '1'
set protocols ospf interface eth2 area '2'

commit

The router with interfaces in multiple areas is an ABR (Area Border Router).

Stub Areas

Stub areas don’t receive external routes (Type 5 LSAs). Useful for areas that only need a default route to the rest of the network:

# Configure area as stub
set protocols ospf area 1 area-type stub

# On ABR, optionally set default route cost
set protocols ospf area 1 area-type stub default-cost '10'

All routers in the area must agree on stub configuration.

Totally Stubby Areas

Block both external routes AND inter-area routes:

# On ABR only
set protocols ospf area 1 area-type stub no-summary

Routers in the area only see a default route. Simplest routing table, least flexibility.

NSSA (Not-So-Stubby Area)

Like stub, but allows local external routes:

set protocols ospf area 1 area-type nssa

Useful when the area has an ASBR (redistributing from another protocol) but you don’t want external routes from other areas.

OSPF Authentication

MD5 Authentication (Recommended)

configure

# Set authentication for interface
set protocols ospf interface eth0 authentication md5 key-id 1 md5-key 'YourSecretKey123'

commit

Both neighbors must have identical key-id and key.

Rotating Keys

OSPF supports multiple keys for hitless rotation:

# Add new key
set protocols ospf interface eth0 authentication md5 key-id 2 md5-key 'NewSecretKey456'

# Both keys active — neighbors using either key will authenticate
# After all neighbors updated, remove old key
delete protocols ospf interface eth0 authentication md5 key-id 1

Plain Text Authentication (Don’t Use)

# Exists but insecure — anyone can sniff the password
set protocols ospf interface eth0 authentication plaintext-password 'visible-password'

Use MD5 or no authentication. Plain text is false security.

Network Types

OSPF behavior changes based on network type:

Point-to-Point Links

For direct router-to-router links, use point-to-point:

set protocols ospf interface eth0 network 'point-to-point'

Benefits:

• No DR/BDR election delay
• Faster adjacency formation
• Works over unnumbered interfaces

Use for: GRE tunnels, VTI interfaces, WireGuard tunnels, direct fiber links.

Route Redistribution

Import routes from other sources into OSPF:

configure

# Redistribute connected routes
set protocols ospf redistribute connected

# Redistribute static routes
set protocols ospf redistribute static

# Redistribute with metric
set protocols ospf redistribute connected metric '100'
set protocols ospf redistribute connected metric-type '2'

commit

Metric types:

• Type 1 (E1): External metric added to internal path cost
• Type 2 (E2): External metric only, internal cost ignored (default)

Filtering Redistributed Routes

Use route-maps to control what gets redistributed:

# Define prefix list
set policy prefix-list OSPF-EXPORT rule 10 action 'permit'
set policy prefix-list OSPF-EXPORT rule 10 prefix '10.10.0.0/16'
set policy prefix-list OSPF-EXPORT rule 10 le '24'

# Define route-map
set policy route-map OSPF-REDISTRIBUTE rule 10 action 'permit'
set policy route-map OSPF-REDISTRIBUTE rule 10 match ip address prefix-list 'OSPF-EXPORT'
set policy route-map OSPF-REDISTRIBUTE rule 10 set metric '50'

# Apply to redistribution
set protocols ospf redistribute connected route-map 'OSPF-REDISTRIBUTE'

Troubleshooting OSPF

Check Neighbor State

show ip ospf neighbor

# Expected: FULL state for all neighbors
# Problem states:
# - INIT: Receiving hellos, but they don't see us
# - 2-WAY: Seen each other, waiting for DR election (normal on broadcast)
# - EXSTART/EXCHANGE: Database sync starting (often MTU mismatch)
# - LOADING: Receiving LSAs

Check Interface Configuration

show ip ospf interface eth0

# Verify:
# - Correct area
# - Hello/Dead intervals match
# - Not passive when shouldn't be
# - Network type appropriate

Check OSPF Database

# Show all LSAs
show ip ospf database

# Show specific LSA type
show ip ospf database router
show ip ospf database network
show ip ospf database external

Check Routes

# OSPF routes
show ip route ospf

# Why isn't a route showing?
# 1. LSA not received (neighbor issue)
# 2. Better route exists
# 3. Filtering applied

Common Problems and Solutions

Complete OSPF Configuration

# === OSPF Core ===
set protocols ospf parameters router-id '10.255.0.1'
set protocols ospf log-adjacency-changes

# === Interfaces ===
set protocols ospf interface eth0 area '0'
set protocols ospf interface eth0 network 'point-to-point'
set protocols ospf interface eth0 authentication md5 key-id 1 md5-key 'SecureKey123'
set protocols ospf interface eth0 bfd

set protocols ospf interface eth1 area '0'
set protocols ospf interface eth1 network 'broadcast'
set protocols ospf interface eth1 priority '100'

# === Passive Interfaces ===
set protocols ospf passive-interface 'eth2'
set protocols ospf passive-interface 'lo'

# === Area Configuration ===
set protocols ospf area 1 area-type stub

# === Redistribution ===
set protocols ospf redistribute connected metric '100'
set protocols ospf redistribute connected route-map 'OSPF-EXPORT'

The Lesson

OSPF fails on details:

1. MTU: Must match. When adjacency sticks at EXSTART, check MTU first.

2. Timers: Hello and dead intervals must be identical. Mismatched timers = no adjacency.

3. Passive interfaces: A passive interface that should be active produces no errors — just silence.

4. Authentication: Both sides need identical keys and key-ids.

5. Network type: Point-to-point for tunnels and direct links. Broadcast for Ethernet LANs.

The pattern: OSPF is strict about requirements but quiet about failures. When something doesn’t work, methodically check each parameter. The problem is always a mismatch somewhere.

Debug OSPF by elimination: Can you ping the neighbor? Is the interface passive? Does MTU match? Do timers match? Is authentication correct? Work through the list, and you’ll find it.