IPsec on VyOS: Site-to-Site Tunnels That Survive Reality

October 17, 2025 · 9 min read

IPsec has a reputation for being complex and fragile. There’s some truth to that — it has more moving parts than WireGuard, more states to manage, more things that can go wrong. But IPsec is also universal. It works with nearly any vendor’s equipment and is often required for corporate connectivity.

The key to reliable IPsec: understand the timers and states. When an IPsec tunnel fails, it’s almost always timer mismatch or phase state issues. This guide covers practical IPsec configuration and the debugging skills to diagnose problems.

IPsec Fundamentals

IPsec has two phases:

IKE Phase 1 (IKE SA): Negotiate encryption, authenticate peers, establish secure channel for Phase 2 negotiations.

IKE Phase 2 (IPsec SA / Child SA): Negotiate the actual tunnel parameters, establish encryption for user traffic.

Both phases have lifetimes. When they expire, rekeying occurs. Mismatched timers between peers cause tunnels to drop.

Site-to-Site Configuration: IKEv2

IKEv2 is preferred over IKEv1 — better NAT traversal, faster failover, simpler configuration. Use IKEv1 only when the peer doesn’t support IKEv2.

Site A Configuration

configure

# IKE group (Phase 1 parameters)
set vpn ipsec ike-group IKE-SITE-B close-action 'none'
set vpn ipsec ike-group IKE-SITE-B dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-SITE-B dead-peer-detection interval '30'
set vpn ipsec ike-group IKE-SITE-B dead-peer-detection timeout '120'
set vpn ipsec ike-group IKE-SITE-B key-exchange 'ikev2'
set vpn ipsec ike-group IKE-SITE-B lifetime '28800'
set vpn ipsec ike-group IKE-SITE-B proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-SITE-B proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-SITE-B proposal 1 hash 'sha256'

# ESP group (Phase 2 parameters)
set vpn ipsec esp-group ESP-SITE-B lifetime '3600'
set vpn ipsec esp-group ESP-SITE-B pfs 'dh-group14'
set vpn ipsec esp-group ESP-SITE-B proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-SITE-B proposal 1 hash 'sha256'

# Interface binding
set vpn ipsec interface 'eth0'

# Site-to-Site connection
set vpn ipsec site-to-site peer SITE-B authentication local-id 'site-a@example.com'
set vpn ipsec site-to-site peer SITE-B authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer SITE-B authentication pre-shared-secret 'YourVeryStrongPSKHere123!'
set vpn ipsec site-to-site peer SITE-B authentication remote-id 'site-b@example.com'
set vpn ipsec site-to-site peer SITE-B connection-type 'initiate'
set vpn ipsec site-to-site peer SITE-B default-esp-group 'ESP-SITE-B'
set vpn ipsec site-to-site peer SITE-B ike-group 'IKE-SITE-B'
set vpn ipsec site-to-site peer SITE-B local-address '203.0.113.1'
set vpn ipsec site-to-site peer SITE-B remote-address '198.51.100.1'

# Traffic selectors (what traffic goes through the tunnel)
set vpn ipsec site-to-site peer SITE-B tunnel 1 local prefix '10.1.0.0/24'
set vpn ipsec site-to-site peer SITE-B tunnel 1 remote prefix '10.2.0.0/24'

commit

Site B Configuration

Mirror configuration with swapped addresses and IDs:

configure

# IKE group - MUST MATCH Site A
set vpn ipsec ike-group IKE-SITE-A close-action 'none'
set vpn ipsec ike-group IKE-SITE-A dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-SITE-A dead-peer-detection interval '30'
set vpn ipsec ike-group IKE-SITE-A dead-peer-detection timeout '120'
set vpn ipsec ike-group IKE-SITE-A key-exchange 'ikev2'
set vpn ipsec ike-group IKE-SITE-A lifetime '28800'
set vpn ipsec ike-group IKE-SITE-A proposal 1 dh-group '14'
set vpn ipsec ike-group IKE-SITE-A proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-SITE-A proposal 1 hash 'sha256'

# ESP group - MUST MATCH Site A
set vpn ipsec esp-group ESP-SITE-A lifetime '3600'
set vpn ipsec esp-group ESP-SITE-A pfs 'dh-group14'
set vpn ipsec esp-group ESP-SITE-A proposal 1 encryption 'aes256'
set vpn ipsec esp-group ESP-SITE-A proposal 1 hash 'sha256'

set vpn ipsec interface 'eth0'

set vpn ipsec site-to-site peer SITE-A authentication local-id 'site-b@example.com'
set vpn ipsec site-to-site peer SITE-A authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer SITE-A authentication pre-shared-secret 'YourVeryStrongPSKHere123!'
set vpn ipsec site-to-site peer SITE-A authentication remote-id 'site-a@example.com'
set vpn ipsec site-to-site peer SITE-A connection-type 'initiate'
set vpn ipsec site-to-site peer SITE-A default-esp-group 'ESP-SITE-A'
set vpn ipsec site-to-site peer SITE-A ike-group 'IKE-SITE-A'
set vpn ipsec site-to-site peer SITE-A local-address '198.51.100.1'
set vpn ipsec site-to-site peer SITE-A remote-address '203.0.113.1'

set vpn ipsec site-to-site peer SITE-A tunnel 1 local prefix '10.2.0.0/24'
set vpn ipsec site-to-site peer SITE-A tunnel 1 remote prefix '10.1.0.0/24'

commit

Critical: Parameter Matching

Both peers MUST have identical:

Key exchange version (ikev2)
IKE lifetime
ESP lifetime
DH group
Encryption algorithm
Hash algorithm
PFS settings
Traffic selectors (swapped local/remote)

Mismatch in any of these = tunnel won’t establish or will randomly fail.

NAT Traversal (NAT-T)

When either peer is behind NAT, IPsec encapsulates packets in UDP 4500 instead of raw ESP (protocol 50). VyOS enables NAT-T automatically, but you may need firewall rules:

set firewall ipv4 name WAN-LOCAL rule 70 action 'accept'
set firewall ipv4 name WAN-LOCAL rule 70 protocol 'udp'
set firewall ipv4 name WAN-LOCAL rule 70 destination port '500'

set firewall ipv4 name WAN-LOCAL rule 71 action 'accept'
set firewall ipv4 name WAN-LOCAL rule 71 protocol 'udp'
set firewall ipv4 name WAN-LOCAL rule 71 destination port '4500'

# Also allow ESP protocol for non-NAT scenarios
set firewall ipv4 name WAN-LOCAL rule 72 action 'accept'
set firewall ipv4 name WAN-LOCAL rule 72 protocol 'esp'

Dead Peer Detection (DPD)

DPD detects when the remote peer becomes unreachable. Without it, your router won’t know the tunnel is dead until traffic fails.

set vpn ipsec ike-group IKE-SITE-B dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-SITE-B dead-peer-detection interval '30'
set vpn ipsec ike-group IKE-SITE-B dead-peer-detection timeout '120'

interval: Send DPD request every 30 seconds
timeout: Declare peer dead after 120 seconds without response
action: restart = try to re-establish, clear = delete SA, none = do nothing

For stable connections, restart is usually best. It automatically recovers from transient outages.

Rekeying: The Lifetime Dance

IKE and ESP SAs have separate lifetimes. When they expire, rekeying occurs. Problems happen when:

Both peers try to rekey simultaneously
Timers differ slightly, causing race conditions
Rekey fails and tunnel drops

Best practices:

Make lifetimes identical on both peers
IKE lifetime should be longer than ESP lifetime
Common values: IKE 28800s (8h), ESP 3600s (1h)

# Site A and Site B MUST match
set vpn ipsec ike-group IKE-SITE-B lifetime '28800'
set vpn ipsec esp-group ESP-SITE-B lifetime '3600'

If rekeying causes drops, increase lifetimes. If security policy requires short lifetimes, ensure DPD is configured to recover quickly.

Debugging IPsec

When IPsec fails, check in order:

1. Check SA Status

show vpn ipsec sa

Shows established Security Associations. You want to see both IKE SA and Child SA (ESP).

2. Check Connection State

show vpn ipsec connections

Shows connection status: ESTABLISHED, CONNECTING, INSTALLED, etc.

3. Check Logs

show log | match -i ipsec
# or
sudo journalctl -u strongswan -f

Common errors:

NO_PROPOSAL_CHOSEN: Algorithm mismatch
AUTHENTICATION_FAILED: Wrong PSK or ID mismatch
TS_UNACCEPTABLE: Traffic selector mismatch
INVALID_IKE_SPI: Stale SA, restart connection

4. Reset the Connection

reset vpn ipsec site-to-site peer SITE-B

Clears SAs and re-initiates. Often fixes “stuck” tunnels.

5. Verify Traffic Selectors

show vpn ipsec sa detail

Shows exactly what traffic selectors are installed. Mismatch here = traffic bypasses tunnel.

Common Issues

Symptom	Cause	Fix
No SA established	Firewall blocking 500/4500/ESP	Open firewall ports
Auth failed	PSK mismatch or wrong local/remote-id	Verify both match exactly
Connects then drops	Timer mismatch or rekey failure	Match lifetimes, check DPD
Traffic doesn’t flow	Traffic selector mismatch	Verify local/remote prefix match
Works then stops	NAT timeout (if behind NAT)	Ensure NAT-T is working

Debug Commands Summary

show vpn ipsec sa              # SA status
show vpn ipsec connections     # Connection state
show vpn ipsec sa detail       # Detailed SA info including traffic selectors
show vpn ipsec status          # Overall IPsec status
reset vpn ipsec site-to-site peer <name>  # Reset specific peer

Route-Based vs Policy-Based IPsec

VyOS supports both:

Policy-based (shown above): Traffic selectors define what goes through tunnel. Configured via tunnel X local/remote prefix. Simpler, but less flexible.

Route-based: Virtual tunnel interface (vti), routes determine what traffic enters. More flexible, better for dynamic routing.

Route-Based Example

configure

# Virtual tunnel interface
set interfaces vti vti0 address '10.255.0.1/30'
set interfaces vti vti0 description 'IPsec to Site B'

# IPsec connection using vti
set vpn ipsec site-to-site peer SITE-B tunnel 1 local prefix '0.0.0.0/0'
set vpn ipsec site-to-site peer SITE-B tunnel 1 remote prefix '0.0.0.0/0'
set vpn ipsec site-to-site peer SITE-B vti bind 'vti0'

# Route traffic to remote network through vti
set protocols static route 10.2.0.0/24 interface vti0

commit

Route-based is better when:

You need dynamic routing (OSPF/BGP over IPsec)
Multiple networks with complex routing
You want firewall rules on the tunnel interface

Firewall for IPsec Traffic

Traffic through IPsec still needs firewall consideration:

# If using VTI, apply firewall via forward filter
# Define what the remote site can access
set firewall ipv4 name IPSEC-IN default-action 'drop'
set firewall ipv4 name IPSEC-IN rule 10 action 'accept'
set firewall ipv4 name IPSEC-IN rule 10 state 'established'
set firewall ipv4 name IPSEC-IN rule 10 state 'related'
set firewall ipv4 name IPSEC-IN rule 20 action 'accept'
set firewall ipv4 name IPSEC-IN rule 20 destination address '10.1.0.0/24'

# Apply to forward filter
set firewall ipv4 forward filter rule 30 inbound-interface name 'vti0'
set firewall ipv4 forward filter rule 30 action 'jump'
set firewall ipv4 forward filter rule 30 jump-target 'IPSEC-IN'

Production Checklist

IKE and ESP parameters match on both peers
Lifetimes match exactly
DPD configured with appropriate action
Firewall allows UDP 500, 4500, and ESP
Traffic selectors match (local/remote swapped)
PSK is strong (20+ random characters)
Local and remote IDs match configuration
Monitoring/alerting for tunnel status

The Lesson

IPsec reliability comes down to discipline:

Timer discipline: Both peers must have identical lifetimes. IKE > ESP lifetime. Configure DPD to detect and recover from failures.
SA verification: Regularly check show vpn ipsec sa. If IKE SA exists but Child SA doesn’t, you have a Phase 2 problem. If neither exists, Phase 1 isn’t completing.
Methodical debugging: Check SA status → check logs for specific error → verify matching configuration → reset and try again.

IPsec is more complex than WireGuard, but it’s deterministic. When you understand the state machine (IKE SA → Child SA → traffic flows), you can diagnose any issue by figuring out where in that sequence things break.