Policy Routing Debug: Why Traffic Takes the Wrong Path

December 30, 2025 · 9 min read

#vyos #networking #routing #troubleshooting

Policy routing configured. Traffic still takes the default route. You add more rules. Still doesn’t work. You start guessing.

Policy-based routing (PBR) is simple in concept but has multiple points of failure. Each must be correct: match criteria, firewall marks, routing tables, rule priority. Miss one, and traffic ignores your policy.

PBR debugging needs systematic verification, not guessing.

PBR Components

Policy routing has four parts. All must be correct:

1. Policy: Match traffic and set mark
   └── firewall rules with mark action

2. Mark: Identify traffic for routing decision
   └── fwmark value (0x1, 0x2, etc.)

3. Table: Alternative routing table
   └── custom routes separate from main

4. Rule: Match mark and use table
   └── ip rule connecting mark to table

Verification Workflow

Step 1: Verify Policy Matches

# Check policy is applied to interface
show configuration commands | grep policy

# Expected:
# set policy route VPN-TRAFFIC rule 10 set mark 0x1
# set interfaces ethernet eth1 policy route VPN-TRAFFIC

# Verify policy rules
show configuration commands | grep "policy route"

Step 2: Verify Traffic Gets Marked

# Check firewall counters (if logging enabled)
show firewall

# Check if marking is happening with iptables
sudo iptables -t mangle -L -v -n

# Output should show packet counts on MARK rules:
# pkts bytes target     prot  in     out  source   destination
# 1234 5678K MARK       all   eth1   *    0.0.0.0/0  10.0.0.0/24   MARK set 0x1
#                                                                      ↑ Packets matched

# If counter is zero → policy not matching traffic

Step 3: Verify Routing Table Exists

# Show custom routing table
show ip route table 10

# Or directly:
ip route show table 10

# Should show routes:
# default via 10.10.0.1 dev tun0
# 10.10.0.0/24 dev tun0 proto kernel scope link src 10.10.0.2

Step 4: Verify Rule Connects Mark to Table

# Show all rules
ip rule show

# Expected output:
# 0:      from all lookup local
# 32765:  from all fwmark 0x1 lookup 10    ← Your rule
# 32766:  from all lookup main
# 32767:  from all lookup default

# If your fwmark rule is missing → rule not created
# If rule priority is wrong → might be evaluated after main table

Step 5: Test End-to-End

# Simulate marked packet lookup
ip route get 8.8.8.8 mark 0x1

# Expected:
# 8.8.8.8 via 10.10.0.1 dev tun0 table 10 mark 0x1

# If it shows main table route → mark not working

Common Problems

Problem 1: Policy Not Applied to Interface

# Symptom: Traffic not marked

# Check interface has policy
show interfaces ethernet eth1

# Should show:
# policy {
#     route VPN-TRAFFIC
# }

# If missing:
configure
set interfaces ethernet eth1 policy route VPN-TRAFFIC
commit

Problem 2: Wrong Match Criteria

# Symptom: Policy exists but doesn't match traffic

# Show policy details
show configuration commands | grep "policy route VPN-TRAFFIC"

# Common mistakes:
# - Source instead of destination (or vice versa)
# - Wrong subnet mask
# - Wrong protocol/port
# - Rule disabled

# Test what traffic should match:
# Rule says: source 192.168.1.0/24, destination 10.0.0.0/8
# Traffic is: source 192.168.2.100 → Won't match!

Problem 3: Mark Not Set

# Symptom: Rule matches but no mark

# Check iptables for mark rules
sudo iptables -t mangle -L PREROUTING -v -n

# Look for MARK target
# If MARK target shows 0 packets → not matching
# If no MARK rule → policy not generating iptables rules

# Verify mark is in policy:
show configuration commands | grep "set mark"
# set policy route VPN-TRAFFIC rule 10 set mark 0x1

Problem 4: Table Missing or Empty

# Symptom: Marked traffic uses main table

# Check table exists
ip route show table 10

# If empty or missing:
configure

# Add table (VyOS creates automatically with protocol static)
set protocols static table 10 route 0.0.0.0/0 next-hop 10.10.0.1

# Or for interface-based:
set protocols static table 10 route 0.0.0.0/0 interface tun0

commit

Problem 5: Rule Priority Wrong

# Symptom: Table has routes but not used

ip rule show
# 32765:  from all lookup main
# 32766:  from all fwmark 0x1 lookup 10    ← Too late!
# 32767:  from all lookup default

# Main table is checked before fwmark rule
# Traffic matches in main, never reaches your rule

# VyOS should set correct priority, but verify
# Lower number = higher priority
# fwmark rules should be before main (32766)

Problem 6: Return Traffic Not Marked

# Symptom: Outbound works, return traffic takes wrong path

# PBR typically marks only initiating direction
# Return traffic must be handled by:
# - Conntrack (automatic if stateful)
# - Separate marking rule for return

# Check if conntrack is preserving marks
sudo conntrack -L | grep mark

# Enable connection mark restore:
# Usually automatic with VyOS, but can verify in iptables
sudo iptables -t mangle -L -v

Debugging Commands

Check What Route Traffic Would Take

# Without mark (normal routing)
ip route get 8.8.8.8

# With mark (policy routing)
ip route get 8.8.8.8 mark 0x1

# Compare outputs to see if PBR is working

Check Packet Counts

# How many packets matched policy?
sudo iptables -t mangle -L PREROUTING -v -n | grep MARK

# Reset counters and test
sudo iptables -t mangle -Z
# Generate test traffic
curl http://10.0.0.100/
# Check counters again
sudo iptables -t mangle -L PREROUTING -v -n | grep MARK

Trace Packet Path

# Enable netfilter trace (temporary debug)
sudo modprobe nf_log_ipv4
sudo sysctl -w net.netfilter.nf_log.2=nf_log_ipv4

# Add trace rule for specific traffic
sudo iptables -t raw -A PREROUTING -s 192.168.1.100 -j TRACE

# Watch kernel log
dmesg -w

# Remove trace when done
sudo iptables -t raw -D PREROUTING -s 192.168.1.100 -j TRACE

Check Firewall Flow

# See where packet is in firewall processing
sudo iptables -t mangle -L -v -n  # Marking happens here
sudo iptables -t nat -L -v -n      # NAT happens here
sudo iptables -t filter -L -v -n   # Filtering happens here

# PBR marks in mangle PREROUTING
# Routing decision happens after mangle

VyOS PBR Configuration Reference

Complete Working Example

configure

# 1. Create routing table with routes
set protocols static table 10 route 0.0.0.0/0 next-hop 10.10.0.1

# 2. Create policy with mark
set policy route PBR-TO-VPN rule 10 destination address 10.0.0.0/8
set policy route PBR-TO-VPN rule 10 set mark 0x1
set policy route PBR-TO-VPN rule 10 set table 10

# 3. Apply policy to interface
set interfaces ethernet eth1 policy route PBR-TO-VPN

commit

Verify Each Component

# 1. Table has routes
ip route show table 10
# → Should show default via 10.10.0.1

# 2. Policy creates iptables rules
sudo iptables -t mangle -L PREROUTING -v -n | grep -i mark
# → Should show MARK rule

# 3. IP rule connects mark to table
ip rule show | grep fwmark
# → Should show: fwmark 0x1 lookup 10

# 4. Test packet routing
ip route get 10.0.0.100 mark 0x1
# → Should show: via 10.10.0.1 table 10

Advanced Debugging

Multiple Tables

# If using multiple tables:
ip route show table 10
ip route show table 20

# Verify rules don't conflict:
ip rule show

# Each mark should have unique table
# 32765:  fwmark 0x1 lookup 10
# 32764:  fwmark 0x2 lookup 20

Source-Based Routing

# If routing by source:
set policy route BY-SOURCE rule 10 source address 192.168.1.0/24
set policy route BY-SOURCE rule 10 set table 10

# Verify source matches
sudo iptables -t mangle -L PREROUTING -v -n
# Should show source match

DSCP/TOS Marking

# If matching on DSCP:
set policy route QOS-ROUTING rule 10 dscp 46
set policy route QOS-ROUTING rule 10 set table 10

# Verify packet has expected DSCP
sudo tcpdump -i eth1 -v | grep "tos"

Testing Strategy

Minimal Test

# 1. Create simple policy
set policy route TEST rule 10 destination address 8.8.8.8/32
set policy route TEST rule 10 set table 10
set protocols static table 10 route 0.0.0.0/0 blackhole

# 2. Apply to interface
set interfaces ethernet eth1 policy route TEST

# 3. Test
ping 8.8.8.8  # Should fail (blackhole)
ping 8.8.4.4  # Should work (not matched)

# 4. Clean up
delete policy route TEST
delete interfaces ethernet eth1 policy route
delete protocols static table 10

Incremental Testing

# Test each component in order:

# Test 1: Does table work?
ip route add blackhole 8.8.8.8 table 10
ip route get 8.8.8.8  # Uses main table → should work
# Clean: ip route del blackhole 8.8.8.8 table 10

# Test 2: Does rule work?
ip rule add fwmark 0x99 table 10
ip route add blackhole 8.8.8.8 table 10
ip route get 8.8.8.8 mark 0x99  # Should show table 10, blackhole
# Clean: ip rule del fwmark 0x99; ip route del blackhole 8.8.8.8 table 10

# Test 3: Does policy create mark?
# Apply policy, check iptables counters

The Lesson

PBR debugging needs systematic verification, not guessing.

When policy routing doesn’t work:

Verify policy applied to correct interface
Verify traffic matches policy rules
Verify mark is set (check iptables counters)
Verify table exists with correct routes
Verify rule connects mark to table
Test with ip route get ... mark

Each step depends on the previous. One failure breaks everything after it.

Don’t add more rules hoping it helps. Verify each component. Find the broken step. Fix that one thing.

PBR is a chain. Find the broken link.