LFCE 18 🐧 Policy Routing — Multiple Tables and Rules - KandZ

⏱️ Estimated read: 16 min | |

The previous chapter established the routing policy database and the rule selectors that drive it. This chapter is the operational counterpart: creating custom routing tables, populating them with routes, writing rules that direct specific traffic to them, and ensuring the entire configuration survives a reboot. A single policy rule is a proof of concept. A working multi-table configuration is a system in which each traffic class follows its own path, the paths do not interfere, and the configuration persists across restarts.

The canonical use case is a host with two network interfaces—one wired, one wireless, or two uplinks to different providers—where traffic from each interface must return through the same interface it arrived on. By default, Linux sends all outbound traffic through the primary interface, regardless of which interface received the incoming connection. Policy routing corrects this by matching the source address of the return traffic and directing it to a table whose default route points back out the correct interface .

Key point: Each custom table must contain not only its default route but also a route to the local network. Without the local route, the table produces no result for on-link destinations, and evaluation continues to the next rule, potentially selecting the wrong interface. The table is a complete routing context, not a single default gateway.

Why multiple tables are necessary

The return-path problem.

A server with two interfaces—eth0 at 192.168.0.129 and eth1 at 192.168.1.138—receives a connection on eth1. The reply packet has a source address of 192.168.1.138. The main routing table’s default route points out eth0. The reply leaves via eth0, which is the wrong interface from the client’s perspective. The client drops the reply because it never sent a packet to eth0‘s address. The connection fails. This is not a firewall issue; it is a routing issue. The main table does not know that packets with source 192.168.1.138 should leave via eth1. A policy rule matching that source and directing it to a table whose default route points out eth1 solves the problem .

The multiple-provider problem.

A host with two uplinks to different ISPs must route outbound traffic from each internal subnet through the correct provider. Traffic from 10.0.1.0/24 must use ISP1’s gateway; traffic from 10.0.2.0/24 must use ISP2’s gateway. The main table can hold only one default route. Policy routing creates two tables, each with its own default route, and two rules that match the source subnets and direct them to the appropriate table .

The persistence problem.

Custom tables and rules created with ip route and ip rule exist only in the running kernel. A reboot removes them. NetworkManager, systemd-networkd, and startup scripts each provide a mechanism for persisting the configuration, but none of them is automatic. The administrator must choose a persistence method, write the configuration, and verify it by rebooting. A policy routing configuration that works in a lab session but disappears on restart is not a configuration; it is a test.

a. Creating and populating custom tables

A custom table is identified by a number and optionally a name registered in /etc/iproute2/rt_tables. The name is for human readability; the kernel uses the number.

# Register table names for readability
# The number must be unique and not conflict with reserved values
echo "100 isp1" >> /etc/iproute2/rt_tables
echo "200 isp2" >> /etc/iproute2/rt_tables

# Verify the entries
cat /etc/iproute2/rt_tables | tail -2
# Output: 100 isp1
#         200 isp2

The reserved table IDs are 255 (local), 254 (main), 253 (default), and 0 (unspec). Custom tables should use numbers between 1 and 252 to avoid conflicts.

Each table must be populated with a complete routing context: a default route to the gateway for that path, and a route to the local network so that on-link destinations are handled correctly within the table.

# Populate the isp1 table
ip route add default via 203.0.113.1 dev ens3 table isp1
ip route add 10.0.1.0/24 dev ens3 table isp1

# Populate the isp2 table
ip route add default via 198.51.100.1 dev ens4 table isp2
ip route add 10.0.2.0/24 dev ens4 table isp2

# Verify the tables
ip route show table isp1
# Output: default via 203.0.113.1 dev ens3
#         10.0.1.0/24 dev ens3

ip route show table isp2
# Output: default via 198.51.100.1 dev ens4
#         10.0.2.0/24 dev ens4

The local route is not optional. Without it, a packet with a destination on the local subnet matches no route in the custom table, and evaluation continues to the next rule. If the main table then matches, the packet may leave via the wrong interface. The custom table must be self-contained for all destinations that traffic in that class will encounter.

b. Writing rules that select the tables

A rule consists of a selector and a table lookup. The selector determines which packets the rule applies to. The priority determines when the rule is evaluated relative to other rules. Lower priority numbers are evaluated first.

# Route traffic from the isp1 subnet through the isp1 table
ip rule add from 10.0.1.0/24 table isp1 priority 100

# Route traffic from the isp2 subnet through the isp2 table
ip rule add from 10.0.2.0/24 table isp2 priority 200

# Verify the rules
ip rule show
# Output includes:
# 100: from 10.0.1.0/24 lookup isp1
# 200: from 10.0.2.0/24 lookup isp2

The from selector matches the source address. For the return-path problem, the source address of the reply is the interface address that received the connection. A rule matching that address directs the reply to the correct table.

# For a host with two interfaces, route return traffic by source address
ip rule add from 192.168.0.129 table net0 priority 250
ip rule add from 192.168.1.138 table net1 priority 251

# Populate the tables with the correct default routes
ip route add default via 192.168.0.1 dev eth0 table net0
ip route add default via 192.168.1.1 dev eth1 table net1

# Add local routes to each table
ip route add 192.168.0.0/24 dev eth0 table net0
ip route add 192.168.1.0/24 dev eth1 table net1

The iif selector is used for forwarded traffic, where the packet arrives on one interface and must leave via another. The fwmark selector is used when the firewall marks packets based on criteria the routing layer cannot see, such as destination port or protocol.

# Route forwarded traffic arriving on eth1 back out eth1
ip rule add iif eth1 table net1 priority 300

# Route packets marked by nftables (e.g., HTTP traffic) to a specific table
ip rule add fwmark 0x1 table isp1 priority 50

The priority order matters. A rule with priority 50 is evaluated before a rule with priority 100. If the priority 50 rule matches and its table yields a route, the priority 100 rule is never consulted. Design the priority numbers so that the most specific selectors are evaluated first.

c. Verification and persistence

The ip route get command verifies the policy decision for a specific packet. The from option specifies the source address, which is essential when the rule selector depends on the source.

# Verify the route for traffic originating from the isp1 subnet
ip route get 8.8.8.8 from 10.0.1.50
# Output: 8.8.8.8 from 10.0.1.50 via 203.0.113.1 dev ens3 table isp1

# Verify the route for traffic originating from the isp2 subnet
ip route get 8.8.8.8 from 10.0.2.50
# Output: 8.8.8.8 from 10.0.2.50 via 198.51.100.1 dev ens4 table isp2

# Verify that traffic from an unmatched source uses the main table
ip route get 8.8.8.8 from 10.0.3.50
# Output: 8.8.8.8 from 10.0.3.50 via <main-table-gateway> dev <primary-interface>

Without the from option, ip route get tests the default source address, which may not match the policy rule being verified. The source address is part of the packet that the rule selector examines.

Persistence depends on the distribution’s network manager. NetworkManager provides a routing-rules property that accepts the same syntax as ip rule.

# Add routing rules and table routes through NetworkManager
nmcli connection modify ens3 +ipv4.routing-rules "priority 100 from 10.0.1.0/24 table 100"
nmcli connection modify ens3 +ipv4.routes "0.0.0.0/0 203.0.113.1 table=100"
nmcli connection modify ens3 +ipv4.routes "10.0.1.0/24 table=100"

nmcli connection modify ens4 +ipv4.routing-rules "priority 200 from 10.0.2.0/24 table 200"
nmcli connection modify ens4 +ipv4.routes "0.0.0.0/0 198.51.100.1 table=200"
nmcli connection modify ens4 +ipv4.routes "10.0.2.0/24 table=200"

# Reactivate the connections
nmcli connection up ens3
nmcli connection up ens4

For systems without NetworkManager, the configuration can be placed in /etc/rc.local or a systemd unit that runs after the network is up. The rc.local approach is simple but requires the service to be enabled and the file to be executable.

# /etc/rc.local
#!/bin/bash
sleep 5
ip route flush table isp1
ip route add default via 203.0.113.1 dev ens3 table isp1
ip route add 10.0.1.0/24 dev ens3 table isp1
ip rule add from 10.0.1.0/24 table isp1 priority 100

ip route flush table isp2
ip route add default via 198.51.100.1 dev ens4 table isp2
ip route add 10.0.2.0/24 dev ens4 table isp2
ip rule add from 10.0.2.0/24 table isp2 priority 200
exit 0

The sleep ensures the interfaces are up before the routes are added. The flush removes any stale entries from a previous run. After a reboot, ip rule show and ip route show table isp1 confirm that the configuration was applied.

Complete Example Session

# ============================================
# PART 1: INSPECT THE DEFAULT RPDB AND TABLES
# ============================================
ip rule show
# Output: 0: from all lookup local
#         32766: from all lookup main
#         32767: from all lookup default

ip route show table main
# Output shows the main table's routes, including the default gateway

# ============================================
# PART 2: REGISTER CUSTOM TABLE NAMES
# ============================================
echo "100 isp1" >> /etc/iproute2/rt_tables
echo "200 isp2" >> /etc/iproute2/rt_tables
cat /etc/iproute2/rt_tables | grep -E "isp1|isp2"
# Output: 100 isp1
#         200 isp2

# ============================================
# PART 3: POPULATE THE ISP1 TABLE
# ============================================
ip route add default via 203.0.113.1 dev ens3 table isp1
ip route add 10.0.1.0/24 dev ens3 table isp1
ip route show table isp1
# Output: default via 203.0.113.1 dev ens3
#         10.0.1.0/24 dev ens3

# ============================================
# PART 4: POPULATE THE ISP2 TABLE
# ============================================
ip route add default via 198.51.100.1 dev ens4 table isp2
ip route add 10.0.2.0/24 dev ens4 table isp2
ip route show table isp2
# Output: default via 198.51.100.1 dev ens4
#         10.0.2.0/24 dev ens4

# ============================================
# PART 5: ADD THE POLICY RULES
# ============================================
ip rule add from 10.0.1.0/24 table isp1 priority 100
ip rule add from 10.0.2.0/24 table isp2 priority 200
ip rule show
# Output includes:
# 100: from 10.0.1.0/24 lookup isp1
# 200: from 10.0.2.0/24 lookup isp2

# ============================================
# PART 6: VERIFY THE ROUTING DECISIONS
# ============================================
ip route get 8.8.8.8 from 10.0.1.50
# Output: 8.8.8.8 from 10.0.1.50 via 203.0.113.1 dev ens3 table isp1

ip route get 8.8.8.8 from 10.0.2.50
# Output: 8.8.8.8 from 10.0.2.50 via 198.51.100.1 dev ens4 table isp2

ip route get 8.8.8.8 from 10.0.3.50
# Output: 8.8.8.8 from 10.0.3.50 via <main-gateway> dev <main-interface>
# Traffic from unmatched sources uses the main table

# ============================================
# PART 7: ADD A FIREWALL-MARKED RULE
# ============================================
# Mark HTTP traffic with nftables
nft add table inet mangle
nft add chain inet mangle prerouting '{ type filter hook prerouting priority -150; }'
nft add rule inet mangle prerouting tcp dport 80 meta mark set 0x1

# Route marked traffic through isp1
ip rule add fwmark 0x1 table isp1 priority 50
ip rule show | grep "fwmark"
# Output: 50: from all fwmark 0x1 lookup isp1

# ============================================
# PART 8: PERSIST WITH NETWORKMANAGER
# ============================================
nmcli connection modify ens3 +ipv4.routing-rules "priority 100 from 10.0.1.0/24 table 100"
nmcli connection modify ens3 +ipv4.routes "0.0.0.0/0 203.0.113.1 table=100"
nmcli connection modify ens3 +ipv4.routes "10.0.1.0/24 table=100"

nmcli connection modify ens4 +ipv4.routing-rules "priority 200 from 10.0.2.0/24 table 200"
nmcli connection modify ens4 +ipv4.routes "0.0.0.0/0 198.51.100.1 table=200"
nmcli connection modify ens4 +ipv4.routes "10.0.2.0/24 table=200"

# ============================================
# PART 9: REACTIVATE AND VERIFY
# ============================================
nmcli connection up ens3
nmcli connection up ens4
ip rule show
# Rules persist after reactivation

# ============================================
# PART 10: TEST PERSISTENCE WITH REBOOT
# ============================================
# Reboot the system
# After reboot, verify the rules and tables are still present
ip rule show
ip route show table isp1
ip route show table isp2
# If the rules are absent, the persistence configuration failed

The ten parts demonstrate the complete workflow: inspect, register, populate, rule, verify, extend with firewall marks, persist, and test. The critical verification steps are Part 6 (the rules select the correct tables) and Part 10 (the configuration survives a reboot).

Quick Reference

Table and Rule Commands

Task	Command
Register table name	`echo "100 isp1" >> /etc/iproute2/rt_tables`
Add route to table	`ip route add <dest> via <gw> dev <if> table <name>`
Add rule	`ip rule add from <prefix> table <name> priority <n>`
Show rules	`ip rule show`
Show table	`ip route show table <name>`
Verify decision	`ip route get <dest> from <source>`
Flush table	`ip route flush table <name>`

Common Rule Selectors

Selector	Matches	Use Case
`from`	Source address	Return-path correction, source-based routing
`iif`	Incoming interface	Forwarded traffic
`fwmark`	Firewall mark	Port/protocol-based routing
`to`	Destination address	Destination-specific policy

Persistence Methods

Method	Command	Note
NetworkManager	`nmcli connection modify +ipv4.routing-rules`	Modern RHEL/Ubuntu
`rc.local`	Add commands to `/etc/rc.local`	Legacy; requires executable bit
systemd unit	Custom unit with `ip` commands	More control over ordering

Best Practices

✅ Do This:

# Register table names for readability
echo "100 isp1" >> /etc/iproute2/rt_tables                     # ✅

# Add a local route to every custom table
ip route add 10.0.1.0/24 dev ens3 table isp1                   # ✅

# Verify with ip route get and the correct source
ip route get 8.8.8.8 from 10.0.1.50                            # ✅

# Use unique priorities for custom rules
ip rule add from 10.0.1.0/24 table isp1 priority 100           # ✅

# Persist through NetworkManager or rc.local
nmcli connection modify ens3 +ipv4.routing-rules "..."         # ✅

# Flush tables before adding routes in rc.local
ip route flush table isp1                                      # ✅

❌ Don’t Do This:

# Create a table without a local route
ip route add default via 203.0.113.1 table isp1
# On-link traffic falls through to the next rule                 # ❌

# Use duplicate priorities without understanding the behavior
# Order becomes ambiguous                                       # ❌

# Assume the main table is consulted when a custom rule matches
# The first matching rule with a route terminates the lookup     # ❌

# Forget to specify the source in verification
ip route get 8.8.8.8    # Does not test the policy rule        # ❌

# Rely on ip rule add surviving reboot
# It modifies running state only                                # ❌

# Use rc.local without checking the service is enabled
# The file may not execute at boot                              # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Return traffic leaves wrong interface	No rule matching the source address	Add `from <interface-address>` rule
Rule matches but no route	Custom table missing default or local route	Populate table completely
Rules lost after reboot	`ip rule add` is not persistent	Configure through NetworkManager or rc.local
Traffic from unmatched source affected	Rule priority too high or selector too broad	Verify priorities and selectors
`ip route get` shows main table	Source not specified in verification	Add `from <source>` to the command
Table name not recognized	Name not registered in `/etc/iproute2/rt_tables`	Add the entry or use the numeric ID

Real-World Examples

1. Source-based routing for two ISPs

ip rule add from 10.0.1.0/24 table isp1 priority 100
ip route add default via 203.0.113.1 table isp1
ip route add 10.0.1.0/24 dev ens3 table isp1

2. Return-path correction for multi-homed host

ip rule add from 192.168.1.138 table net1 priority 251
ip route add default via 192.168.1.1 dev eth1 table net1
ip route add 192.168.1.0/24 dev eth1 table net1

3. Interface-based routing for forwarded traffic

ip rule add iif eth1 table net1 priority 300
ip route add default via 192.168.1.1 dev eth1 table net1

4. Firewall-marked routing for HTTP

nft add rule inet mangle prerouting tcp dport 80 meta mark set 0x1
ip rule add fwmark 0x1 table isp1 priority 50
ip route add default via 203.0.113.1 table isp1

5. Verify policy decision

ip route get 8.8.8.8 from 10.0.1.50

6. Persist rules with nmcli

nmcli connection modify ens3 +ipv4.routing-rules "priority 100 from 10.0.1.0/24 table 100"

7. Persist routes with nmcli

nmcli connection modify ens3 +ipv4.routes "0.0.0.0/0 203.0.113.1 table=100"

8. Flush a custom table

ip route flush table isp1

9. Register table name

echo "100 isp1" >> /etc/iproute2/rt_tables

10. rc.local configuration

sleep 5
ip route flush table isp1
ip route add default via 203.0.113.1 dev ens3 table isp1
ip rule add from 10.0.1.0/24 table isp1 priority 100

Visual

Multi-Table Policy Routing Architecture

┌──────────────────────────────────────────────────────────────┐
│  TWO UPLINKS, TWO TABLES, TWO RULES                          │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Traffic from 10.0.1.0/24                               │  │
│  └──────────────────────────┬─────────────────────────────┘  │
│                             │                                │
│                             ▼                                │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Rule priority 100: from 10.0.1.0/24 lookup isp1        │  │
│  └──────────────────────────┬─────────────────────────────┘  │
│                             │                                │
│                             ▼                                │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Table isp1                                             │  │
│  │ default via 203.0.113.1 dev ens3                       │  │
│  │ 10.0.1.0/24 dev ens3                                   │  │
│  └──────────────────────────┬─────────────────────────────┘  │
│                             │                                │
│                             ▼                                │
│                     Out ens3 to ISP1                         │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Traffic from 10.0.2.0/24                               │  │
│  └──────────────────────────┬─────────────────────────────┘  │
│                             │                                │
│                             ▼                                │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Rule priority 200: from 10.0.2.0/24 lookup isp2        │  │
│  └──────────────────────────┬─────────────────────────────┘  │
│                             │                                │
│                             ▼                                │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Table isp2                                             │  │
│  │ default via 198.51.100.1 dev ens4                      │  │
│  │ 10.0.2.0/24 dev ens4                                   │  │
│  └──────────────────────────┬─────────────────────────────┘  │
│                             │                                │
│                             ▼                                │
│                     Out ens4 to ISP2                         │
└──────────────────────────────────────────────────────────────┘

Return-Path Correction

┌──────────────────────────────────────────────────────────────┐
│  WHY THE CUSTOM TABLE NEEDS A LOCAL ROUTE                    │
│                                                              │
│  Client connects to 192.168.1.138 on eth1                    │
│                                                              │
│  Reply packet: src=192.168.1.138, dst=<client-ip>            │
│                                                              │
│  Rule: from 192.168.1.138 lookup net1                        │
│       │                                                      │
│       ▼                                                      │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Table net1                                             │  │
│  │ default via 192.168.1.1 dev eth1                       │  │
│  │ 192.168.1.0/24 dev eth1                                │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                              │
│  Reply destination is the client, which is on 192.168.1.0/24 │
│  If the local route is present:                              │
│    → Match 192.168.1.0/24 → out eth1 → correct               │
│                                                              │
│  If the local route is MISSING:                              │
│    → No match in net1 → continue to main table               │
│    → Main table matches → out eth0 → WRONG                   │
└──────────────────────────────────────────────────────────────┘

Rule Priority and Evaluation

┌──────────────────────────────────────────────────────────────┐
│  RULE EVALUATION ORDER (LOWEST PRIORITY NUMBER FIRST)        │
│                                                              │
│  Priority 0    from all lookup local                         │
│  │                                                           │
│  Priority 50   fwmark 0x1 lookup isp1   (HTTP traffic)       │
│  │                                                           │
│  Priority 100  from 10.0.1.0/24 lookup isp1                  │
│  │                                                           │
│  Priority 200  from 10.0.2.0/24 lookup isp2                  │
│  │                                                           │
│  Priority 300  iif eth1 lookup net1     (forwarded traffic)  │
│  │                                                           │
│  Priority 32766 from all lookup main    (default table)      │
│  │                                                           │
│  Priority 32767 from all lookup default (empty)              │
│                                                              │
│  A packet matching priority 50 never reaches 100.            │
│  A packet not matching 50, 100, 200, or 300 reaches main.    │
└──────────────────────────────────────────────────────────────┘

Summary

Item	Value
Custom table registration	`/etc/iproute2/rt_tables`
Table population	Default route + local route
Rule syntax	`ip rule add <selector> table <name> priority <n>`
Common selectors	`from`, `iif`, `fwmark`
Verification	`ip route get <dest> from <source>`
Persistence	NetworkManager, `rc.local`, systemd unit
Critical table content	Default route and local route
Rule evaluation	Lowest priority number first
First match with route	Terminates lookup

Key takeaways:

Each custom table must be a complete routing context. The default route handles off-link destinations; the local route handles on-link destinations. A table with only a default route falls through to the next rule for local traffic, potentially selecting the wrong interface .
The from selector solves the return-path problem. When a host receives a connection on one interface, the reply’s source address matches that interface’s address. A rule matching that source directs the reply to a table whose default route points back out the same interface .
Rule priority determines evaluation order. Lower numbers are evaluated first. The first rule whose selector matches and whose table yields a route terminates the lookup. Design priorities so that specific selectors are evaluated before broad ones.
ip route get with from is the verification tool. Without the source address, the command does not test the policy rule. The source is part of the packet that the selector examines.
Persistence is not automatic. Custom tables and rules exist only in the running kernel. NetworkManager, rc.local, or a systemd unit must write them at boot. Verify persistence by rebooting, not by inspecting the configuration file .
The iif selector is for forwarded traffic. Packets that arrive on one interface and are forwarded to another are matched by their incoming interface. The from selector does not match forwarded traffic because the source address is the original sender, not the router.

Remember: Multiple tables and rules are the mechanism that lets one Linux host serve multiple network paths without conflict. The table is the path; the rule is the selector that decides which packets use it. A table without a local route is incomplete. A rule without a populated table is inert. A configuration without persistence is temporary. Build each table completely, write rules that match the traffic that must use each path, verify with ip route get and the correct source, and persist the configuration so it survives the reboot that ends every lab session.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!