LFCE 122 🐧 rsyslog — Client Forwarding
The rsyslog client is the sending side of centralized logging. It reads messages generated by the local system and forwards them to a remote collector. The forwarding is configured as an output action using the omfwd module, which supports UDP, TCP, and TLS-encrypted TCP. The choice of transport and queue configuration determines whether messages survive a collector outage, whether forwarding blocks local logging, and whether the client can recover from temporary network failures.
This chapter covers the client-side configuration: the omfwd action and its essential parameters, the transport options and their reliability characteristics, the queue configuration that prevents message loss during collector downtime, and the filtering that selects which messages to forward. It also covers the target-pool feature for load balancing across multiple collectors, and the TLS configuration for encrypted forwarding.
Key point: A forwarding action without queue.type="linkedList" can block local logging if the collector becomes unreachable. The queue decouples the local log write from the network send, and queue.saveOnShutdown="on" preserves queued messages across a restart.
Why client forwarding has these requirements
The blocking problem. By default, an omfwd action sends messages synchronously. If the collector is unreachable, the action waits for the connection to time out before returning, and during that wait, the local log write is blocked. The rsyslog documentation explicitly recommends queue.type="linkedList" for TCP forwarding to prevent this blocking behavior . The queue holds messages in memory while the send is in progress or retrying, and the local write completes immediately.
The message loss problem. A memory-only queue is lost if rsyslog restarts or the system crashes. Adding queue.filename creates a disk-assisted queue: messages that cannot be sent are written to disk and survive a restart. The queue.saveOnShutdown="on" directive ensures that queued messages are persisted when rsyslog shuts down cleanly . For log data that must not be lost, this combination is essential.
The transport reliability problem. UDP is the traditional syslog transport and the default for omfwd. It has no delivery guarantee. TCP adds connection-oriented reliability. The rsyslog documentation recommends TCP for forwarding where reliability matters . For encrypted forwarding, TCP with a TLS stream driver (gtls, ossl, or mbedtls) provides both reliability and confidentiality .
The retry problem. When a TCP connection fails, the action has to decide whether to retry and for how long. The action.resumeRetryCount parameter controls this. A value of -1 means retry indefinitely, which is appropriate for a client that must eventually deliver its logs when the collector returns . Without it, the action may give up after a finite number of attempts and drop the queued messages.
a. The omfwd action and basic parameters
The omfwd module is built into rsyslog and does not require a separate module(load=...) directive . The action is configured with type="omfwd" and specifies the target, port, and protocol.
# Forward all messages to a collector over TCP
*.* action(
type="omfwd"
target="logserver.example.com"
port="514"
protocol="tcp"
)The target parameter accepts a hostname or IP address. The port parameter defaults to 514 for UDP and TCP. The protocol parameter accepts udp or tcp . Without a queue configuration, this action sends synchronously and can block.
b. Adding the queue for reliability
The queue is configured inside the action. queue.type="linkedList" creates an in-memory linked-list queue. queue.filename creates a disk-assisted queue that spills to disk when the memory queue is full or when rsyslog shuts down.
# Forward with a disk-assisted queue
*.* action(
type="omfwd"
target="logserver.example.com"
port="514"
protocol="tcp"
queue.type="linkedList"
queue.filename="fwd_logs"
queue.saveOnShutdown="on"
action.resumeRetryCount="-1"
)The queue.filename value must be unique within the configuration. No two actions can share the same queue file . The queue.saveOnShutdown directive writes queued messages to disk when rsyslog stops, so they can be sent after the next start . The action.resumeRetryCount="-1" tells the action to retry indefinitely when the target is unreachable .
c. Filtering and selective forwarding
A forwarding action can be preceded by a selector to forward only specific facilities or priorities. The selector syntax is the same as for local log rules.
# Forward only authentication messages
auth,authpriv.* action(
type="omfwd"
target="logserver.example.com"
port="514"
protocol="tcp"
queue.type="linkedList"
)The AWS CloudWatch documentation uses this pattern to send only auth and authpriv messages to a remote endpoint, while leaving other messages to local logging . The selector auth,authpriv.* matches all priorities from those two facilities. Other facilities are not forwarded by this action.
d. Target pools for load balancing
The target parameter can accept an array of hostnames, forming a target pool. The pool distributes messages in round-robin order across the online targets. Unreachable hosts are removed and retried every 30 seconds. If all targets fail, the action suspends until one recovers .
# Forward to a pool of collectors
*.* action(
type="omfwd"
target=["logserver1.example.com", "logserver2.example.com", "logserver3.example.com"]
port="514"
protocol="tcp"
queue.type="linkedList"
queue.filename="fwd_pool"
queue.saveOnShutdown="on"
)Target pools require TCP. UDP sends only to the first target . This feature is useful when a single collector cannot handle the aggregate log volume, or when high availability across multiple collectors is required.
e. TLS-encrypted forwarding
Encrypted forwarding uses TCP with a stream driver. The StreamDriver parameter is set to gtls (GnuTLS), ossl (OpenSSL), or mbedtls. The StreamDriverMode="1" enables TLS, and StreamDriverAuthMode controls certificate validation.
# Forward over TLS with certificate validation
*.* action(
type="omfwd"
target="logserver.example.com"
port="6514"
protocol="tcp"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/certvalid"
queue.type="linkedList"
queue.filename="fwd_tls"
queue.saveOnShutdown="on"
action.resumeRetryCount="-1"
)The StreamDriverAuthMode="x509/certvalid" requires the collector to present a valid certificate signed by a trusted CA. The CA certificate is configured globally with global(DefaultNetstreamDriverCAFile="...") . The port 6514 is the standard syslog TLS port, though the collector can be configured to listen on any port.
Complete Example Session
# ============================================
# PART 1: THE BASIC FORWARDING ACTION
# ============================================
sudo tee /etc/rsyslog.d/50-forward.conf <<'EOF'
*.* action(
type="omfwd"
target="logserver.example.com"
port="514"
protocol="tcp"
)
EOF
# This works, but the action is synchronous.
# If the collector is unreachable, local logging blocks.# ============================================
# PART 2: ADD THE QUEUE
# ============================================
sudo tee /etc/rsyslog.d/50-forward.conf <<'EOF'
*.* action(
type="omfwd"
target="logserver.example.com"
port="514"
protocol="tcp"
queue.type="linkedList"
queue.filename="fwd_logs"
queue.saveOnShutdown="on"
action.resumeRetryCount="-1"
)
EOF
# The queue decouples local writes from the network send.
# Messages are buffered in memory and spilled to disk if needed.# ============================================
# PART 3: VERIFY THE CONFIGURATION
# ============================================
sudo rsyslogd -N1
# Expected: config validation run succeeded.# ============================================
# PART 4: RESTART AND CHECK
# ============================================
sudo systemctl restart rsyslog
systemctl status rsyslog --no-pager# ============================================
# PART 5: TEST FORWARDING
# ============================================
logger -t fwd-test "hello from the client"
# On the collector:
# sudo tail -f /var/log/remote/<client-hostname>/fwd-test.log# ============================================
# PART 6: FILTER BY FACILITY
# ============================================
sudo tee /etc/rsyslog.d/50-forward.conf <<'EOF'
# Forward only auth messages
auth,authpriv.* action(
type="omfwd"
target="logserver.example.com"
port="514"
protocol="tcp"
queue.type="linkedList"
queue.filename="fwd_auth"
queue.saveOnShutdown="on"
)
EOF
# Other facilities remain local.# ============================================
# PART 7: TARGET POOL
# ============================================
sudo tee /etc/rsyslog.d/50-forward.conf <<'EOF'
*.* action(
type="omfwd"
target=["logserver1.example.com", "logserver2.example.com"]
port="514"
protocol="tcp"
queue.type="linkedList"
queue.filename="fwd_pool"
queue.saveOnShutdown="on"
action.resumeRetryCount="-1"
)
EOF
# Round-robin across the pool.
# Requires TCP; UDP sends only to the first target.# ============================================
# PART 8: TLS CONFIGURATION
# ============================================
# Global CA configuration (required for TLS)
sudo tee /etc/rsyslog.d/00-tls.conf <<'EOF'
global(DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/logserver-ca.pem")
EOF
# TLS forwarding action
sudo tee /etc/rsyslog.d/50-forward.conf <<'EOF'
*.* action(
type="omfwd"
target="logserver.example.com"
port="6514"
protocol="tcp"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/certvalid"
queue.type="linkedList"
queue.filename="fwd_tls"
queue.saveOnShutdown="on"
action.resumeRetryCount="-1"
)
EOF# ============================================
# PART 9: CHECK THE QUEUE FILES
# ============================================
# Queue files are written to the work directory
# (default /var/spool/rsyslog)
ls -la /var/spool/rsyslog/fwd_logs*
# If the collector was unreachable, queued messages
# are stored here until the connection is restored.# ============================================
# PART 10: TROUBLESHOOT A FAILED FORWARD
# ============================================
# 1. Check the collector is reachable: nc -zv logserver 514
# 2. Check rsyslog logs: journalctl -u rsyslog | tail -20
# 3. Check queue files exist: ls /var/spool/rsyslog/
# 4. Verify config: rsyslogd -N1The ten parts covered the basic action, adding the queue, validation, restart, testing, filtering, target pools, TLS, queue file inspection, and troubleshooting.
Quick Reference
omfwd parameters
| Parameter | Purpose | Example |
|---|---|---|
target | Collector hostname or IP (or array for pool) | "logserver" |
port | Destination port | "514" |
protocol | udp or tcp | "tcp" |
queue.type | Queue type | "linkedList" |
queue.filename | Disk-assisted queue prefix | "fwd_logs" |
queue.saveOnShutdown | Persist queue on shutdown | "on" |
action.resumeRetryCount | Retry count (-1 = forever) | "-1" |
StreamDriver | TLS driver | "gtls" |
StreamDriverMode | Enable TLS | "1" |
StreamDriverAuthMode | Certificate validation | "x509/certvalid" |
Transport options
| Protocol | Reliability | Encryption | Queue required |
|---|---|---|---|
udp | None | No | Recommended |
tcp | Connection-oriented | No | Recommended |
tcp + TLS | Connection-oriented | Yes | Recommended |
Queue types
| Type | Storage | Survives restart |
|---|---|---|
direct | No queue | No |
linkedList | Memory | No |
linkedList + filename | Memory + disk | Yes (with saveOnShutdown) |
Best Practices
✅ Do This:
# Always add a linked-list queue for TCP forwarding
queue.type="linkedList" # ✅# Add a filename for disk-assisted persistence
queue.filename="fwd_logs" # ✅# Persist the queue on shutdown
queue.saveOnShutdown="on" # ✅# Retry indefinitely for critical log streams
action.resumeRetryCount="-1" # ✅# Validate before restarting
rsyslogd -N1 # ✅❌ Don’t Do This:
# Forward without a queue
action(type="omfwd" target="logserver" protocol="tcp") # ❌ (can block)# Use UDP for reliable forwarding
protocol="udp" # ❌ (no guarantee)# Share a queue filename between actions
queue.filename="shared" # in two actions # ❌ (must be unique)# Skip TLS certificate validation
StreamDriverAuthMode="anon" # in production # ❌Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Local logging blocks when collector is down | No queue configured | Add queue.type="linkedList" |
| Queued messages lost on restart | No queue.filename or saveOnShutdown | Add both |
| Forwarding stops after collector recovers | resumeRetryCount not set to -1 | Add action.resumeRetryCount="-1" |
| Target pool does not load-balance | UDP protocol used | Use protocol="tcp" |
| TLS fails with certificate error | CA not configured | Add global(DefaultNetstreamDriverCAFile=...) |
| Queue file conflict | Same queue.filename used twice | Use unique filenames per action |
Real-World Examples
1. Basic TCP forwarding
*.* action(type="omfwd" target="logserver" port="514" protocol="tcp")2. TCP with queue
*.* action(type="omfwd" target="logserver" port="514" protocol="tcp" queue.type="linkedList")3. Disk-assisted queue
*.* action(type="omfwd" target="logserver" port="514" protocol="tcp" queue.type="linkedList" queue.filename="fwd" queue.saveOnShutdown="on")4. Infinite retry
action.resumeRetryCount="-1"5. Filter auth only
auth,authpriv.* action(type="omfwd" target="logserver" port="514" protocol="tcp")6. Target pool
target=["logserver1", "logserver2"] port="514" protocol="tcp"7. TLS forwarding
StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/certvalid"8. CA configuration
global(DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca.pem")9. Validate config
rsyslogd -N110. Check queue files
ls -la /var/spool/rsyslog/Visual
The forwarding action with queue
┌─────────────────────────────────────────────────────────────┐
│ CLIENT FORWARDING │
│ │
│ Local application │
│ │ │
│ ▼ │
│ rsyslog local rules ──► local files │
│ │ │
│ ▼ │
│ Selector: *.* │
│ │ │
│ ▼ │
│ Action: omfwd │
│ │ │
│ ▼ │
│ Queue (linkedList) │
│ ├── memory buffer │
│ └── disk file (if filename set) │
│ │ │
│ ▼ │
│ Network send (TCP/UDP/TLS) │
│ │ │
│ ▼ │
│ Collector │
│ │
│ The queue decouples local write from network send. │
└─────────────────────────────────────────────────────────────┘Queue persistence
┌─────────────────────────────────────────────────────────────┐
│ QUEUE TYPES │
│ │
│ linkedList only: │
│ Memory only │
│ Lost on restart │
│ │
│ linkedList + filename: │
│ Memory + disk spill │
│ Survives restart if saveOnShutdown="on" │
│ Queue file: /var/spool/rsyslog/fwd_logs │
│ │
│ When collector is down: │
│ Messages accumulate in memory queue │
│ If memory fills, spill to disk │
│ On restart, disk queue is read and sent │
└─────────────────────────────────────────────────────────────┘Target pool
┌─────────────────────────────────────────────────────────────┐
│ TARGET POOL (TCP ONLY) │
│ │
│ Client │
│ │ │
│ ├── Message 1 ──► logserver1 │
│ ├── Message 2 ──► logserver2 │
│ ├── Message 3 ──► logserver3 │
│ └── ... round-robin │
│ │
│ logserver2 goes down: │
│ ├── Removed from pool │
│ ├── Retried every 30 seconds │
│ └── Messages continue to logserver1 and logserver3 │
│ │
│ All servers down: │
│ └── Action suspends until one recovers │
└─────────────────────────────────────────────────────────────┘Transport comparison
┌─────────────────────────────────────────────────────────────┐
│ TRANSPORT CHOICES │
│ │
│ UDP │
│ Speed: highest │
│ Reliability: none │
│ Encryption: no │
│ Use: acceptable loss, legacy devices │
│ │
│ TCP │
│ Speed: moderate │
│ Reliability: connection-oriented │
│ Encryption: no │
│ Use: standard centralized logging │
│ │
│ TCP + TLS │
│ Speed: lowest │
│ Reliability: connection-oriented │
│ Encryption: yes │
│ Use: sensitive logs, compliance requirements │
└─────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Forwarding module | omfwd (built-in, no load required) |
| Basic action | action(type="omfwd" target="..." port="..." protocol="...") |
| Recommended protocol | tcp (with queue) |
| Queue type | queue.type="linkedList" |
| Disk persistence | queue.filename + queue.saveOnShutdown="on" |
| Infinite retry | action.resumeRetryCount="-1" |
| Target pool | target=[...] (TCP only) |
| TLS driver | StreamDriver="gtls" |
| TLS validation | StreamDriverAuthMode="x509/certvalid" |
| CA configuration | global(DefaultNetstreamDriverCAFile="...") |
Key takeaways:
- Always configure a queue for
omfwd. Withoutqueue.type="linkedList", the action sends synchronously and can block local logging when the collector is unreachable. The queue holds messages while the network operation completes . queue.filenameandqueue.saveOnShutdownprevent message loss across restarts. A memory-only queue is lost when rsyslog stops. The disk-assisted queue persists queued messages so they can be delivered when the collector returns .action.resumeRetryCount="-1"retries indefinitely. For log streams that must eventually deliver, this setting prevents the action from giving up after a finite number of attempts .- TCP is the recommended transport for production. UDP has no delivery guarantee. TCP with a queue provides both reliability and non-blocking local logging .
- Target pools require TCP and provide load balancing. The
targetparameter accepts an array of hostnames. Messages are distributed round-robin, and unreachable hosts are removed and retried every 30 seconds . - TLS forwarding requires a CA certificate. The
StreamDriver="gtls"withStreamDriverAuthMode="x509/certvalid"validates the collector’s certificate against a CA configured inDefaultNetstreamDriverCAFile.
Remember: The rsyslog client configuration is a reliability problem. The action sends messages to the collector. The queue prevents the send from blocking local logging. The disk file prevents the queue from being lost on restart. The retry count prevents the action from giving up when the collector is temporarily unavailable. The selector determines what gets forwarded. Get the queue right first—without it, a collector outage becomes a local logging outage. Then choose the transport and filtering that match the log stream’s requirements.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!