LFCE 120 🐧 Centralized Logging — Concepts
Centralized logging is the practice of collecting log messages from many machines onto a single server. Instead of logging into each host to read /var/log/messages, an administrator reads one consolidated location. The standard Linux implementation uses rsyslog, the same daemon that handles local logging, configured to forward messages to a remote collector or to receive them from clients.
This chapter covers the conceptual foundation of centralized logging with rsyslog: the client-server model, the transport protocols and their trade-offs, the facility and priority selectors that determine which messages are forwarded, and the filtering and templating mechanisms that organize incoming logs by host. It explains why a central collector exists, what problems it solves that local logging cannot, and what risks it introduces. By the end, you will understand the architecture that the configuration chapter builds on.
Key point: Centralized logging with rsyslog is not a separate service. It is the same rsyslogd daemon configured with input modules to receive remote messages and output actions to forward local messages. The collector and the client run the same software.
Why centralized logging exists
The distributed investigation problem. When an application fails across three servers, the evidence is scattered across three /var/log/ directories. Correlating timestamps across machines requires logging into each one, and by the time the investigation begins, log rotation may have already discarded the relevant entries. A central collector receives messages as they are generated, preserves them in one place, and makes cross-host correlation a single search rather than a multi-host expedition .
The retention and capacity problem. A single server’s disk is sized for its own logs, not for the logs of an entire fleet. A central collector can be provisioned with dedicated storage, separate from the OS partition, and sized for the aggregate volume and retention period the organization requires. Red Hat’s documentation notes that centralizing logs “reduces the need to log into individual machines to check the log messages” and increases the chances that logs are stored in “a secure and tamper-proof environment” .
The tamper-resistance problem. Logs stored only on the machine that generated them are vulnerable to the same compromise that generated the log entries. An attacker who gains root on a host can delete or modify the local logs. A central collector receives messages over the network and stores them on a different machine, which raises the bar for an attacker who wants to erase evidence. This is not tamper-proof—a compromised collector is still a single point of failure—but it is strictly better than local-only storage .
The protocol standardization problem. rsyslog implements the syslog protocol, which has a defined message format: a priority value combining facility and severity, a timestamp, a hostname, a program name, and the message body. Because the format is standardized, a central collector can receive logs from Linux servers, network devices, and appliances that speak syslog. The collector does not need a custom parser for each source type; it needs a ruleset that routes based on the standard fields.
a. The client-server model
The architecture is simple in principle. A client is any system configured to forward some or all of its log messages to a remote destination. A server (or collector) is a system configured to listen for incoming syslog messages on a network port and write them to local files or forward them elsewhere.
The same rsyslog package provides both roles. The client uses the omfwd output module to forward messages. The server uses the imtcp or imudp input modules to receive them . A single rsyslog.conf can contain both input and output configuration, which means a system can be a client to one collector and a collector for other clients simultaneously.
# Client side: forward all messages to a collector over TCP
action(type="omfwd" target="logserver.example.com" port="514" protocol="tcp" queue.type="linkedList")
# Server side: listen for incoming messages over TCP
module(load="imtcp")
input(type="imtcp" port="514")The client does not need to know how the server stores the logs. The server does not need to know which applications generated them. The protocol carries the metadata, and the server’s rules decide what to do with each message. The queue.type="linkedList" on the client is a recommended practice for TCP forwarding: it prevents the forwarding action from blocking local logging if the collector is temporarily unreachable .
b. Transport protocols: UDP, TCP, and RELP
rsyslog supports three transport mechanisms for forwarding and receiving remote logs, and the choice has reliability and performance consequences.
UDP is the traditional syslog transport, defined in the original RFC. It is connectionless, has no delivery guarantee, and does not detect message loss. It is the lowest-overhead option and remains the default protocol for omfwd on port 514 . It is acceptable when losing a few log messages during network congestion is tolerable, and unacceptable when it is not.
TCP adds connection-oriented delivery and error handling. The imtcp and omfwd modules support it, and it copes better with network congestion than UDP. The rsyslog documentation explicitly recommends TCP for forwarding where reliability matters, paired with a linked-list queue to absorb temporary collector outages . It is the standard choice for centralized logging.
RELP (Reliable Event Logging Protocol) is a third option designed specifically for syslog. It runs over TCP but adds application-level acknowledgments, so the sender knows whether the receiver accepted the message. It requires the rsyslog-relp and librelp packages on both sides. It is the choice when message loss is not acceptable and the additional complexity is justified.
c. Facilities and priorities
Syslog messages are classified by two attributes that determine how they are routed: facility and priority. The facility identifies the subsystem that generated the message: kern for kernel messages, auth and authpriv for authentication, mail for mail services, daemon for system daemons, and user for user-level messages, among others. The priority (also called severity) identifies the urgency: emerg, alert, crit, err, warning, notice, info, and debug.
A selector combines facility and priority with a dot: authpriv.* matches all priorities from the authpriv facility. *.info matches info and higher priorities from all facilities. A rule consists of a selector and an action: the selector chooses which messages match, and the action decides what happens to them, such as writing to a file or forwarding to a remote host .
# Forward all authpriv messages to the collector over TCP
authpriv.* action(type="omfwd" target="logserver.example.com" port="514" protocol="tcp")
# Forward info and higher from all facilities except mail
*.info;mail.none action(type="omfwd" target="logserver.example.com" port="514" protocol="tcp")The facility and priority model is why a central collector can be selective. A client can forward only security-relevant messages, or only errors, rather than the entire log stream.
d. Templates and per-host organization
A collector that receives logs from dozens of hosts cannot write them all to /var/log/messages. The file would be unreadable and the per-host context would be lost. rsyslog provides templates to construct dynamic file paths from message properties.
A template is a named definition that combines constant strings with message properties. The hostname property yields the client’s hostname, and the programname property yields the name of the program that generated the message. A list-type template can construct a path like /var/log/remote/<hostname>/<programname>.log, which separates each client’s logs into its own directory and each program’s logs into its own file .
template(name="RemoteLogs" type="list") {
constant(value="/var/log/remote/")
property(name="hostname" securePath="replace")
constant(value="/")
property(name="programname" securePath="replace")
constant(value=".log")
}The securePath="replace" option is not optional. Without it, a malicious client could send a hostname containing path traversal characters (../) and cause the collector to write outside the intended directory. The replace mode sanitizes the property value before it is used in a file path. Red Hat’s documentation uses SecurePath="replace" in its own template examples for the same reason .
e. Configuration file structure
rsyslog reads /etc/rsyslog.conf and all files in /etc/rsyslog.d/ in lexical order . The main file contains global directives and module loads; the drop-in directory is where package-specific or site-specific rules belong. This structure means a remote logging configuration can be added as a new file in /etc/rsyslog.d/ without editing the main configuration, which reduces the risk of breaking the base system’s logging and avoids conflicts with package upgrades .
The modern configuration syntax uses module(load="...") for inputs and action(type="...") for outputs. The older $ModLoad and *.* @@host syntax still works, but the module and action syntax is what current documentation and examples use .
Complete Example Session
# ============================================
# PART 1: THE CLIENT-SERVER ROLES
# ============================================
# Client: forwards local logs to a remote host
# Server: receives remote logs and writes them to disk
#
# Both run rsyslogd.
# The difference is configuration, not software.# ============================================
# PART 2: THE MESSAGE FORMAT
# ============================================
# A syslog message carries:
# Priority (facility + severity)
# Timestamp
# Hostname
# Program name
# Message body
#
# The collector uses these fields for routing.# ============================================
# PART 3: FACILITY EXAMPLES
# ============================================
# kern kernel messages
# auth authentication (deprecated)
# authpriv authentication (private)
# mail mail service
# daemon system daemons
# user user-level messages
# local0-7 custom facilities# ============================================
# PART 4: PRIORITY EXAMPLES
# ============================================
# emerg system is unusable
# alert action must be taken immediately
# crit critical conditions
# err error conditions
# warning warning conditions
# notice normal but significant
# info informational
# debug debug-level messages# ============================================
# PART 5: SELECTOR SYNTAX
# ============================================
# authpriv.* all priorities, authpriv facility
# *.info info and higher, all facilities
# *.info;mail.none info and higher, except mail
# authpriv.err err and higher, authpriv# ============================================
# PART 6: UDP VERSUS TCP
# ============================================
# UDP (port 514):
# No delivery guarantee
# Lowest overhead
# Default protocol for omfwd
#
# TCP (port 514):
# Connection-oriented
# Error handling
# Recommended with queue.type="linkedList"# ============================================
# PART 7: RELP FOR RELIABILITY
# ============================================
# RELP = Reliable Event Logging Protocol
# Runs over TCP with application-level acks
# Requires rsyslog-relp and librelp
# Use when message loss is unacceptable# ============================================
# PART 8: TEMPLATE FOR PER-HOST FILES
# ============================================
template(name="RemoteLogs" type="list") {
constant(value="/var/log/remote/")
property(name="hostname" securePath="replace")
constant(value="/")
property(name="programname" securePath="replace")
constant(value=".log")
}
# Result: /var/log/remote/client01/sshd.log# ============================================
# PART 9: THE securePath OPTION
# ============================================
# Without securePath="replace":
# A client with hostname "../etc" could write to /var/log/etc/
#
# With securePath="replace":
# Path traversal characters are sanitized
#
# Red Hat uses SecurePath="replace" in its own templates.# ============================================
# PART 10: CONFIGURATION LAYOUT
# ============================================
# /etc/rsyslog.conf main file, global directives
# /etc/rsyslog.d/*.conf drop-in files, site rules
#
# Files are processed in lexical order.
# Site-specific rules belong in rsyslog.d/.The ten parts covered the client-server model, the message format, facilities, priorities, selector syntax, UDP versus TCP, RELP, templates for per-host files, the securePath security requirement, and the configuration file layout.
Quick Reference
Roles
| Role | Module | Direction |
|---|---|---|
| Client | omfwd | Forwards local logs to remote |
| Server | imtcp / imudp | Receives remote logs |
Transport options
| Protocol | Port | Delivery guarantee | Overhead |
|---|---|---|---|
| UDP | 514 | None | Lowest |
| TCP | 514 | Connection-oriented | Moderate |
| RELP | 6514 (typical) | Application-level acks | Highest |
Common facilities
| Facility | Subsystem |
|---|---|
kern | Kernel |
auth / authpriv | Authentication |
mail | Mail services |
daemon | System daemons |
user | User-level messages |
local0–local7 | Custom |
Common priorities
| Priority | Meaning |
|---|---|
emerg | System unusable |
alert | Immediate action required |
crit | Critical conditions |
err | Error conditions |
warning | Warning conditions |
info | Informational |
debug | Debug-level |
Message properties for templates
| Property | Value |
|---|---|
hostname | Client hostname |
programname | Program that generated the message |
fromhost-ip | Source IP address |
syslogfacility | Facility name |
syslogpriority | Priority name |
Best Practices
✅ Do This:
# Use TCP with a linked-list queue for forwarding
action(type="omfwd" protocol="tcp" target="logserver" port="514" queue.type="linkedList") # ✅# Use securePath="replace" in templates with dynamic paths
property(name="hostname" securePath="replace") # ✅# Put site-specific rules in /etc/rsyslog.d/
# (new file, not edits to rsyslog.conf) # ✅# Use templates to separate logs by host
constant(value="/var/log/remote/") # ✅# Test the configuration syntax before restarting
rsyslogd -N1 # ✅❌ Don’t Do This:
# Rely on UDP when message loss is unacceptable
input(type="imudp" port="514") # ❌# Use dynamic paths without securePath
property(name="hostname") # ❌ (path traversal)# Edit /etc/rsyslog.conf directly for site rules
vim /etc/rsyslog.conf # ❌ (overwritten by upgrades)# Store all remote logs in /var/log/messages
# Mixed hosts, unreadable, no context # ❌# Forward without a queue when using TCP
action(type="omfwd" protocol="tcp" target="logserver" port="514") # ❌ (can block)Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Remote logs mixed with local | No template applied to incoming messages | Create a per-host template and ruleset |
| Path traversal vulnerability | securePath omitted from hostname property | Add securePath="replace" |
| Messages lost under load | UDP transport used without acknowledgment | Switch to TCP with a linked-list queue |
| Forwarding blocks local logging | TCP forwarding without a queue | Add queue.type="linkedList" |
| Collector storage fills | No log rotation on the remote directory | Configure logrotate for /var/log/remote/ |
| Client forwards nothing | Firewall blocks port 514 or module not loaded | Open the port and verify imtcp/imudp is loaded |
Real-World Examples
1. Client forwarding all logs over TCP with queue
*.* action(type="omfwd" target="logserver.example.com" port="514" protocol="tcp" queue.type="linkedList")2. Server listening over TCP
module(load="imtcp")
input(type="imtcp" port="514")3. Server listening over UDP
module(load="imudp")
input(type="imudp" port="514")4. Per-host template with securePath
template(name="RemoteLogs" type="list") {
constant(value="/var/log/remote/")
property(name="hostname" securePath="replace")
constant(value="/")
property(name="programname" securePath="replace")
constant(value=".log")
}5. Forward only authpriv
authpriv.* action(type="omfwd" target="logserver" port="514" protocol="tcp")6. Forward info and higher, exclude mail
*.info;mail.none action(type="omfwd" target="logserver" port="514" protocol="tcp")7. RELP output
action(type="omrelp" target="logserver" port="6514")8. RELP input
module(load="imrelp")
input(type="imrelp" port="6514")9. Drop-in configuration file
# /etc/rsyslog.d/50-remote.conf
*.* action(type="omfwd" target="logserver" port="514" protocol="tcp" queue.type="linkedList")10. Verify listening port
ss -tlnp | grep 514Visual
The centralized logging architecture
┌─────────────────────────────────────────────────────────────┐
│ CENTRALIZED LOGGING │
│ │
│ client01 ──┐ │
│ │ │
│ client02 ──┼── TCP/UDP ──► logserver │
│ │ port 514 │ │
│ client03 ──┘ ▼ │
│ /var/log/remote/client01/ │
│ /var/log/remote/client02/ │
│ /var/log/remote/client03/ │
│ │
│ Each client forwards its logs. │
│ The server separates them by hostname. │
└─────────────────────────────────────────────────────────────┘Facility and priority selector
┌─────────────────────────────────────────────────────────────┐
│ SELECTOR: facility.priority │
│ │
│ authpriv.* │
│ │ │ │
│ │ └── all priorities │
│ └── authentication facility │
│ │
│ *.info │
│ │ │ │
│ │ └── info and higher (info, notice, warning, err...) │
│ └── all facilities │
│ │
│ *.info;mail.none │
│ → all facilities except mail, priority info and higher │
└─────────────────────────────────────────────────────────────┘Transport trade-offs
┌─────────────────────────────────────────────────────────────┐
│ TRANSPORT COMPARISON │
│ │
│ UDP │
│ Speed: highest │
│ Reliability: none │
│ Use: acceptable loss, legacy devices │
│ │
│ TCP │
│ Speed: moderate │
│ Reliability: connection-oriented, error handling │
│ Use: standard centralized logging, with queue │
│ │
│ RELP │
│ Speed: lowest │
│ Reliability: application-level acknowledgment │
│ Use: no message loss acceptable │
└─────────────────────────────────────────────────────────────┘Template path construction
┌─────────────────────────────────────────────────────────────┐
│ TEMPLATE: RemoteLogs │
│ │
│ constant("/var/log/remote/") │
│ property(hostname, securePath="replace") │
│ constant("/") │
│ property(programname, securePath="replace") │
│ constant(".log") │
│ │
│ Message from hostname=web01, programname=sshd: │
│ /var/log/remote/web01/sshd.log │
│ │
│ Message from hostname=db01, programname=postgres: │
│ /var/log/remote/db01/postgres.log │
└─────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Daemon | rsyslogd |
| Client forward module | omfwd |
| Server receive modules | imtcp, imudp |
| Default port | 514 |
| Reliable transport | RELP (port 6514 typical) |
| Message classification | Facility + priority |
| Dynamic path mechanism | Templates |
| Path traversal protection | securePath="replace" |
| Site configuration location | /etc/rsyslog.d/ |
| Forwarding queue (TCP) | queue.type="linkedList" |
Key takeaways:
- Centralized logging uses the same daemon on both sides.
rsyslogdis a client when configured withomfwdand a server when configured withimtcporimudp. The roles are configuration, not different software. - Transport choice is a reliability decision. UDP has no delivery guarantee. TCP adds connection-oriented reliability, and the
rsyslogdocumentation recommends pairing it with a linked-list queue so that a collector outage does not block local logging . RELP adds application-level acknowledgments. - Facilities and priorities determine what gets forwarded. A client can send only security-relevant messages or only errors, rather than the entire log stream, by using selectors.
- Templates are mandatory for multi-host collectors. Without a per-host template, all remote logs mix into one file, which defeats the purpose of centralization.
securePath="replace"is a security requirement. A client-controlled hostname used in a file path without sanitization allows path traversal. Thereplacemode sanitizes the value before it reaches the filesystem, and Red Hat’s own documentation uses it in template examples .- Site rules belong in
/etc/rsyslog.d/. Editing the mainrsyslog.confdirectly risks conflicts with package upgrades. Drop-in files are the supported extension mechanism, andrsyslogreads them in lexical order .
Remember: Centralized logging is not a separate product. It is the same syslog daemon, configured to send and receive over the network. The architecture is a client that forwards and a server that receives, connected by a transport that trades reliability against overhead. The server’s rules determine where the incoming messages land, and the template determines how they are organized. The three decisions that matter most are the transport (UDP, TCP, or RELP), the selector (what to forward), and the template (where to write it). Get those right, and the collector becomes a searchable, tamper-resistant record of the fleet’s behavior.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!