LFCS 130 🐧 journalctl — Filtering and Persistence - KandZ

⏱️ Estimated read: 9 min | |

The previous chapter introduced the journalctl command and its basic filtering options. This chapter goes deeper into the filtering syntax and the persistence configuration. The LFCS exam tests the _FIELD matching, the --grep option, the -o output format, the boot-time filtering, and the journald.conf settings that control whether logs survive a reboot. The critical distinction is between filtering by a syslog identifier (the -t flag or the SYSLOG_IDENTIFIER field) and filtering by a systemd unit (the -u flag). A service may log under a different identifier than its unit name, and a single unit may spawn processes that log under multiple identifiers. The --grep option searches the message text, not the metadata, and it is a regex match, not a literal string match.

Key point: The journal indexes every entry by structured fields. The FIELD=value syntax filters by any of these fields, not just the ones with dedicated flags. The -t flag is shorthand for SYSLOG_IDENTIFIER=value. The -u flag is shorthand for _SYSTEMD_UNIT=value. The --grep flag searches the message body. The -o flag controls the output format: short, json, cat, verbose, export. The persistence is controlled by the Storage= setting in journald.conf and the existence of the /var/log/journal/ directory.

Why advanced filtering and persistence require explicit configuration

The field problem. Every journal entry has metadata fields that are not displayed in the default output. The _PID, _UID, _GID, _COMM, _EXE, _CMDLINE, _HOSTNAME, _SYSTEMD_UNIT, SYSLOG_IDENTIFIER, and PRIORITY fields are indexed and queryable. The journalctl _COMM=sshd filters by the process name. The journalctl _PID=1234 filters by the PID. The journalctl _UID=1000 filters by the user ID. These fields are not exposed through dedicated flags; the FIELD=value syntax is the only way to query them .

The identifier problem. A service is a systemd unit. The unit name is sshd.service. The service process may set its syslog identifier to sshd or to a different name. The -u sshd.service filters by the unit. The -t sshd filters by the syslog identifier. The two are not interchangeable. A service that logs under a different identifier will not appear with -u, and a service that spawns multiple processes will show multiple identifiers with -t.

The output format problem. The default output is the short format: timestamp, hostname, identifier, PID, and message. The cat format shows only the message, without metadata. The json format outputs one JSON object per line, which is parseable by jq. The verbose format shows every field. The export format is the binary export format. The -o flag selects the format. The json and cat formats are the most useful for scripts and pipelines.

The persistence problem. The journal’s storage location determines whether logs survive a reboot. The Storage=auto setting uses the persistent location if /var/log/journal/ exists, and the volatile location otherwise. The default on most distributions is auto without the persistent directory, which means the journal is volatile. The administrator must create the directory or set Storage=persistent to enable persistence. The systemd-tmpfiles --create --prefix /var/log/journal command creates the directory with the correct permissions.

The trade-off. The field filtering is precise but requires knowing the field names. The --grep is simpler but searches only the message body, not the metadata. The persistent storage survives reboots but consumes disk space. The volatile storage is fast but ephemeral. The exam tests the syntax and the configuration.

a. Field filtering and the `--grep` option

The journal indexes structured fields. The FIELD=value syntax filters by any of them.

# Filter by process name
journalctl _COMM=sshd

# Filter by PID
journalctl _PID=1234

# Filter by user ID
journalctl _UID=1000

# Filter by systemd unit
journalctl _SYSTEMD_UNIT=sshd.service

# Filter by syslog identifier
journalctl SYSLOG_IDENTIFIER=sshd

The -t flag is shorthand for SYSLOG_IDENTIFIER=. The -u flag is shorthand for _SYSTEMD_UNIT=. The -p flag is shorthand for PRIORITY<=.

# These two are equivalent
journalctl -t sshd
journalctl SYSLOG_IDENTIFIER=sshd

# These two are equivalent
journalctl -u sshd.service
journalctl _SYSTEMD_UNIT=sshd.service

The --grep option searches the message body. It is a regular expression match.

# Search for messages containing "failed"
journalctl --grep "failed"

# Search for messages matching a regex
journalctl --grep "error.*timeout"

The --case-sensitive flag (or --case-sensitive=yes) enables case-sensitive matching. The default is case-insensitive. The --grep can be combined with other filters.

# Failed messages from sshd
journalctl -u sshd.service --grep "failed"

b. The `-o` output format

The -o flag controls the output format. The default is short.

# Short format (default)
journalctl -o short

# Cat format (message only)
journalctl -o cat

# JSON format (one object per line)
journalctl -o json

# Verbose format (all fields)
journalctl -o verbose

The cat format is useful when the message text is the only thing that matters. The json format is useful when the output is piped to jq for structured parsing.

# Extract the message and PID from each entry
journalctl -o json | jq -r '.MESSAGE + " " + ._PID'

The verbose format shows every field, which is useful for understanding what metadata is available for filtering. The export format is the binary export format used for transferring entries between systems.

c. Persistence and disk management

The persistence is controlled by the Storage= setting and the existence of the /var/log/journal/ directory.

# Check the current storage location
ls /var/log/journal/
# If the directory exists, persistent storage is active.

# Enable persistent storage
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

The journald.conf file controls the storage behavior.

# /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=2G
SystemMaxFileSize=50M

The Storage=persistent makes the journal write to /var/log/journal/. The SystemMaxUse limits the total disk usage. The SystemKeepFree leaves free space on the filesystem. The SystemMaxFileSize limits the size of individual journal files.

The journalctl --disk-usage command shows the current usage. The journalctl --vacuum-size=500M command removes the oldest entries to bring the usage under the limit. The journalctl --vacuum-time=7d removes entries older than seven days.

# Show current usage
journalctl --disk-usage

# Remove old entries
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d

The journalctl --verify command checks the integrity of the journal files. The journalctl --flush command flushes the volatile entries to the persistent storage.

Complete Example Session

# ============================================
# PART 1: FIELD FILTER
# ============================================

journalctl _COMM=sshd
# Shows entries from the sshd process.

# ============================================
# PART 2: IDENTIFIER FILTER
# ============================================

journalctl -t sshd
# Shows entries with SYSLOG_IDENTIFIER=sshd.

# ============================================
# PART 3: UNIT FILTER
# ============================================

journalctl -u sshd.service
# Shows entries from the sshd.service unit.

# ============================================
# PART 4: GREP
# ============================================

journalctl --grep "failed"
# Shows messages containing "failed".

# ============================================
# PART 5: OUTPUT FORMAT
# ============================================

journalctl -o cat -n 5
# Shows the 5 most recent messages, without metadata.

# ============================================
# PART 6: JSON OUTPUT
# ============================================

journalctl -o json -n 1 | jq -r '.MESSAGE'
# Extracts the message from the most recent entry.

# ============================================
# PART 7: PERSISTENT STORAGE
# ============================================

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

# ============================================
# PART 8: CONFIGURE LIMITS
# ============================================

# /etc/systemd/journald.conf
# [Journal]
# Storage=persistent
# SystemMaxUse=1G
# SystemKeepFree=2G

# ============================================
# PART 9: DISK MANAGEMENT
# ============================================

journalctl --disk-usage
sudo journalctl --vacuum-size=500M

The nine parts trace the advanced journalctl workflow: field filter, identifier filter, unit filter, grep, output format, JSON output, persistent storage, configuration limits, and disk management.

Quick Reference

Field Filters

Field	Purpose
`_COMM`	Process name
`_PID`	Process ID
`_UID`	User ID
`_SYSTEMD_UNIT`	Systemd unit
`SYSLOG_IDENTIFIER`	Syslog tag
`PRIORITY`	Priority level
`_HOSTNAME`	Hostname

Flag Equivalents

Flag	Field Filter
`-t <id>`	`SYSLOG_IDENTIFIER=<id>`
`-u <unit>`	`_SYSTEMD_UNIT=<unit>`
`-p <level>`	`PRIORITY<=<level>`

Output Formats

Format	Description
`short`	Default
`cat`	Message only
`json`	JSON per entry
`verbose`	All fields
`export`	Binary export

Persistence Settings

Setting	Purpose
`Storage=`	`volatile`, `persistent`, `auto`, `none`
`SystemMaxUse=`	Total disk limit
`SystemKeepFree=`	Filesystem free space
`SystemMaxFileSize=`	Individual file size

Disk Management

Command	Purpose
`journalctl --disk-usage`	Show usage
`journalctl --vacuum-size=N`	Reduce to N
`journalctl --vacuum-time=N`	Remove older than N
`journalctl --verify`	Check integrity

Best Practices

✅ Do This:

journalctl _COMM=sshd --grep "failed"
<!-- Combine field filter and grep -->                # ✅

journalctl -o json | jq -r '.MESSAGE'
<!-- Parse output as JSON -->                         # ✅

Storage=persistent
<!-- Enable persistence explicitly -->                # ✅

❌ Don’t Do This:

# Assume -u and -t are interchangeable
<!-- Unit name != syslog identifier -->              # ❌

# Grep the raw journal file
<!-- The file is binary -->                          # ❌

# Ignore disk usage
<!-- The journal can grow indefinitely -->            # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
`-u` finds no logs	Wrong unit name	Check `systemctl list-units`
`-t` finds no logs	Wrong syslog identifier	Use `_COMM` or check the process
JSON output invalid	Entries span multiple lines	Use `jq -c` or `--output-fields`
Logs lost after reboot	Storage not persistent	Enable `/var/log/journal/`
Disk full	No `SystemMaxUse`	Set limit and vacuum

Real-World Examples

1. Field Filter

journalctl _COMM=sshd

2. Identifier Filter

journalctl -t sshd

3. Unit Filter

journalctl -u sshd.service

4. Grep

journalctl --grep "failed"

5. Cat Format

journalctl -o cat -n 5

6. JSON Format

journalctl -o json | jq -r '.MESSAGE'

7. Persistent Storage

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

8. Limit Disk

SystemMaxUse=1G

9. Vacuum

sudo journalctl --vacuum-size=500M

10. Verify

sudo journalctl --verify

Visual

Field vs Flag

┌──────────────────────────────────────────────┐
│  journalctl -u sshd.service                  │
│  → _SYSTEMD_UNIT=sshd.service                │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  journalctl -t sshd                          │
│  → SYSLOG_IDENTIFIER=sshd                    │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  journalctl _COMM=sshd                       │
│  → Process name is sshd                      │
└──────────────────────────────────────────────┘

Persistence Decision

┌──────────────────────────────────────────────┐
│  /var/log/journal/ exists?                   │
│  ├─ YES → Persistent storage                 │
│  └─ NO  → Check Storage= setting             │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│  Storage=persistent → /var/log/journal/      │
│  Storage=volatile   → /run/log/journal/      │
│  Storage=auto       → persistent if dir exists│
│  Storage=none       → discarded              │
└──────────────────────────────────────────────┘

Summary

Item	Value
Field filter	`FIELD=value`
Unit filter	`-u <unit>` or `_SYSTEMD_UNIT=`
Identifier filter	`-t <id>` or `SYSLOG_IDENTIFIER=`
Grep	`--grep <pattern>`
Output format	`-o short\|cat\|json\|verbose`
Persistent storage	`/var/log/journal/`
Config file	`/etc/systemd/journald.conf`
Disk limit	`SystemMaxUse=`
Vacuum	`journalctl --vacuum-size=`

Key takeaways:

The field syntax is the general filter. Every structured field is queryable with FIELD=value. The -u and -t flags are shorthands for _SYSTEMD_UNIT and SYSLOG_IDENTIFIER, but they are not interchangeable.
--grep searches the message body. It is a regex match, not a metadata filter. The -u and -t filters restrict the set of entries; the --grep filters within that set.
The -o flag controls the output format. The cat format is message-only. The json format is one JSON object per line. The verbose format shows every field.
Persistence requires explicit configuration. The /var/log/journal/ directory must exist, or Storage=persistent must be set. The default auto uses persistent storage only if the directory exists.
Disk management is mandatory. The journal grows indefinitely without a limit. The SystemMaxUse setting and the journalctl --vacuum-size command are the tools for controlling the usage.

Remember: The journal is the structured log store. The journalctl is the query engine. The FIELD=value is the filter. The -o is the format. The --grep is the message search. The Storage= is the persistence. The exam tests the field syntax and the persistence configuration.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!