LFCA 64 🐧 Reading Logs with journalctl - KandZ

The systemd journal is a binary log system that stores messages from the kernel, system services, and applications in a structured format. Unlike text logs that you read with cat or tail, the journal requires a dedicated tool: journalctl. This command is the primary interface for querying systemd logs, and it provides filtering capabilities that grep cannot match — filtering by service, by boot, by priority, and by time.

The LFCA exam places log management under System Administration Fundamentals, which carries 20–30% of the total weight. The study plan explicitly lists journalctl alongside /var/log as a key topic. Knowing that logs exist is not enough. Knowing how to query them efficiently is the skill the exam tests.

Key point: journalctl queries the systemd journal, which is stored in a binary format at /var/log/journal/ (persistent) or /run/log/journal/ (volatile). The journal captures messages from the kernel, from systemd units, and from applications that use the journal API. Every log entry has structured metadata — the unit that generated it, the process ID, the user, the priority — and journalctl can filter on any of these fields .

Why journalctl exists

Before systemd, Linux logging was text-based. rsyslog wrote messages to files under /var/log/, and you read them with grep, tail, and less. This worked, but it had limitations that became more apparent as systems grew more complex.

The structure problem. A text log line contains a timestamp, a hostname, a process name, and a message. It does not contain structured fields like the systemd unit name, the process ID, the user ID, or the executable path. If you wanted to find all messages from a specific service, you had to grep for the process name — which might appear in unrelated lines. The journal stores these fields explicitly, so filtering by unit is exact and reliable .

The rotation problem. Text logs rotate. When syslog rotates to syslog.1, and syslog.1 rotates to syslog.2.gz, finding a message from three weeks ago means searching through multiple files. The journal treats all entries as a single stream, regardless of how the underlying files are stored. You query the journal, not the files .

The boot problem. After a reboot, text logs from the previous boot may be compressed or gone. The journal can preserve entries from multiple boots, and journalctl -b -1 shows logs from the previous boot without any file manipulation .

The priority problem. Text logs have a priority field in syslog format, but filtering by priority with grep is fragile. The journal stores priority as a structured field, and journalctl -p err reliably shows only error-level messages and higher .

The trade-off. The journal is binary. You cannot read it with cat. You cannot copy a journal file to another machine and read it with less. You need journalctl on the same system, or you need to export the journal in a portable format. This is the cost of structured logging .

a. Basic journalctl Usage

When you run journalctl without arguments, it displays all journal entries from the oldest to the newest, paged through less . This is rarely what you want. The journal contains thousands of entries, and scrolling from the beginning is impractical. The first thing to learn is how to limit the output.

The -n option limits the number of lines shown. journalctl -n 20 shows the most recent 20 entries . The -f option follows the journal in real time, like tail -f. journalctl -f shows the most recent entries and continues to print new ones as they are written. Press Ctrl+C to stop .

The -r option reverses the order, showing the newest entries first. journalctl -r -n 50 shows the 50 most recent entries in reverse chronological order .

# Show the last 20 entries
journalctl -n 20

# Follow the journal in real time
journalctl -f

# Show the 50 most recent entries, newest first
journalctl -r -n 50

# Show all entries, paged
journalctl

The --no-pager option disables the pager and prints directly to the terminal. This is useful when piping output to another command or when you want to see the output without the less interface .

# Print without paging
journalctl --no-pager | head -n 20

b. Filtering by Time, Boot, and Service

The journal’s structured metadata makes filtering precise. The most common filters are by time, by boot, and by systemd unit.

The --since and --until options filter by time. They accept absolute timestamps ("2025-01-15 18:10:20"), relative times ("1 hour ago"), and special strings like today, yesterday, and now .

# Entries from today
journalctl --since today

# Entries from the last hour
journalctl --since "1 hour ago"

# Entries between two timestamps
journalctl --since "2025-01-15 18:00:00" --until "2025-01-15 19:00:00"

The -b option filters by boot. journalctl -b shows entries from the current boot. journalctl -b -1 shows entries from the previous boot. The --list-boots option lists all available boots with their IDs and timestamps .

# Current boot
journalctl -b

# Previous boot
journalctl -b -1

# List all boots
journalctl --list-boots

The -u option filters by systemd unit. journalctl -u nginx.service shows all entries from the Nginx service. This is the single most useful filter for service troubleshooting .

# Logs from a specific service
journalctl -u ssh.service

# Logs from a service since a specific time
journalctl -u nginx.service --since "1 hour ago"

The -t option filters by syslog identifier. journalctl -t sudo shows messages from the sudo command .

# Logs from sudo
journalctl -t sudo

c. Filtering by Priority and Using the Catalog

Every journal entry has a priority level, inherited from the syslog standard. The levels range from emerg (0, most severe) to debug (7, least severe). The -p option filters by priority, showing entries at that level and higher .

# Errors and above
journalctl -p err

# Critical and above
journalctl -p crit

# Warnings and above
journalctl -p warning

The priority levels are: emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7). A filter of -p err shows levels 0 through 3 .

The -x option adds explanatory text from the message catalog. For many common errors, the systemd project provides a short explanation of what the error means and how to resolve it. The catalog text is displayed after the log entry .

# Show errors with catalog explanations
journalctl -p err -x

The --grep option searches for a string or regular expression within the message text. If the string is all lowercase, the search is case-insensitive by default. Use --case-sensitive to override .

# Search for a pattern in messages
journalctl --grep "failed password"

# Case-sensitive search
journalctl --grep "Failed" --case-sensitive

Complete Example Session

This session demonstrates the most common journalctl queries for troubleshooting and investigation.

# ============================================
# PART 1: BASIC VIEWING
# ============================================

# Show the most recent 20 entries
journalctl -n 20

# Follow the journal in real time
journalctl -f

# ============================================
# PART 2: FILTERING BY SERVICE
# ============================================

# SSH service logs
journalctl -u ssh.service

# Nginx service logs since one hour ago
journalctl -u nginx.service --since "1 hour ago"

# ============================================
# PART 3: FILTERING BY TIME
# ============================================

# Entries from today
journalctl --since today

# Entries from the last 30 minutes
journalctl --since "30 minutes ago"

# Entries between two times
journalctl --since "2025-01-15 18:00" --until "2025-01-15 19:00"

# ============================================
# PART 4: FILTERING BY BOOT
# ============================================

# Current boot
journalctl -b

# Previous boot
journalctl -b -1

# List all available boots
journalctl --list-boots

# ============================================
# PART 5: FILTERING BY PRIORITY
# ============================================

# Errors and above
journalctl -p err

# Critical and above
journalctl -p crit

# Errors from the last hour
journalctl -p err --since "1 hour ago"

# ============================================
# PART 6: USING THE CATALOG
# ============================================

# Errors with explanations
journalctl -p err -x

# ============================================
# PART 7: SEARCHING MESSAGES
# ============================================

# Search for a pattern
journalctl --grep "failed password"

# ============================================
# PART 8: DISK USAGE AND MANAGEMENT
# ============================================

# Check how much space the journal uses
journalctl --disk-usage

# Rotate the journal
sudo journalctl --rotate

# Vacuum to a size limit
sudo journalctl --vacuum-size=500M

# Vacuum to a time limit
sudo journalctl --vacuum-time=2weeks

# ============================================
# PART 9: PERSISTENT STORAGE
# ============================================

# Check if persistent journal is enabled
ls /var/log/journal/ 2>/dev/null

# Check the configured journal path
journalctl -F JOURNAL_PATH

# ============================================
# PART 10: COMBINED FILTERS
# ============================================

# Errors from SSH in the last hour
journalctl -u ssh.service -p err --since "1 hour ago"

# All logs from today with explanations
journalctl --since today -x

# Critical errors from the current boot
journalctl -b -p crit

The ten parts cover basic viewing, filtering by service, filtering by time, filtering by boot, filtering by priority, using the catalog, searching messages, disk usage and management, persistent storage, and combined filters.

Quick Reference

The Basic Options

Option	Purpose
`-n N`	Show the last N entries
`-f`	Follow the journal in real time
`-r`	Reverse order (newest first)
`--no-pager`	Print without paging

The Filtering Options

Option	Filters By
`-u UNIT`	Systemd unit (service)
`-b`	Current boot
`-b -1`	Previous boot
`--since TIME`	Entries after TIME
`--until TIME`	Entries before TIME
`-p LEVEL`	Priority and above
`-t IDENT`	Syslog identifier
`--grep PATTERN`	Message text

The Priority Levels

Level	Name	Meaning
0	`emerg`	System is unusable
1	`alert`	Action must be taken immediately
2	`crit`	Critical conditions
3	`err`	Error conditions
4	`warning`	Warning conditions
5	`notice`	Normal but significant
6	`info`	Informational
7	`debug`	Debug-level messages

The Management Options

Option	Purpose
`--disk-usage`	Show journal disk usage
`--rotate`	Force journal rotation
`--vacuum-size=SIZE`	Limit journal to SIZE
`--vacuum-time=TIME`	Keep journal for TIME
`--flush`	Flush volatile to persistent

The Persistent Storage

Directory	Type
`/var/log/journal/`	Persistent (survives reboot)
`/run/log/journal/`	Volatile (lost on reboot)

Best Practices

✅ Do This:

# Filter by service for troubleshooting
journalctl -u nginx.service --since "1 hour ago"              # ✅

# Use relative time specifications
journalctl --since "30 minutes ago"                           # ✅

# Filter by priority for errors
journalctl -p err --since today                               # ✅

# Use -x to get catalog explanations
journalctl -p err -x                                          # ✅

# Check disk usage and vacuum when needed
journalctl --disk-usage                                       # ✅

❌ Don’t Do This:

# Don't scroll from the beginning
journalctl                                                    # ❌ too much output

# Don't grep the binary journal directly
grep "error" /var/log/journal/*                               # ❌ binary data

# Don't forget -b for boot-specific investigation
journalctl -u service                                         # ❌ mixes boots

# Don't assume logs persist across reboots
# Check for /var/log/journal/ first                          # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Journal empty after reboot	No persistent storage	Create `/var/log/journal/` or set `Storage=persistent`
Cannot filter by boot	Persistent storage not enabled	Enable persistent journal first
Too many lines	No filter applied	Use `-n`, `--since`, or `-u`
Permission denied	Not root or not in `adm`/`systemd-journal` group	Use `sudo` or add user to group
Binary garbage from grep	Grep on binary journal files	Use `journalctl --grep` instead

Real-World Examples

1. Follow SSH Logs

journalctl -u ssh.service -f

2. Errors Today

journalctl -p err --since today

3. Previous Boot Logs

journalctl -b -1

4. Service Logs Since an Hour Ago

journalctl -u nginx.service --since "1 hour ago"

5. Search for Failed Passwords

journalctl --grep "failed password"

6. Critical Errors with Explanations

journalctl -p crit -x

7. Check Journal Disk Usage

journalctl --disk-usage

8. Vacuum to Size

sudo journalctl --vacuum-size=500M

9. List All Boots

journalctl --list-boots

10. Combined Filter

journalctl -u ssh.service -p err --since "1 hour ago"

Visual

The journalctl Filter Combination

┌──────────────────────────────────────────────┐
│  FILTERS COMBINE WITH AND                    │
│                                              │
│  journalctl -u ssh.service -p err            │
│       --since "1 hour ago"                   │
│                                              │
│  Results: entries that satisfy ALL:          │
│    ├─ unit = ssh.service                     │
│    ├─ priority <= err (0-3)                  │
│    └─ time >= 1 hour ago                     │
│                                              │
│  Each additional filter narrows the results. │
│                                              │
└──────────────────────────────────────────────┘

The Priority Filter

┌──────────────────────────────────────────────┐
│  PRIORITY FILTER                             │
│                                              │
│  -p err shows levels:                        │
│    0 emerg                                   │
│    1 alert                                   │
│    2 crit                                    │
│    3 err                                     │
│                                              │
│  It does NOT show:                           │
│    4 warning                                 │
│    5 notice                                  │
│    6 info                                    │
│    7 debug                                   │
│                                              │
│  Lower number = more severe.                 │
│                                              │
└──────────────────────────────────────────────┘

The Boot Filter

┌──────────────────────────────────────────────┐
│  BOOT FILTER                                 │
│                                              │
│  journalctl -b       → current boot          │
│  journalctl -b -1    → previous boot         │
│  journalctl -b -2    → two boots ago         │
│                                              │
│  Requires persistent storage.                │
│  Without /var/log/journal/, only the         │
│  current boot is available.                  │
│                                              │
│  List boots: journalctl --list-boots         │
│                                              │
└──────────────────────────────────────────────┘

Persistent vs Volatile Storage

┌──────────────────────────────────────────────┐
│  PERSISTENT vs VOLATILE                      │
│                                              │
│  /var/log/journal/                           │
│    ├─ Survives reboot                        │
│    ├─ Supports -b -1, -b -2                  │
│    └─ Created with:                          │
│         sudo mkdir -p /var/log/journal       │
│         sudo systemd-tmpfiles --create       │
│           --prefix /var/log/journal          │
│                                              │
│  /run/log/journal/                           │
│    ├─ Lost on reboot                         │
│    ├─ Default when no persistent dir         │
│    └─ Only current boot available            │
│                                              │
└──────────────────────────────────────────────┘

Summary

Item	Value
Journal tool	`journalctl`
Storage	Binary, `/var/log/journal/` (persistent) or `/run/log/journal/` (volatile)
Basic view	`journalctl -n 20`
Follow	`journalctl -f`
Service filter	`journalctl -u service`
Time filter	`journalctl --since` / `--until`
Boot filter	`journalctl -b` / `-b -1`
Priority filter	`journalctl -p err`
Search	`journalctl --grep`
Catalog	`journalctl -x`
Disk usage	`journalctl --disk-usage`
Vacuum	`journalctl --vacuum-size` / `--vacuum-time`

Key takeaways:

journalctl is the only way to read the systemd journal. The journal is stored in a binary format at /var/log/journal/ or /run/log/journal/. You cannot read it with cat or grep. The journalctl command is the dedicated interface .
The default output is too long. Always limit with -n, --since, or -u. Running journalctl without filters displays every entry from the beginning, which is impractical for troubleshooting .
Filter by service with -u. This is the single most useful filter for debugging a specific service. journalctl -u nginx.service shows only Nginx logs .
Filter by time with --since and --until. Relative times like "1 hour ago" and "today" are supported, as are absolute timestamps .
Filter by boot with -b. -b shows the current boot, -b -1 shows the previous boot. This requires persistent storage to be enabled .
Filter by priority with -p. -p err shows error-level and higher. This is the fastest way to find problems .
Use -x for catalog explanations. The systemd message catalog provides human-readable explanations for many common errors .
Manage disk usage with --disk-usage, --rotate, and --vacuum. The journal grows over time. These commands let you inspect and limit its size .

Remember: journalctl is not cat with a filter. It is a structured query tool for a structured log system. The journal stores fields — unit, priority, boot ID, PID, user — that text logs do not. Every filter you apply is a field match, not a text search. When you troubleshoot a service, start with journalctl -u service --since "1 hour ago". When you investigate an error, add -p err. When you need context, add -x. When you need to find a pattern, use --grep. The command is designed for investigation, not reading. Learn the filters, and the journal becomes the fastest diagnostic tool on the system.

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!