LFCA 63 🐧 Where Logs Live — /var/log - KandZ

The Linux filesystem has a place for everything. Binaries live in /usr/bin, configuration in /etc, user data in /home, temporary files in /tmp. Logs live in /var/log. This is not an arbitrary choice. The Filesystem Hierarchy Standard reserves /var for variable data — files whose size and content change during normal operation. Logs are the purest example of that: they grow with every event the system records .

The LFCA exam places log management under System Administration Fundamentals, which carries 20–30% of the total weight . The study plan explicitly lists /var/log as a key topic alongside journalctl . Knowing that logs live in /var/log is the first step. Knowing which file holds which kind of message — and why — is the skill the exam tests.

Key point: Almost every log file on a Linux system lives under /var/log/. Some are plain text files readable with cat and grep. Others are binary files that require specific commands. Some applications create their own subdirectories under /var/log/. And on systemd-based distributions, the journal stores logs in a separate binary location entirely. The directory is the starting point, not the whole story.

Why /var/log exists

The Linux filesystem is organized by purpose. Each top-level directory has a defined role, and /var exists specifically for data that changes at runtime.

The variable data problem. The FHS defines /var as the location for variable data files — spool directories, administrative data, and log files . A log file changes every time a message is written. Its size is unpredictable, its content is transient, and it must not be on a read-only partition. Putting logs in /usr would be wrong because /usr is meant to be shareable and read-only. Putting them in /tmp would be wrong because /tmp may be cleared on reboot. /var/log is the designated home for persistent, growing, runtime data.

The permission problem. Log files contain sensitive information — authentication attempts, system errors, user activity. They must be readable by administrators but not by regular users. /var/log is typically owned by root and syslog (or adm on some systems), with permissions that prevent unprivileged access. The Ubuntu documentation notes that /var/log/auth.log contains password prompts and sudo usage, which is why it is restricted .

The organization problem. Hundreds of processes write logs. Without a convention, each would choose its own location. /var/log provides a single directory where administrators know to look. System logs go to /var/log/syslog, authentication goes to /var/log/auth.log, kernel messages go to /var/log/kern.log . Applications that need more structure create subdirectories: /var/log/apache2/, /var/log/mysql/, /var/log/nginx/ .

The rotation problem. Logs grow without bound. If nothing managed them, they would eventually fill the disk. logrotate exists to solve this, and it assumes logs live under /var/log. The configuration files in /etc/logrotate.d/ reference paths like /var/log/syslog and /var/log/nginx/*.log . The rotation system and the directory are designed together.

The trade-off. Not all logs live under /var/log. The systemd journal stores its binary files in /var/log/journal/ or /run/log/journal/ depending on configuration. Application-specific logs may be placed elsewhere by their developers. And on some systems, logging is centralized to a remote server. /var/log is the default, not the absolute.

a. The System Logs

The system logs are the files that record the core activity of the operating system. They are the first place to check when something breaks, and they follow a consistent naming convention on Debian/Ubuntu systems that differs from Red Hat-family systems.

/var/log/syslog is the general system log on Debian and Ubuntu. It contains messages from the kernel, system daemons, and applications that do not have their own log files. The Ubuntu documentation describes it as the catch-all: “If you can’t find anything in the other logs, it’s probably here” . On Red Hat-family systems, the equivalent file is /var/log/messages .

/var/log/auth.log records authentication events. This includes password prompts, sudo command invocations, remote logins via SSH, and failed authentication attempts . On Red Hat-family systems, the equivalent is /var/log/secure . The Last9 guide identifies this as the primary file for security-related events.

/var/log/kern.log contains messages from the Linux kernel . This includes hardware detection during boot, driver messages, and kernel warnings or errors. The dmesg command shows the same messages from the kernel ring buffer, and journalctl -k shows them from the systemd journal.

/var/log/daemon.log records messages from daemons — background processes that run without user interaction. This includes the display server, SSH sessions, printing services, and Bluetooth . The name “daemon” comes from the Greek concept of a spirit that works in the background.

/var/log/debug provides debugging information from the system and applications . In practice, this file is often empty or minimally used. It exists as a designated place for debug-level messages that would otherwise flood the main logs.

/var/log/boot.log records the boot sequence — the messages that appear as each service starts during system initialization . This is useful for diagnosing boot failures. The file is overwritten on each boot, so it only shows the most recent boot sequence.

/var/log/cron records the execution of cron jobs . Every time a scheduled task runs, an entry is written here. This is how you verify that a cron job actually executed and what output it produced.

b. The Application Logs

Applications that produce significant log volume create their own subdirectories under /var/log/. This keeps their logs separate from the system logs and allows per-application rotation policies.

/var/log/apache2/ (Debian/Ubuntu) or /var/log/httpd/ (Red Hat family) contains Apache web server logs. The access.log file records every HTTP request — the client IP, the requested URL, the response code, and the user agent. The error.log records server errors . These are the two files you check when a website is not working.

/var/log/nginx/ contains Nginx web server logs, following the same pattern: access.log for requests and error.log for errors. The logrotate configuration for Nginx is typically found in /etc/logrotate.d/nginx and uses postrotate to signal Nginx to reopen its log files after rotation .

/var/log/mysql/ contains MySQL database logs. The error.log records server startup, shutdown, and errors. The slow.log (when enabled) records queries that exceed a configured duration threshold, useful for performance tuning .

/var/log/samba/ contains Samba file-sharing logs. The smbd.log records file and print service activity, and nmbd.log records NetBIOS name service activity .

/var/log/Xorg.0.log contains X11 display server logs. Each display gets its own log file — Xorg.0.log for display 0, Xorg.1.log for display 1, and so on . This is where you check when the graphical interface fails to start.

c. The Binary Logs and the Journal

Not all logs under /var/log/ are text files. Several critical files are binary and require specific commands to read.

/var/log/wtmp records every login and logout. It is read by the who command, which shows currently logged-in users, and by last, which shows login history . The file is binary because it stores structured records, not lines of text.

/var/log/btmp records failed login attempts. It is read by the lastb command . This is the file security analysts check for brute-force attacks.

/var/log/faillog contains information about failed authentication attempts per user. It is read by the faillog command .

/var/log/lastlog records the most recent login for each user. It is read by the lastlog command . This shows when each account was last accessed, which is useful for identifying dormant accounts.

The systemd journal is stored in /var/log/journal/ (persistent) or /run/log/journal/ (volatile, lost on reboot). It is a binary format with structured metadata, queried exclusively with journalctl . The journal captures messages from the kernel, systemd services, and applications that use the journal API. On many systems, journald is configured to forward messages to rsyslog, so the same event appears in both the journal and the text files.

Complete Example Session

This session demonstrates exploring /var/log, reading text logs, reading binary logs, and understanding what each file contains.

# ============================================
# PART 1: LISTING THE LOG DIRECTORY
# ============================================

# See everything in /var/log
ls -la /var/log

# See just the files, sorted by size
ls -lhS /var/log

# ============================================
# PART 2: READING THE SYSTEM LOG
# ============================================

# View recent system messages
tail -n 20 /var/log/syslog

# Search for errors
grep -i "error" /var/log/syslog

# ============================================
# PART 3: READING AUTHENTICATION LOGS
# ============================================

# View recent authentication events
tail -n 30 /var/log/auth.log

# Count failed SSH logins
grep -c "Failed password" /var/log/auth.log

# ============================================
# PART 4: READING KERNEL LOGS
# ============================================

# View kernel messages
tail -n 20 /var/log/kern.log

# Same messages via dmesg
dmesg | tail -n 20

# ============================================
# PART 5: READING BINARY LOGS
# ============================================

# Who is logged in? (reads /var/log/wtmp)
who

# Last logins for all users (reads /var/log/lastlog)
lastlog

# Failed login attempts (reads /var/log/faillog)
faillog -a

# ============================================
# PART 6: CHECKING APPLICATION LOGS
# ============================================

# Apache logs (if installed)
ls -la /var/log/apache2/
tail -n 10 /var/log/apache2/access.log

# Nginx logs (if installed)
ls -la /var/log/nginx/
tail -n 10 /var/log/nginx/error.log

# ============================================
# PART 7: THE JOURNAL
# ============================================

# Check if journal directory exists
ls -la /var/log/journal/ 2>/dev/null || echo "No persistent journal"

# View recent journal entries
journalctl --no-pager | tail -n 20

# ============================================
# PART 8: IDENTIFYING FILE TYPES
# ============================================

# Check if a file is text or binary
file /var/log/syslog        # ASCII text
file /var/log/wtmp          # data
file /var/log/auth.log      # ASCII text
file /var/log/lastlog       # data

# ============================================
# PART 9: LOG ROTATION ARTIFACTS
# ============================================

# Rotated logs have numbers or dates appended
ls /var/log/syslog*
# syslog  syslog.1  syslog.2.gz  syslog.3.gz

# The .1 file is the most recent rotation
# The .gz files are compressed older rotations

# ============================================
# PART 10: THE LOGROTATE CONFIGURATION
# ============================================

# Global configuration
cat /etc/logrotate.conf

# Per-application configurations
ls /etc/logrotate.d/

# Example: how syslog is rotated
cat /etc/logrotate.d/rsyslog

The ten parts cover listing the directory, reading system logs, reading authentication logs, reading kernel logs, reading binary logs, checking application logs, the journal, identifying file types, rotation artifacts, and the logrotate configuration.

Quick Reference

The System Log Files

File	Purpose	Distribution
`/var/log/syslog`	General system messages	Debian/Ubuntu
`/var/log/messages`	General system messages	RHEL/CentOS
`/var/log/auth.log`	Authentication events	Debian/Ubuntu
`/var/log/secure`	Authentication events	RHEL/CentOS
`/var/log/kern.log`	Kernel messages	Debian/Ubuntu
`/var/log/daemon.log`	Daemon messages	Debian/Ubuntu
`/var/log/boot.log`	Boot sequence messages	Both
`/var/log/cron`	Cron job execution	Both

The Binary Log Files

File	Command	Purpose
`/var/log/wtmp`	`who`, `last`	Login/logout records
`/var/log/btmp`	`lastb`	Failed login attempts
`/var/log/faillog`	`faillog`	Failed auth per user
`/var/log/lastlog`	`lastlog`	Last login per user

The Application Log Directories

Directory	Application	Key Files
`/var/log/apache2/`	Apache (Debian)	`access.log`, `error.log`
`/var/log/httpd/`	Apache (RHEL)	`access_log`, `error_log`
`/var/log/nginx/`	Nginx	`access.log`, `error.log`
`/var/log/mysql/`	MySQL	`error.log`, `slow.log`
`/var/log/samba/`	Samba	`smbd.log`, `nmbd.log`

The Journal Locations

Location	Type	Persistence
`/var/log/journal/`	Binary	Persistent across reboots
`/run/log/journal/`	Binary	Volatile, lost on reboot

Best Practices

✅ Do This:

# Start with /var/log when troubleshooting
ls -la /var/log/                                              # ✅

# Check the right file for the right problem
tail /var/log/auth.log    # authentication issue                # ✅
tail /var/log/kern.log    # kernel/hardware issue               # ✅

# Use the correct command for binary logs
who                       # reads /var/log/wtmp                  # ✅
lastlog                   # reads /var/log/lastlog               # ✅

# Check application subdirectories
ls /var/log/apache2/      # Apache logs                          # ✅

# Verify journal persistence
ls /var/log/journal/      # if missing, journal is volatile      # ✅

❌ Don’t Do This:

# Don't try to cat binary logs
cat /var/log/wtmp                                             # ❌ garbage

# Don't assume all logs are in /var/log
# Some applications use /opt/ or /var/lib/                      # ❌

# Don't ignore rotated logs
# syslog.1 and syslog.2.gz contain history                     # ❌

# Don't forget the journal
# journalctl has messages that text files may not                # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Cannot find a log file	Looking in wrong distribution’s location	Check `/var/log/messages` vs `/var/log/syslog`
Binary file unreadable	Using `cat` on wtmp/btmp	Use `who`, `last`, `lastb`, `faillog`
Log directory empty	Journal-only system, no rsyslog	Use `journalctl` instead
Rotated logs confusing	Numbers and .gz extensions	Check `logrotate.conf` for retention policy
Application logs missing	Application not installed	Check if service is running first

Real-World Examples

1. List the Log Directory

ls -lh /var/log/

2. Check Authentication Failures

grep "Failed password" /var/log/auth.log | tail -n 10

3. Check Kernel Errors

grep -i "error" /var/log/kern.log | tail -n 20

4. See Who Is Logged In

who

5. See Failed Login Attempts

sudo lastb | head -n 20

6. Check Last Login Per User

lastlog

7. Check Apache Access Log

tail -n 20 /var/log/apache2/access.log

8. Check Nginx Error Log

tail -n 20 /var/log/nginx/error.log

9. Check the Journal

journalctl --no-pager | tail -n 30

10. Verify Log Rotation

ls /var/log/syslog*

Visual

The /var/log Directory Layout

┌──────────────────────────────────────────────┐
│  /var/log/                                   │
│                                              │
│  SYSTEM LOGS (text):                         │
│  ├── syslog        general system messages   │
│  ├── auth.log      authentication events     │
│  ├── kern.log      kernel messages           │
│  ├── daemon.log    daemon messages           │
│  ├── boot.log      boot sequence             │
│  └── cron          cron job execution        │
│                                              │
│  BINARY LOGS:                                │
│  ├── wtmp          login records (who)       │
│  ├── btmp          failed logins (lastb)     │
│  ├── faillog       failed auth (faillog)     │
│  └── lastlog       last login (lastlog)      │
│                                              │
│  APPLICATION DIRECTORIES:                    │
│  ├── apache2/      access.log, error.log     │
│  ├── nginx/        access.log, error.log     │
│  ├── mysql/        error.log, slow.log       │
│  └── samba/        smbd.log, nmbd.log        │
│                                              │
│  JOURNAL:                                    │
│  └── journal/      binary, read with          │
│                    journalctl                │
│                                              │
└──────────────────────────────────────────────┘

Text vs Binary Files

┌──────────────────────────────────────────────┐
│  TEXT vs BINARY                              │
│                                              │
│  TEXT (read with cat, tail, grep):           │
│  ├── syslog                                  │
│  ├── auth.log                                │
│  ├── kern.log                                │
│  ├── daemon.log                              │
│  └── Application logs                        │
│                                              │
│  BINARY (read with specific commands):       │
│  ├── wtmp      → who, last                   │
│  ├── btmp      → lastb                       │
│  ├── faillog   → faillog                     │
│  ├── lastlog   → lastlog                     │
│  └── journal   → journalctl                  │
│                                              │
│  The 'file' command identifies which is which│
│                                              │
└──────────────────────────────────────────────┘

Log File by Distribution

┌──────────────────────────────────────────────┐
│  DEBIAN/UBUNTU vs RHEL/CENTOS                │
│                                              │
│  General system:                             │
│    Debian: /var/log/syslog                   │
│    RHEL:   /var/log/messages                 │
│                                              │
│  Authentication:                             │
│    Debian: /var/log/auth.log                 │
│    RHEL:   /var/log/secure                   │
│                                              │
│  Web server:                                 │
│    Debian: /var/log/apache2/                 │
│    RHEL:   /var/log/httpd/                   │
│                                              │
│  The content is similar; the paths differ.   │
│                                              │
└──────────────────────────────────────────────┘

Rotated Log Naming

┌──────────────────────────────────────────────┐
│  LOG ROTATION ARTIFACTS                      │
│                                              │
│  /var/log/syslog        ← current log        │
│  /var/log/syslog.1      ← most recent        │
│  /var/log/syslog.2.gz   ← older, compressed  │
│  /var/log/syslog.3.gz   ← oldest kept        │
│                                              │
│  Rotation controlled by:                     │
│  /etc/logrotate.conf                         │
│  /etc/logrotate.d/*                          │
│                                              │
│  Default: weekly, keep 4 weeks               │
│  Configurable: daily, weekly, monthly, size  │
│                                              │
└──────────────────────────────────────────────┘

Summary

Item	Value
Log directory	`/var/log/`
General system log	`syslog` (Debian), `messages` (RHEL)
Authentication log	`auth.log` (Debian), `secure` (RHEL)
Kernel log	`kern.log`, `dmesg`
Binary login logs	`wtmp`, `btmp`, `faillog`, `lastlog`
Application logs	Subdirectories under `/var/log/`
Journal	`/var/log/journal/`, read with `journalctl`
Rotation config	`/etc/logrotate.conf`, `/etc/logrotate.d/`
FHS purpose	Variable data — logs change at runtime

Key takeaways:

/var/log is the designated home for log files under the Filesystem Hierarchy Standard. /var holds variable data — files that change during normal operation. Logs are the canonical example .
System logs follow a naming convention that differs by distribution. Debian/Ubuntu uses syslog and auth.log. Red Hat-family uses messages and secure. The content is similar; the paths differ .
Binary logs require specific commands. /var/log/wtmp is read with who and last. /var/log/btmp is read with lastb. /var/log/faillog is read with faillog. /var/log/lastlog is read with lastlog .
Applications create subdirectories under /var/log/. Apache, Nginx, MySQL, and Samba each have their own directory with access.log and error.log (or equivalent) files .
The systemd journal lives in /var/log/journal/ or /run/log/journal/. It is binary, structured, and read exclusively with journalctl. Persistent journals survive reboots; volatile journals do not .
Log rotation is configured in /etc/logrotate.conf and /etc/logrotate.d/. Rotated logs gain numeric suffixes (.1, .2.gz) and are compressed and eventually deleted according to the configured retention policy .
The file command identifies whether a log is text or binary. Use it when cat produces garbage instead of readable output.

Remember: /var/log is where logs live. System logs, authentication logs, kernel logs, and application logs all find their home under this directory. Some are text files you can read with tail and grep. Others are binary files that require who, last, lastb, or journalctl. Knowing the directory is the first step. Knowing which file holds which message — and which command reads it — is the skill. When you troubleshoot, start in /var/log. When you investigate a security event, start in /var/log/auth.log or /var/log/secure. When you need structured, filterable logs, use the journal. The directory is not just a location; it is the organized memory of the entire system.

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!