LFCA 116 🐧 Troubleshooting — A Methodical Approach
Troubleshooting is not guessing. It is not randomly restarting services or rebooting the machine in the hope that something changes. A methodical approach to troubleshooting is a repeatable process: gather information, form a hypothesis, test it, and either confirm or discard it. The difference between a technician who solves problems and one who creates new ones is discipline.
This chapter covers troubleshooting as a system administration competency within the LFCA System Administration Fundamentals domain. You will learn the phases of a structured investigation: initial assessment, resource analysis, process investigation, log analysis, network diagnostics, service troubleshooting, and resolution. Each phase has specific commands and specific questions it answers. The goal is not to memorize every tool but to internalize the order in which questions should be asked.
By the end, you will understand why checking logs comes before restarting services, why documenting findings matters even when the fix seems obvious, and how to avoid the trap of fixing symptoms instead of causes.
Key point: Troubleshooting is a process of elimination, not a process of inspiration. Start broad, narrow systematically, and never change more than one thing at a time.
Why a methodical approach exists
The random-fix problem. When a service fails, the instinct is to try the first solution that comes to mind. Restart the service. Reboot the server. Reinstall the package. Sometimes this works. More often, it masks the problem temporarily, and when the failure recurs, the technician has no idea what actually changed or why the fix worked. A methodical approach eliminates this uncertainty by ensuring every action is based on evidence, not guesswork.
The complexity problem. A modern Linux system has hundreds of interacting components: kernel, systemd, network stack, filesystem, application services, security modules. A single symptom can have dozens of possible causes. Without a structured investigation, the search space is impossibly large. The methodical approach narrows that space layer by layer, from general system health to specific component behavior.
The documentation problem. The LFCA exam tests troubleshooting as scenario-based reasoning, not just command recognition . You will be given a symptom and asked to identify the correct diagnostic step. This requires understanding why certain checks come before others. Checking network connectivity before checking whether the network interface is up is backwards. Checking disk space after investigating application logs is inefficient. The order matters, and the exam tests that understanding.
The production problem. On a production system, every action has consequences. Restarting a service disrupts users. Rebooting a server takes down applications. Running untested commands as root can cause data loss. A methodical approach minimizes disruption by requiring evidence before action. You do not restart a service until you have identified the reason it failed, because restarting without understanding means the failure will recur.
The prevention problem. The final phase of troubleshooting is not “the problem is fixed.” It is “the problem is understood, documented, and prevented from recurring.” A methodical approach produces documentation that feeds into monitoring, alerting, and capacity planning. It turns a one-time incident into institutional knowledge.
a. Initial assessment and information gathering
The first phase is not about fixing anything. It is about understanding the problem. Before you run a single diagnostic command, you need answers to basic questions: What is the symptom? When did it start? What changed recently? Who is affected?
The commands in this phase are about orientation. uptime tells you how long the system has been running and what the load average looks like. hostnamectl identifies the operating system and kernel version. dmesg | tail -50 shows the most recent kernel messages, which often contain hardware errors, driver failures, or boot-time warnings .
uptime
hostnamectl
cat /etc/os-release
dmesg | tail -50These commands do not diagnose the problem. They establish the baseline. A load average of 12 on a 4-core system is a signal. A kernel version that is unexpectedly old is a signal. A dmesg output full of ATA errors is a signal. You are collecting evidence before forming a hypothesis.
The output of these commands should be documented. If the problem requires escalation to a colleague or a support ticket, this is the information they will ask for first.
b. Resource analysis
Once you know what the system is and how long it has been running, the next question is whether the system has the resources it needs. CPU, memory, disk space, and I/O are the four fundamental resources. A deficiency in any one of them can manifest as application slowness, service failures, or unresponsive behavior .
top -bn1 | head -20
free -h
df -h
iostat -x 1 5top -bn1 provides a snapshot of CPU usage and the most active processes. free -h shows memory availability in human-readable units. df -h shows disk space on all mounted filesystems. iostat -x 1 5 shows disk I/O statistics over a five-second interval .
The key insight is that resource problems often present as application problems. A web server that refuses connections might be out of file descriptors, not broken. A database that runs slowly might be waiting on disk I/O, not misconfigured. Checking resources before checking application logs prevents wasted effort in the wrong layer of the stack.
A common pitfall is checking only one resource. Memory exhaustion can cause the kernel to kill processes. Disk exhaustion can prevent services from writing logs or temporary files. High I/O wait can make a system appear unresponsive even when CPU utilization is low. The four checks together provide a complete picture.
c. Process investigation, log analysis, and network diagnostics
If resources are adequate but the problem persists, the investigation moves to processes, logs, and network. These three areas overlap, and the order depends on the symptom. A service that will not start suggests process investigation. An application that throws errors suggests log analysis. A connectivity failure suggests network diagnostics.
Process investigation begins with identifying what is running and what is consuming resources.
ps aux --sort=-%cpu | head -10
pstree -p
lsof -p PID
strace -p PIDps aux --sort=-%cpu shows the top CPU consumers. pstree -p shows process parent-child relationships, which reveals whether a service is managed by systemd or spawned manually. lsof -p PID lists all files and network connections opened by a process. strace -p PID traces system calls, which is useful for debugging a process that hangs or fails silently .
Log analysis is the most information-dense phase of troubleshooting. Linux maintains logs in /var/log/ and through systemd’s journal . The most important files are /var/log/syslog or /var/log/messages (general system messages), /var/log/auth.log (authentication events), and /var/log/kern.log (kernel messages) . For systemd-managed services, journalctl -u service_name provides service-specific logs .
journalctl -xe
tail -f /var/log/syslog
grep -i error /var/log/*journalctl -xe shows recent systemd journal entries with explanations. tail -f follows a log file in real time, which is useful for watching a problem reproduce. grep -i error searches across log files for error messages .
A critical principle in log analysis is correlation. A single error message in isolation may be a red herring. The same error repeated across multiple services at the same timestamp suggests a shared cause—a disk failure, a network outage, or a resource exhaustion event. The timeline of events is often more informative than any individual message.
Network diagnostics follow a layer-by-layer approach . The first question is whether the network interface is up. The second is whether the system can reach its gateway. The third is whether DNS resolution works. The fourth is whether the target service is reachable.
ip addr show
ss -tulpn
curl -v http://target
dig domainip addr show displays interface configuration. ss -tulpn shows listening ports and active connections. curl -v tests HTTP connectivity with verbose output. dig tests DNS resolution .
The layering matters. Testing DNS before confirming the interface is up is wasted effort. Testing HTTP connectivity before confirming DNS works is wasted effort. Each test builds on the previous one.
Complete Example Session
# ============================================
# PART 1: INITIAL ASSESSMENT
# ============================================
uptime
hostnamectl
dmesg | tail -20# ============================================
# PART 2: RESOURCE CHECK
# ============================================
free -h
df -h
top -bn1 | head -15# ============================================
# PART 3: PROCESS INSPECTION
# ============================================
ps aux --sort=-%mem | head -10
systemctl status nginx# ============================================
# PART 4: LOG EXAMINATION
# ============================================
journalctl -u nginx --since "1 hour ago"
tail -20 /var/log/nginx/error.log# ============================================
# PART 5: NETWORK VERIFICATION
# ============================================
ip addr show
ss -tulpn | grep :80
curl -v http://localhost# ============================================
# PART 6: CONFIGURATION CHECK
# ============================================
nginx -t
ls -la /etc/nginx/sites-enabled/# ============================================
# PART 7: DEPENDENCY VERIFICATION
# ============================================
systemctl list-dependencies nginx
ldd $(which nginx)# ============================================
# PART 8: SECURITY LAYER CHECK
# ============================================
getenforce
iptables -L -n# ============================================
# PART 9: RESOLUTION AND VERIFICATION
# ============================================
systemctl restart nginx
systemctl status nginx
curl -I http://localhost# ============================================
# PART 10: DOCUMENTATION
# ============================================
# Record in ticket:
# - Symptom: nginx not responding on port 80
# - Root cause: configuration syntax error in sites-enabled/default
# - Fix: corrected missing semicolon, ran nginx -t, restarted
# - Prevention: add nginx -t to deployment pipelineThe ten parts moved through the troubleshooting workflow: orientation, resource check, process inspection, log analysis, network verification, configuration validation, dependency check, security layer review, resolution, and documentation.
Quick Reference
Troubleshooting Phases
| Phase | Purpose | Key Commands |
|---|---|---|
| Assessment | Understand the problem | uptime, dmesg, hostnamectl |
| Resources | Check CPU, memory, disk, I/O | top, free, df, iostat |
| Processes | Identify running services | ps, systemctl status, lsof |
| Logs | Find error messages | journalctl, tail, grep |
| Network | Verify connectivity | ip, ss, curl, dig |
| Resolution | Fix and verify | systemctl restart, test |
Log File Locations
| File | Content |
|---|---|
/var/log/syslog | General system messages |
/var/log/auth.log | Authentication events |
/var/log/kern.log | Kernel messages |
/var/log/dmesg | Boot-time kernel messages |
/var/log/nginx/ | Nginx-specific logs |
Diagnostic Commands
| Command | Purpose |
|---|---|
journalctl -u svc | Service logs from systemd |
journalctl -xe | Recent journal entries with details |
| `dmesg | tail` |
ss -tulpn | Listening ports and connections |
systemctl status svc | Service state and recent logs |
Best Practices
✅ Do This:
# Check logs before restarting services
journalctl -u nginx --since "10 min ago" # ✅
# Document findings as you investigate
echo "14:32 - nginx error: bind() failed" >> /tmp/troubleshoot.log # ✅
# Verify one layer at a time
ping gateway && dig domain && curl localhost # ✅
# Test configuration after changes
nginx -t && systemctl reload nginx # ✅❌ Don’t Do This:
# Restart without understanding the failure
systemctl restart nginx # ❌
# Change multiple things at once
sed -i 's/80/8080/' config && systemctl restart nginx # ❌
# Ignore logs because the fix worked
# (the same problem will recur) # ❌
# Skip documentation
# (no record of what was done or why) # ❌Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Fixing symptom, not cause | Restart masks underlying issue | Read logs before restarting |
| Skipping resource check | Assumes application is at fault | Check CPU, memory, disk first |
| Ignoring timestamps | Individual errors mislead | Correlate events by time |
| Changing multiple variables | Unclear which change fixed it | Change one thing at a time |
| Not documenting | Knowledge lost after resolution | Record findings immediately |
Real-World Examples
1. Service Fails to Start
systemctl status myservice
journalctl -u myservice --no-pager | tail -302. Disk Full Prevents Logging
df -h
du -sh /var/log/* | sort -rh | head -103. High Load Average
uptime
top -bn1 | head -154. Network Unreachable
ip addr show
ping -c 3 8.8.8.85. DNS Resolution Failure
dig example.com
cat /etc/resolv.conf6. Port Already in Use
ss -tulpn | grep :8080
lsof -i :80807. Permission Denied
ls -la /path/to/file
namei -l /path/to/file8. SSH Connection Refused
systemctl status sshd
ss -tulpn | grep :229. Memory Exhaustion
free -h
ps aux --sort=-%mem | head -510. Configuration Syntax Error
nginx -t
apachectl configtestVisual
The Troubleshooting Funnel
┌─────────────────────────────────────────────────────────────┐
│ METHODICAL TROUBLESHOOTING FUNNEL │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ INITIAL ASSESSMENT │ │
│ │ What is the system? What changed? │ │
│ └─────────────────────┬───────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ RESOURCE CHECK │ │
│ │ CPU, memory, disk, I/O — is the system healthy? │ │
│ └─────────────────────┬───────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ PROCESS & LOG ANALYSIS │ │
│ │ What is running? What does it say? │ │
│ └─────────────────────┬───────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ NETWORK VERIFICATION │ │
│ │ Is the path to the service clear? │ │
│ └─────────────────────┬───────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ RESOLUTION │ │
│ │ Fix, verify, document. │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘Layer-by-Layer Network Diagnosis
┌─────────────────────────────────────────────────────────────┐
│ NETWORK TROUBLESHOOTING LAYERS │
│ │
│ Layer 1: Interface │
│ ip addr show → is the interface UP with an IP? │
│ │
│ Layer 2: Gateway │
│ ping gateway → can we reach the local router? │
│ │
│ Layer 3: DNS │
│ dig domain → does the name resolve? │
│ │
│ Layer 4: Service │
│ curl localhost → is the service listening? │
│ │
│ Layer 5: Firewall │
│ iptables -L → is traffic blocked? │
│ │
│ Each layer assumes the previous layer is working. │
│ │
└─────────────────────────────────────────────────────────────┘The Investigation Loop
┌─────────────────────────────────────────────────────────────┐
│ HYPOTHESIS-DRIVEN TROUBLESHOOTING │
│ │
│ ┌──────────────────┐ │
│ │ OBSERVE │ │
│ │ (symptoms) │ │
│ └────────┬─────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ HYPOTHESIZE │ │
│ │ (likely cause) │ │
│ └────────┬─────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ TEST │ │
│ │ (one variable) │ │
│ └────────┬─────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ CONFIRM? │ │
│ └────────┬─────────┘ │
│ │ │
│ ┌─────────┴─────────┐ │
│ ▼ ▼ │
│ ┌─────────┐ ┌─────────┐ │
│ │ FIX │ │ DISCARD │ │
│ └─────────┘ └────┬────┘ │
│ │ │
│ └──────▶ back to HYPOTHESIZE │
│ │
└─────────────────────────────────────────────────────────────┘Documentation Template
┌─────────────────────────────────────────────────────────────┐
│ INCIDENT DOCUMENTATION │
│ │
│ Date/Time: │
│ Reported by: │
│ │
│ SYMPTOM: │
│ What was observed? │
│ │
│ INVESTIGATION: │
│ Commands run, outputs observed. │
│ │
│ ROOT CAUSE: │
│ What was actually wrong? │
│ │
│ RESOLUTION: │
│ What was changed to fix it? │
│ │
│ PREVENTION: │
│ What monitoring or process change prevents recurrence? │
│ │
└─────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Domain | System Administration Fundamentals |
| First phase | Initial assessment and orientation |
| Second phase | Resource analysis |
| Third phase | Process and log investigation |
| Fourth phase | Network verification |
| Fifth phase | Resolution and verification |
| Final phase | Documentation and prevention |
| Key principle | Change one thing at a time |
| Log command | journalctl -xe for systemd journal |
| Network command | ss -tulpn for listening ports |
| Documentation | Required for every incident |
Key takeaways:
- Troubleshooting is a process, not an instinct. The methodical approach provides a repeatable sequence that narrows the problem space layer by layer.
- Logs before restarts. Restarting a service without reading its logs destroys the evidence of why it failed. The failure will recur, and you will have learned nothing.
- Check resources early. CPU, memory, disk, and I/O problems masquerade as application problems. Ruling out resource exhaustion prevents wasted time in the wrong layer.
- Network diagnostics are layered. Test the interface, then the gateway, then DNS, then the service. Each layer assumes the previous one works.
- One change at a time. Changing multiple variables makes it impossible to know which change fixed the problem. The methodical approach requires controlled experiments.
- Documentation is part of the fix. An undocumented resolution is a temporary fix. The next occurrence will require the entire investigation to be repeated.
Remember: The LFCA exam tests troubleshooting as scenario-based reasoning. You will be given a symptom and asked which diagnostic step comes next. The answer is always the step that gathers evidence without assuming the cause. A methodical approach is not just good practice for real systems—it is the specific reasoning pattern the exam evaluates.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!