🔍 Troubleshooting & Logs

Where to Look When Things Break

Linux logs everything. When something fails, the answer is usually in a log file.

---

journalctl — Systemd Logs

Modern logging system for systemd:

journalctl                      # All logs
journalctl -b                   # Since last boot
journalctl -b -1                # Previous boot
journalctl --since "1 hour ago" # Recent logs
journalctl --since today        # Today's logs
journalctl -p err               # Only errors
journalctl -f                   # Follow (live)

Service-Specific Logs

journalctl -u ssh               # SSH service logs
journalctl -u nginx -f          # Follow nginx logs
journalctl -u docker --since "10 min ago"

Priority Levels

| Level | Name | Meaning | |-------|------|---------| | 0 | emerg | System is unusable | | 1 | alert | Immediate action needed | | 2 | crit | Critical conditions | | 3 | err | Error conditions | | 4 | warning | Warning conditions | | 5 | notice | Normal but significant | | 6 | info | Informational | | 7 | debug | Debug messages |

journalctl -p warning           # Warnings and above
journalctl -p 0..4              # Emergency through error

---

dmesg — Kernel Messages

The kernel logs hardware events, driver messages, and boot info:

dmesg                           # All kernel messages
dmesg | tail -20                # Last 20 lines
dmesg -T                        # Human-readable timestamps
dmesg -T | grep -i error        # Find errors
dmesg -T | grep -i usb          # USB events
dmesg -w                        # Follow (watch for new)

Common Uses

• Hardware issues: Disk errors, USB problems
• Driver loading: Module load failures
• Boot problems: What happened during startup

---

/var/log — Traditional Logs

Classic log files in /var/log:

| File | Contents | |------|----------| | syslog | General system messages | | auth.log | Authentication (logins, sudo) | | kern.log | Kernel messages | | dpkg.log | Package installations | | apt/history.log | APT operations | | boot.log | Boot messages | | nginx/ | Web server logs | | mysql/ | Database logs |

Reading Logs

cat /var/log/syslog             # Full file
tail -20 /var/log/syslog        # Last 20 lines
tail -f /var/log/syslog         # Follow live
less /var/log/syslog            # Scrollable viewer
grep error /var/log/syslog      # Find errors

Authentication Logs

# Who logged in?
tail /var/log/auth.log

# Failed login attempts
grep "Failed password" /var/log/auth.log

# sudo usage
grep sudo /var/log/auth.log

---

Common Troubleshooting Patterns

1. Service Won’t Start

sudo systemctl status myservice
journalctl -u myservice -n 50
# Look for error messages

2. Application Crashes

dmesg | tail -20                # Kernel messages
journalctl -b | grep -i "myapp"
# Check app's own logs if it has them

3. Disk Problems

dmesg | grep -i "error\|fail"
journalctl -p err               # All errors
df -h                           # Check disk space

4. Network Issues

journalctl -u NetworkManager
ping 8.8.8.8                    # Can we reach internet?
ip addr                         # Do we have an IP?

5. Login Problems

tail /var/log/auth.log
journalctl -u ssh

---

Disk Space for Logs

Logs can fill up your disk:

du -sh /var/log                 # Total log size
ls -lhS /var/log/*.log | head   # Largest log files

# Clean old journals
sudo journalctl --vacuum-time=7d   # Keep 7 days
sudo journalctl --vacuum-size=500M # Keep 500MB max

---

Useful Tools

less Navigation

| Key | Action | |-----|--------| | q | Quit | | /pattern | Search forward | | ?pattern | Search backward | | n | Next match | | N | Previous match | | g | Go to start | | G | Go to end | | F | Follow mode |

grep Patterns

grep -i error file.log          # Case insensitive
grep -C 3 error file.log        # 3 lines context
grep -v DEBUG file.log          # Exclude DEBUG lines
grep -E "error|fail" file.log   # Multiple patterns
zgrep error file.log.gz         # Search compressed

---

Try It!

Practice log investigation in the terminal:

Exercises:

1. View kernel messages: dmesg | head -20
2. Check for errors: dmesg | grep -i error
3. View systemd journal: journalctl -n 20
4. Check auth log: cat /var/log/auth.log 2>/dev/null || echo "Permission denied"

Try it in the terminal

A sandboxed shell with this lesson’s commands. Type help to see what’s available.