Log Analysis Mastery: The Ultimate Cheat Sheet

logs.md

📊 Log Analysis Mastery: The Ultimate Cheat Sheet

📜 Basic Log Viewing

Command	Description	Example
cat	📄 Display entire log file	cat /var/log/syslog
less	📃 View log file with pagination	less /var/log/auth.log
tail	🔚 View end of log file	tail /var/log/apache2/access.log
head	🔝 View beginning of log file	head /var/log/mysql/error.log
tail -f	👀 Follow log file in real-time	tail -f /var/log/syslog

🔍 Searching and Filtering

Command	Description	Example
grep	🔎 Search for patterns in logs	grep "error" /var/log/syslog
egrep	🔬 Extended grep (supports regex)	egrep "error|warning" /var/log/syslog
fgrep	🚀 Fast grep (literal strings only)	fgrep "literal string" /var/log/syslog
zgrep	🗜️ Search compressed log files	zgrep "error" /var/log/syslog.1.gz
sed	🔀 Stream editor for filtering/transforming	sed -n '/error/p' /var/log/syslog
awk	🔧 Pattern scanning and processing	awk '/error/ {print $0}' /var/log/syslog

📊 Log Analysis Tools

Tool	Description	Example
logwatch	📈 Summarize log files	logwatch --detail high
goaccess	📊 Real-time web log analyzer	goaccess /var/log/apache2/access.log
lnav	🔍 Log file navigator	lnav /var/log/syslog
multitail	👀 Monitor multiple log files	multitail /var/log/syslog /var/log/auth.log
journalctl	📚 Query systemd journal	journalctl -u nginx.service

🕰️ Time-based Log Analysis

Command	Description	Example
date -d	📅 Convert log timestamps	date -d "2023-05-01 14:30:00" +%s
awk with time	⏱️ Filter logs by time	awk '$1$2 >= "2023-05-01 14:30:00" && $1$2 < "2023-05-01 15:00:00"' /var/log/syslog
sed with time	🕰️ Extract time range	sed -n '/2023-05-01 14:30:00/,/2023-05-01 15:00:00/p' /var/log/syslog

📈 Log Statistics

Command	Description	Example
wc -l	🔢 Count log entries	wc -l /var/log/auth.log
sort | uniq -c	📊 Count unique entries	cat /var/log/auth.log | sort | uniq -c | sort -nr
cut -d' ' -f4 | sort | uniq -c	📊 Count events by field	cat /var/log/auth.log | cut -d' ' -f4 | sort | uniq -c | sort -nr

🔄 Log Rotation

Command	Description	Example
logrotate	🔄 Rotate, compress, and mail logs	logrotate /etc/logrotate.conf
ls -lth /var/log	📁 List log files by modification time	ls -lth /var/log | head

🔒 Secure Log Viewing

Command	Description	Example
sudo	🔐 Run commands with superuser privileges	sudo tail /var/log/auth.log
zcat	🗜️ View compressed log files	zcat /var/log/syslog.2.gz
zless	📃 View compressed logs with pagination	zless /var/log/syslog.2.gz

🔧 Advanced Techniques

Technique	Description	Example
Combining tools	🔗 Use pipes to combine commands	grep "error" /var/log/syslog | awk '{print $1, $2, $3}'
Regular expressions	🔬 Use regex for complex pattern matching	grep -E "error|warning" /var/log/syslog
Log parsing scripts	📜 Write custom scripts for log analysis	#!/bin/bash\ngrep "error" $1 | awk '{print $1, $2}'

🏆 Pro Tips

1. 📚 Always check log locations in /etc/rsyslog.conf or /etc/syslog-conf
2. 🔍 Use grep -v to exclude patterns from your search
3. 🔢 Use sed -n '10,20p' to view specific line ranges in a log file
4. 📊 Combine sort, uniq, and cut for powerful log analysis
5. 🔐 Remember to use sudo when accessing system logs
6. 🗜️ Use zgrep, zcat, and zless for compressed logs
7. 🔄 Set up log rotation to manage log file sizes
8. 📈 Use watch command to repeatedly run log analysis commands
9. 🕰️ Convert timestamps to epoch time for easier comparison
10. 🔧 Learn awk for powerful text processing in logs

📜 Example: Complex Log Analysis

#!/bin/bash

# Count HTTP status codes in Apache access log
echo "HTTP Status Code Distribution:"
awk '{print $9}' /var/log/apache2/access.log | sort | uniq -c | sort -rn

# Find top 10 IP addresses
echo -e "\nTop 10 IP Addresses:"
awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -rn | head -n 10

# Count errors in syslog for the last hour
echo -e "\nErrors in syslog (last hour):"
sudo awk -v date=$(date -d '1 hour ago' '+%Y-%m-%d %H:%M:%S') '$0 > date && /error/' /var/log/syslog | wc -l

# Find failed SSH login attempts
echo -e "\nFailed SSH Login Attempts:"
sudo grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -nr

# Monitor real-time error rate
echo -e "\nReal-time Error Rate (Ctrl+C to stop):"
tail -f /var/log/syslog | grep --line-buffered "error" | awk '{print $1, $2, $3}' | 
    while read date ; do 
        echo "$(date): Error occurred" ; 
    done | uniq -c

Remember, log analysis is crucial for system administration and security. Always handle logs securely and respect privacy concerns! 🔒📊