📊 Log Analysis for Security Events on AlmaLinux: Find Threats Hidden in Your Logs!

Your servers are talking to you every second, leaving breadcrumbs about everything that happens - logins, file access, network connections, errors! 📝 But here’s the thing: hackers leave footprints in these logs too! Today, we’re becoming log detectives, learning to spot security threats hiding in plain sight among millions of log entries. Get ready to transform from log-confused to log-enlightened! 🕵️‍♂️

🤔 Why is Log Analysis Important?

Think of logs as your server’s diary - they record EVERYTHING! And just like a detective reads clues at a crime scene, you can read logs to catch hackers red-handed! 🔍

Here’s why log analysis is your security superpower:

• 🎯 Detect breaches early - Spot attacks before damage is done
• 🕰️ Forensic investigations - Reconstruct exactly what happened
• 📈 Pattern recognition - Identify attack trends and methods
• 🚨 Real-time alerting - Get notified of suspicious activities instantly
• 📊 Compliance requirements - Many standards require log analysis
• 🔐 Insider threat detection - Catch malicious employees
• 💡 Security insights - Understand your attack surface
• 🛡️ Proactive defense - Stop attacks before they succeed

🎯 What You Need

Before we start hunting through logs, let’s check our toolkit! Simple requirements:

• ✅ AlmaLinux system with logs to analyze
• ✅ Root or sudo access (we need to read all logs! 💪)
• ✅ Basic command line knowledge
• ✅ About 30 minutes to learn
• ✅ Curiosity about what’s happening on your system
• ✅ Coffee ready (log analysis needs focus! ☕)

📝 Step 1: Understanding AlmaLinux Log Structure

First, let’s understand where logs live and what they contain!

# View main log locations
ls -la /var/log/
# Shows all system logs

# Key security-related logs:
# /var/log/secure - Authentication and authorization
# /var/log/messages - General system messages
# /var/log/audit/audit.log - Audit subsystem
# /var/log/httpd/ - Web server logs
# /var/log/firewalld - Firewall logs

# Check log sizes
du -sh /var/log/* | sort -h
# Shows which logs are largest

# View recent security events
sudo tail -f /var/log/secure
# Real-time security log monitoring

Let’s understand log formats:

# Typical log entry anatomy
# Nov 15 14:23:45 server01 sshd[12345]: Failed password for root from 192.168.1.100 port 22 ssh2
# Date | Time | Hostname | Service[PID] | Message

# Create a log parsing reference
cat << 'EOF' > /opt/log-formats.txt
Common Log Formats:

1. Syslog Format:
   Nov 15 14:23:45 hostname service[PID]: message

2. Apache Access Log:
   IP - - [timestamp] "METHOD /path HTTP/1.1" status size "referer" "user-agent"

3. Audit Log:
   type=EVENT msg=audit(timestamp:id): key=value pairs

4. JSON Logs:
   {"timestamp":"2024-11-15T14:23:45","level":"ERROR","message":"Failed login"}
EOF

🔧 Step 2: Essential Log Analysis Commands

Master these commands to become a log analysis ninja! 🥷

# Create security analysis script
cat << 'EOF' > /opt/security-log-analyzer.sh
#!/bin/bash
# Security Log Analyzer

echo "🔍 Security Log Analysis Report"
echo "================================"
echo "Generated: $(date)"
echo ""

# Failed SSH attempts
echo "📊 Failed SSH Login Attempts (Last 24 hours):"
grep "Failed password" /var/log/secure | \
  grep "$(date '+%b %_d')" | \
  awk '{print $11}' | \
  sort | uniq -c | sort -rn | head -10
echo ""

# Successful logins
echo "✅ Successful Logins:"
grep "Accepted password\|Accepted publickey" /var/log/secure | \
  tail -10
echo ""

# Sudo commands executed
echo "🔐 Recent Sudo Commands:"
grep "sudo" /var/log/secure | \
  grep "COMMAND" | \
  tail -10
echo ""

# New user accounts
echo "👤 User Account Changes:"
grep "useradd\|userdel\|usermod" /var/log/secure | \
  tail -5
echo ""

# Service failures
echo "❌ Service Failures:"
journalctl -p err -since "24 hours ago" | \
  head -10
echo ""

# Firewall blocks
echo "🛡️ Firewall Blocked Connections:"
grep "REJECT\|DROP" /var/log/firewalld 2>/dev/null | \
  tail -10
echo ""

# Suspicious processes
echo "⚠️ Suspicious Process Activity:"
grep -E "nc |netcat|/dev/tcp|curl.*sh|wget.*sh" /var/log/messages | \
  tail -5
echo ""

# File integrity changes
echo "📁 File Modification Alerts:"
grep -E "changed|modified|created|deleted" /var/log/audit/audit.log 2>/dev/null | \
  grep -E "/etc/|/usr/bin/|/usr/sbin/" | \
  tail -5
EOF

chmod +x /opt/security-log-analyzer.sh

🌟 Step 3: Advanced Pattern Detection

Let’s create sophisticated detection rules for common attacks!

# Create intrusion detection patterns
cat << 'EOF' > /opt/attack-patterns.sh
#!/bin/bash
# Attack Pattern Detector

echo "🚨 Checking for Attack Patterns..."

# Brute Force Detection
BRUTE_FORCE=$(grep "Failed password" /var/log/secure | \
  awk '{print $11}' | \
  sort | uniq -c | \
  awk '$1 > 5 {print $2 " (" $1 " attempts)"}')

if [ ! -z "$BRUTE_FORCE" ]; then
  echo "⚠️ BRUTE FORCE DETECTED from:"
  echo "$BRUTE_FORCE"
fi

# Port Scanning Detection
PORT_SCAN=$(grep "DPT=" /var/log/firewalld 2>/dev/null | \
  awk '{print $NF}' | \
  cut -d= -f2 | \
  sort | uniq -c | \
  awk '$1 > 10 {print "Multiple ports from same source"}' | head -1)

if [ ! -z "$PORT_SCAN" ]; then
  echo "⚠️ PORT SCAN DETECTED"
fi

# Privilege Escalation Attempts
PRIV_ESC=$(grep -E "su:|sudo:" /var/log/secure | \
  grep "authentication failure" | \
  wc -l)

if [ $PRIV_ESC -gt 3 ]; then
  echo "⚠️ PRIVILEGE ESCALATION ATTEMPTS: $PRIV_ESC"
fi

# Web Application Attacks
WEB_ATTACKS=$(grep -E "\.\.\/|SELECT.*FROM|<script>|eval\(|base64_decode" \
  /var/log/httpd/access_log 2>/dev/null | wc -l)

if [ $WEB_ATTACKS -gt 0 ]; then
  echo "⚠️ WEB APPLICATION ATTACKS: $WEB_ATTACKS attempts"
fi

# Malware Indicators
MALWARE=$(grep -E "crypto|miner|xmr|monero" /var/log/messages | \
  wc -l)

if [ $MALWARE -gt 0 ]; then
  echo "⚠️ POSSIBLE MALWARE ACTIVITY: $MALWARE indicators"
fi

# Data Exfiltration
EXFIL=$(netstat -an | grep ESTABLISHED | \
  awk '{print $5}' | \
  cut -d: -f1 | \
  sort | uniq -c | \
  awk '$1 > 100 {print $2 " (" $1 " connections)"}')

if [ ! -z "$EXFIL" ]; then
  echo "⚠️ POSSIBLE DATA EXFILTRATION to:"
  echo "$EXFIL"
fi
EOF

chmod +x /opt/attack-patterns.sh

✅ Step 4: Automate Security Monitoring

Let’s set up automated log monitoring and alerting!

# Create real-time monitoring script
cat << 'EOF' > /opt/realtime-security-monitor.sh
#!/bin/bash
# Real-time Security Monitor

# Configuration
ALERT_EMAIL="[email protected]"
LOG_DIR="/var/log/security-alerts"
mkdir -p $LOG_DIR

# Monitor function
monitor_logs() {
  echo "🔍 Starting real-time security monitoring..."
  
  # Monitor multiple logs simultaneously
  tail -F /var/log/secure /var/log/messages /var/log/httpd/access_log 2>/dev/null | \
  while read line; do
    TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
    
    # Check for critical events
    if echo "$line" | grep -qE "Failed password.*root"; then
      echo "[$TIMESTAMP] 🔴 CRITICAL: Root login attempt!" | tee -a $LOG_DIR/critical.log
      echo "Root login attempt detected: $line" | mail -s "SECURITY ALERT: Root Login" $ALERT_EMAIL 2>/dev/null
    fi
    
    if echo "$line" | grep -qE "COMMAND=/usr/bin/passwd|/usr/sbin/useradd"; then
      echo "[$TIMESTAMP] 🟠 HIGH: User management activity!" | tee -a $LOG_DIR/high.log
    fi
    
    if echo "$line" | grep -qE "Failed password" | grep -qv "root"; then
      echo "[$TIMESTAMP] 🟡 MEDIUM: Failed login attempt" | tee -a $LOG_DIR/medium.log
    fi
    
    if echo "$line" | grep -qE "\.\.\/|SELECT.*FROM|<script>"; then
      echo "[$TIMESTAMP] 🔴 CRITICAL: Web attack detected!" | tee -a $LOG_DIR/critical.log
    fi
    
    if echo "$line" | grep -qE "sudo.*COMMAND"; then
      echo "[$TIMESTAMP] ℹ️ INFO: Sudo command executed" | tee -a $LOG_DIR/info.log
    fi
  done
}

# Start monitoring
monitor_logs
EOF

chmod +x /opt/realtime-security-monitor.sh

# Create systemd service for monitoring
cat << 'EOF' > /etc/systemd/system/security-monitor.service
[Unit]
Description=Real-time Security Log Monitor
After=network.target

[Service]
Type=simple
ExecStart=/opt/realtime-security-monitor.sh
Restart=always
User=root

[Install]
WantedBy=multi-user.target
EOF

# Enable service
sudo systemctl daemon-reload
sudo systemctl enable security-monitor

🎮 Quick Examples

Let’s analyze real security scenarios! 🔥

Example 1: Detect SSH Brute Force Attack

# Find brute force attempts
echo "=== SSH Brute Force Analysis ==="

# Get attacking IPs
echo "Top attacking IPs:"
grep "Failed password" /var/log/secure | \
  awk '{print $11}' | \
  sort | uniq -c | \
  sort -rn | \
  head -5

# Timeline of attack
echo -e "\nAttack timeline:"
grep "Failed password" /var/log/secure | \
  awk '{print $1, $2, $3}' | \
  uniq -c

# Targeted usernames
echo -e "\nTargeted usernames:"
grep "Failed password" /var/log/secure | \
  awk '{print $9}' | \
  sort | uniq -c | \
  sort -rn

# Check if any succeeded
echo -e "\nChecking for successful breach:"
ATTACKER_IP=$(grep "Failed password" /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -rn | head -1 | awk '{print $2}')
grep "Accepted" /var/log/secure | grep "$ATTACKER_IP" && echo "⚠️ BREACH DETECTED!" || echo "✅ No successful breach"

Example 2: Web Application Attack Analysis

# Analyze web attacks
cat << 'EOF' > /opt/web-attack-analyzer.sh
#!/bin/bash

echo "=== Web Attack Analysis ==="

# SQL Injection attempts
echo "SQL Injection Attempts:"
grep -E "SELECT|UNION|INSERT|UPDATE|DELETE|DROP" /var/log/httpd/access_log | \
  awk '{print $1}' | \
  sort | uniq -c | \
  sort -rn | \
  head -5

# XSS attempts
echo -e "\nXSS Attempts:"
grep -E "<script>|javascript:|onerror=|onload=" /var/log/httpd/access_log | \
  wc -l

# Directory traversal
echo -e "\nDirectory Traversal Attempts:"
grep -E "\.\.\/|\.\.%2f" /var/log/httpd/access_log | \
  awk '{print $1, $7}' | \
  head -5

# Suspicious user agents
echo -e "\nSuspicious User Agents:"
grep -E "sqlmap|nikto|nmap|masscan|metasploit" /var/log/httpd/access_log | \
  awk -F'"' '{print $6}' | \
  sort | uniq
EOF

chmod +x /opt/web-attack-analyzer.sh

Example 3: User Behavior Analysis

# Create user behavior analyzer
cat << 'EOF' > /opt/user-behavior-analyzer.sh
#!/bin/bash

USER=$1
if [ -z "$USER" ]; then
  echo "Usage: $0 <username>"
  exit 1
fi

echo "=== User Behavior Analysis for: $USER ==="

# Login times
echo "Login Pattern:"
last $USER | head -10

# Commands executed
echo -e "\nCommands Run (sudo):"
grep "sudo.*$USER" /var/log/secure | \
  grep "COMMAND" | \
  tail -5

# File access
echo -e "\nFile Access (audit):"
ausearch -ua $USER 2>/dev/null | \
  grep "type=PATH" | \
  tail -5

# Network connections
echo -e "\nNetwork Activity:"
ss -tunp | grep -i $USER

# Process activity
echo -e "\nCurrent Processes:"
ps aux | grep "^$USER"
EOF

chmod +x /opt/user-behavior-analyzer.sh

🚨 Fix Common Problems

Don’t worry if log analysis seems overwhelming! Here are solutions! 💪

Problem 1: “Too many logs to analyze manually”

# Solution: Use log aggregation and filtering
# Create daily summary script
cat << 'EOF' > /opt/daily-log-summary.sh
#!/bin/bash

REPORT="/var/log/daily-security-summary-$(date +%Y%m%d).txt"

{
  echo "Daily Security Summary - $(date)"
  echo "====================================="
  echo ""
  echo "Failed Logins: $(grep -c 'Failed password' /var/log/secure)"
  echo "Successful Logins: $(grep -c 'Accepted' /var/log/secure)"
  echo "Sudo Commands: $(grep -c 'sudo.*COMMAND' /var/log/secure)"
  echo "Service Errors: $(journalctl -p err --since yesterday | wc -l)"
  echo "Firewall Blocks: $(grep -c 'REJECT\|DROP' /var/log/firewalld 2>/dev/null)"
  echo ""
  echo "Top 5 Failed Login Sources:"
  grep "Failed password" /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -rn | head -5
} > $REPORT

cat $REPORT
# Email report
mail -s "Daily Security Summary" [email protected] < $REPORT
EOF

chmod +x /opt/daily-log-summary.sh
# Add to crontab
echo "0 6 * * * /opt/daily-log-summary.sh" | crontab -

Problem 2: “Missing important events”

# Solution: Increase logging verbosity
# Configure rsyslog for better security logging
cat << 'EOF' >> /etc/rsyslog.conf

# Enhanced security logging
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
*.emerg :omusrmsg:*

# Log all commands
local6.* /var/log/commands.log
EOF

# Restart rsyslog
systemctl restart rsyslog

# Enable audit logging for commands
auditctl -a always,exit -F arch=b64 -S execve -k command_exec

Problem 3: “Logs rotating too quickly”

# Solution: Adjust log rotation
# Edit logrotate configuration
cat << 'EOF' > /etc/logrotate.d/security-logs
/var/log/secure
/var/log/messages
{
    daily
    rotate 90
    compress
    delaycompress
    notifempty
    create 0600 root root
    sharedscripts
    postrotate
        /usr/bin/systemctl reload rsyslog > /dev/null 2>&1 || true
    endscript
}
EOF

# Force rotation to test
logrotate -f /etc/logrotate.d/security-logs

📋 Simple Commands Summary

Your log analysis cheat sheet! 📌

💡 Tips for Success

Become a log analysis master with these tips! 🚀

Log Analysis Best Practices

• 📅 Analyze logs daily, not just during incidents
• 🔄 Automate repetitive analysis tasks
• 📊 Create baselines of normal behavior
• 🎯 Focus on high-value security events first

Security Patterns to Watch

# Critical patterns to monitor
- Multiple failed logins from same IP
- Sudo usage by non-admin users
- Outbound connections to unusual ports
- Large data transfers at odd hours
- Service account interactive logins
- File changes in system directories

Performance Tips

• 💾 Use log indexing for faster searches
• 🔍 Pre-filter logs before complex analysis
• 📈 Aggregate similar events
• ⚡ Use parallel processing for large logs

Integration Ideas

• 📧 Email critical alerts immediately
• 📊 Send metrics to monitoring dashboards
• 🎫 Create tickets for security events
• 🤖 Integrate with SIEM platforms

🏆 What You Learned

Amazing work! You’re now a log analysis expert! 🎊

• ✅ Understood AlmaLinux log structure and locations
• ✅ Mastered essential log analysis commands
• ✅ Created security event detection scripts
• ✅ Built automated monitoring systems
• ✅ Learned attack pattern recognition
• ✅ Implemented real-time alerting
• ✅ Analyzed multiple attack scenarios
• ✅ Set up log aggregation and reporting
• ✅ Troubleshot common log issues
• ✅ Built professional log analysis workflow

🎯 Why This Matters

You’ve transformed from someone who ignores logs to a security analyst who finds needles in haystacks! 🔍 Logs are no longer mysterious text files - they’re your window into everything happening on your systems.

This skill is invaluable. You can now detect breaches early, prove compliance, investigate incidents, and understand your security posture. You’re not waiting for alerts from expensive tools - you’re finding threats yourself using built-in Linux capabilities!

Your AlmaLinux system is now under constant surveillance by your own custom security monitoring. You’ve built the foundation for professional security operations using nothing but bash scripts and standard commands! 💪

Keep analyzing, keep learning, and remember - every security incident leaves traces in the logs! You’ve got this! ⭐

Happy hunting, AlmaLinux log detective! 🙌