📊 Creating Security Dashboards on AlmaLinux: Visualize Threats in Real-Time!

Imagine walking into a Security Operations Center with walls of screens showing real-time threats, attack maps, and system health - looks cool, right? 😎 Well, today we’re building exactly that for your AlmaLinux infrastructure! No more digging through logs or running commands repeatedly. We’re creating beautiful, interactive dashboards that show you everything at a glance. Let’s transform your security monitoring from command-line chaos to visual victory! 🚀

🤔 Why are Security Dashboards Important?

Think of security dashboards as your mission control center - showing everything important on one screen, updating in real-time! It’s like having X-ray vision for your infrastructure! 👁️

Here’s why security dashboards are game-changing:

• 👀 Instant visibility - See threats as they happen
• 🎯 Pattern recognition - Spot trends humans might miss
• ⚡ Faster response - Visual alerts grab attention immediately
• 📊 Executive reporting - Show security posture to management
• 🤝 Team coordination - Everyone sees the same picture
• 📈 Historical analysis - Track security trends over time
• 🔔 Smart alerting - Visual and audio alerts for critical events
• 💡 Situational awareness - Understand your security landscape

🎯 What You Need

Before we build your security command center, let’s check requirements! Don’t worry, it’s manageable:

• ✅ AlmaLinux system (2GB+ RAM recommended)
• ✅ Root or sudo access (we’re installing services! 💪)
• ✅ Web browser for viewing dashboards
• ✅ Basic understanding of JSON
• ✅ About 45 minutes to set up
• ✅ Logs to visualize (obviously! 😄)
• ✅ Coffee or energy drink (this is fun but detailed! ⚡)

📝 Step 1: Install Dashboard Stack

We’ll use Grafana for visualization and Prometheus for metrics! Let’s set them up.

# Add Grafana repository
cat << 'EOF' > /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

# Install Grafana
sudo dnf install -y grafana

# Install Prometheus
cd /opt
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xzf prometheus-2.45.0.linux-amd64.tar.gz
mv prometheus-2.45.0.linux-amd64 prometheus

# Install Loki for log aggregation
wget https://github.com/grafana/loki/releases/download/v2.9.0/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
mv loki-linux-amd64 /opt/loki

# Install Promtail (log collector)
wget https://github.com/grafana/loki/releases/download/v2.9.0/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
mv promtail-linux-amd64 /opt/promtail

# Enable and start Grafana
sudo systemctl enable --now grafana-server

# Verify Grafana is running
sudo systemctl status grafana-server
# Should show active (running)

echo "Grafana URL: http://$(hostname -I | awk '{print $1}'):3000"
echo "Default login: admin/admin"

🔧 Step 2: Configure Log Collection and Metrics

Let’s set up data sources for our dashboards!

# Configure Prometheus
cat << 'EOF' > /opt/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
  
  - job_name: 'security-metrics'
    static_configs:
      - targets: ['localhost:9090']
EOF

# Create Prometheus service
cat << 'EOF' > /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
After=network.target

[Service]
User=root
ExecStart=/opt/prometheus/prometheus \
  --config.file=/opt/prometheus/prometheus.yml \
  --storage.tsdb.path=/opt/prometheus/data
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# Configure Loki
cat << 'EOF' > /opt/loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    shared_store: filesystem
  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
EOF

# Configure Promtail to collect logs
cat << 'EOF' > /opt/promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  - job_name: security
    static_configs:
      - targets:
          - localhost
        labels:
          job: security-logs
          __path__: /var/log/secure
      
      - targets:
          - localhost
        labels:
          job: audit-logs
          __path__: /var/log/audit/audit.log
      
      - targets:
          - localhost
        labels:
          job: system-logs
          __path__: /var/log/messages
EOF

# Start all services
sudo systemctl daemon-reload
sudo systemctl start prometheus
/opt/loki -config.file=/opt/loki-config.yaml &
/opt/promtail -config.file=/opt/promtail-config.yaml &

🌟 Step 3: Create Security Metrics Collector

Let’s build a custom exporter for security metrics!

# Create security metrics exporter
cat << 'EOF' > /opt/security-metrics.py
#!/usr/bin/env python3
import time
import re
import subprocess
from prometheus_client import start_http_server, Gauge, Counter
from datetime import datetime, timedelta

# Define metrics
failed_login_attempts = Gauge('security_failed_logins', 'Failed login attempts in last hour')
active_connections = Gauge('security_active_connections', 'Current active network connections')
suspicious_processes = Gauge('security_suspicious_processes', 'Number of suspicious processes')
firewall_blocks = Counter('security_firewall_blocks', 'Total firewall blocked connections')
sudo_commands = Counter('security_sudo_commands', 'Total sudo commands executed')
ssh_sessions = Gauge('security_ssh_sessions', 'Active SSH sessions')
selinux_denials = Counter('security_selinux_denials', 'SELinux denial count')
open_ports = Gauge('security_open_ports', 'Number of open ports')

def collect_metrics():
    while True:
        try:
            # Failed login attempts
            result = subprocess.run(['grep', 'Failed password', '/var/log/secure'], 
                                  capture_output=True, text=True)
            failed_count = len(result.stdout.splitlines())
            failed_login_attempts.set(failed_count)
            
            # Active connections
            result = subprocess.run(['ss', '-tun'], capture_output=True, text=True)
            conn_count = len(result.stdout.splitlines()) - 1
            active_connections.set(conn_count)
            
            # Suspicious processes
            result = subprocess.run(['ps', 'aux'], capture_output=True, text=True)
            suspicious = sum(1 for line in result.stdout.splitlines() 
                           if any(s in line for s in ['nc', 'nmap', 'tcpdump']))
            suspicious_processes.set(suspicious)
            
            # SSH sessions
            result = subprocess.run(['who'], capture_output=True, text=True)
            ssh_count = len([l for l in result.stdout.splitlines() if 'pts' in l])
            ssh_sessions.set(ssh_count)
            
            # Open ports
            result = subprocess.run(['ss', '-ltn'], capture_output=True, text=True)
            ports = len(result.stdout.splitlines()) - 1
            open_ports.set(ports)
            
        except Exception as e:
            print(f"Error collecting metrics: {e}")
        
        time.sleep(10)  # Collect every 10 seconds

if __name__ == '__main__':
    # Start Prometheus metrics server
    start_http_server(9091)
    print("Security metrics exporter started on port 9091")
    collect_metrics()
EOF

chmod +x /opt/security-metrics.py

# Install Python Prometheus client
pip3 install prometheus-client

# Create service for metrics exporter
cat << 'EOF' > /etc/systemd/system/security-metrics.service
[Unit]
Description=Security Metrics Exporter
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/security-metrics.py
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now security-metrics

✅ Step 4: Build Security Dashboards

Now for the fun part - creating beautiful dashboards! 🎨

# Create dashboard configuration
cat << 'EOF' > /opt/security-dashboard.json
{
  "dashboard": {
    "title": "Security Operations Dashboard",
    "panels": [
      {
        "title": "🚨 Failed Login Attempts",
        "type": "graph",
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
        "targets": [
          {
            "expr": "rate(security_failed_logins[5m])",
            "legendFormat": "Failed Logins/sec"
          }
        ]
      },
      {
        "title": "🔐 Active SSH Sessions",
        "type": "stat",
        "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
        "targets": [
          {
            "expr": "security_ssh_sessions"
          }
        ]
      },
      {
        "title": "🌐 Network Connections",
        "type": "gauge",
        "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
        "targets": [
          {
            "expr": "security_active_connections"
          }
        ]
      },
      {
        "title": "⚠️ Suspicious Processes",
        "type": "stat",
        "gridPos": {"h": 4, "w": 6, "x": 12, "y": 4},
        "targets": [
          {
            "expr": "security_suspicious_processes"
          }
        ],
        "thresholds": {
          "mode": "absolute",
          "steps": [
            {"color": "green", "value": 0},
            {"color": "yellow", "value": 1},
            {"color": "red", "value": 3}
          ]
        }
      },
      {
        "title": "🛡️ Firewall Blocks",
        "type": "graph",
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8},
        "targets": [
          {
            "expr": "rate(security_firewall_blocks[5m])",
            "legendFormat": "Blocks/sec"
          }
        ]
      },
      {
        "title": "📊 Security Event Log",
        "type": "logs",
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 8},
        "targets": [
          {
            "expr": "{job=\"security-logs\"} |~ \"Failed|error|attack\""
          }
        ]
      }
    ],
    "refresh": "5s",
    "time": {"from": "now-1h", "to": "now"}
  }
}
EOF

# Import dashboard via API
curl -X POST \
  http://admin:admin@localhost:3000/api/dashboards/db \
  -H 'Content-Type: application/json' \
  -d @/opt/security-dashboard.json

🎮 Quick Examples

Let’s create specialized security dashboards! 🔥

Example 1: Attack Map Dashboard

# Create geographic attack visualization
cat << 'EOF' > /opt/attack-map-collector.sh
#!/bin/bash
# Collect IP addresses and geolocate

FAILED_IPS=$(grep "Failed password" /var/log/secure | \
  awk '{print $11}' | \
  sort -u)

for IP in $FAILED_IPS; do
  # Get geolocation (requires geoiplookup)
  LOCATION=$(geoiplookup $IP 2>/dev/null | grep "GeoIP Country" | cut -d: -f2)
  echo "attack_source{ip=\"$IP\",country=\"$LOCATION\"} 1" >> /opt/attack-metrics.txt
done

# Expose metrics for Prometheus
cat /opt/attack-metrics.txt
EOF

chmod +x /opt/attack-map-collector.sh

Example 2: Real-time Threat Feed

# Create threat intelligence dashboard
cat << 'EOF' > /opt/threat-dashboard.py
#!/usr/bin/env python3
import json
import time
from flask import Flask, render_template
from flask_socketio import SocketIO, emit
import subprocess
import threading

app = Flask(__name__)
socketio = SocketIO(app)

def monitor_threats():
    """Monitor logs and emit threats in real-time"""
    proc = subprocess.Popen(['tail', '-F', '/var/log/secure'],
                           stdout=subprocess.PIPE,
                           stderr=subprocess.PIPE)
    
    while True:
        line = proc.stdout.readline().decode('utf-8')
        if 'Failed password' in line:
            threat = {
                'type': 'brute_force',
                'severity': 'high',
                'message': line.strip(),
                'timestamp': time.time()
            }
            socketio.emit('new_threat', threat, broadcast=True)
        elif 'sudo' in line:
            threat = {
                'type': 'privilege_escalation',
                'severity': 'medium',
                'message': line.strip(),
                'timestamp': time.time()
            }
            socketio.emit('new_threat', threat, broadcast=True)

@app.route('/')
def index():
    return '''
    <!DOCTYPE html>
    <html>
    <head>
        <title>Real-time Threat Dashboard</title>
        <script src="https://cdn.socket.io/4.0.0/socket.io.min.js"></script>
        <style>
            .threat { padding: 10px; margin: 5px; border-radius: 5px; }
            .high { background: #ff4444; color: white; }
            .medium { background: #ffaa00; }
            .low { background: #44ff44; }
        </style>
    </head>
    <body>
        <h1>🚨 Real-time Threat Feed</h1>
        <div id="threats"></div>
        <script>
            const socket = io();
            socket.on('new_threat', function(threat) {
                const div = document.createElement('div');
                div.className = 'threat ' + threat.severity;
                div.innerHTML = `<strong>${threat.type}</strong>: ${threat.message}`;
                document.getElementById('threats').prepend(div);
            });
        </script>
    </body>
    </html>
    '''

if __name__ == '__main__':
    # Start threat monitoring in background
    thread = threading.Thread(target=monitor_threats)
    thread.daemon = True
    thread.start()
    
    # Run web server
    socketio.run(app, host='0.0.0.0', port=5000)
EOF

# Install requirements
pip3 install flask flask-socketio

# Run threat dashboard
python3 /opt/threat-dashboard.py &

Example 3: Executive Security Dashboard

# Create executive summary dashboard
cat << 'EOF' > /opt/executive-dashboard.sh
#!/bin/bash
# Generate executive security metrics

generate_report() {
  cat << HTML > /var/www/html/executive-dashboard.html
<!DOCTYPE html>
<html>
<head>
  <title>Executive Security Dashboard</title>
  <meta http-equiv="refresh" content="30">
  <style>
    body { font-family: Arial; background: #1a1a1a; color: white; }
    .metric { 
      display: inline-block; 
      margin: 20px; 
      padding: 20px; 
      background: #2a2a2a; 
      border-radius: 10px; 
      text-align: center;
    }
    .value { font-size: 48px; font-weight: bold; }
    .label { font-size: 18px; color: #888; }
    .green { color: #4CAF50; }
    .yellow { color: #FFC107; }
    .red { color: #f44336; }
  </style>
</head>
<body>
  <h1>🛡️ Security Status - $(date)</h1>
  
  <div class="metric">
    <div class="value green">$(systemctl is-active firewalld | grep -c active)</div>
    <div class="label">Firewall Active</div>
  </div>
  
  <div class="metric">
    <div class="value yellow">$(grep -c "Failed password" /var/log/secure)</div>
    <div class="label">Failed Logins (24h)</div>
  </div>
  
  <div class="metric">
    <div class="value green">$(dnf check-update 2>/dev/null | grep -c "^$" || echo "0")</div>
    <div class="label">Pending Updates</div>
  </div>
  
  <div class="metric">
    <div class="value red">$(grep -c "CRITICAL" /var/log/messages)</div>
    <div class="label">Critical Events</div>
  </div>
  
  <div class="metric">
    <div class="value green">$(who | wc -l)</div>
    <div class="label">Active Users</div>
  </div>
  
  <div class="metric">
    <div class="value yellow">$(ss -tun | wc -l)</div>
    <div class="label">Network Connections</div>
  </div>
</body>
</html>
HTML
}

# Generate report every minute
while true; do
  generate_report
  sleep 60
done
EOF

chmod +x /opt/executive-dashboard.sh
/opt/executive-dashboard.sh &

🚨 Fix Common Problems

Don’t worry if dashboards don’t work perfectly at first! Here are fixes! 💪

Problem 1: “No data showing in Grafana”

# Solution: Check data sources
# Verify Prometheus is collecting data
curl http://localhost:9090/metrics

# Check Loki is receiving logs
curl http://localhost:3100/ready

# Add data sources in Grafana
curl -X POST http://admin:admin@localhost:3000/api/datasources \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Prometheus",
    "type": "prometheus",
    "url": "http://localhost:9090",
    "access": "proxy"
  }'

curl -X POST http://admin:admin@localhost:3000/api/datasources \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Loki",
    "type": "loki",
    "url": "http://localhost:3100",
    "access": "proxy"
  }'

Problem 2: “Dashboard too slow”

# Solution: Optimize queries and retention
# Reduce data retention
cat << EOF >> /opt/prometheus/prometheus.yml
storage:
  tsdb:
    retention.time: 7d
    retention.size: 5GB
EOF

# Optimize Grafana queries
# Use recording rules for complex queries
cat << EOF > /opt/prometheus/rules.yml
groups:
  - name: security_rules
    interval: 30s
    rules:
      - record: security:failed_logins:rate5m
        expr: rate(security_failed_logins[5m])
EOF

# Restart services
systemctl restart prometheus
systemctl restart grafana-server

Problem 3: “Alerts not working”

# Solution: Configure alerting
# Set up alert rules
cat << 'EOF' > /opt/grafana-alerts.json
{
  "alert": {
    "name": "High Failed Logins",
    "message": "Failed login attempts exceeded threshold",
    "conditions": [
      {
        "evaluator": {
          "params": [10],
          "type": "gt"
        },
        "query": {
          "model": "security_failed_logins",
          "params": ["5m", "now"]
        }
      }
    ],
    "notifications": [
      {
        "uid": "email-channel"
      }
    ]
  }
}
EOF

# Configure email notifications
cat << EOF >> /etc/grafana/grafana.ini
[smtp]
enabled = true
host = localhost:25
from_address = grafana@$(hostname)
EOF

systemctl restart grafana-server

📋 Simple Commands Summary

Your dashboard management cheat sheet! 📌

💡 Tips for Success

Master security dashboards with these pro tips! 🚀

Dashboard Design Principles

• 🎯 Most important metrics at top
• 🎨 Use color coding (red=bad, green=good)
• 📊 Mix graph types for variety
• ⏱️ Auto-refresh critical dashboards

Essential Security Metrics

Authentication:
  - Failed login attempts
  - Successful logins
  - Password changes
  - Account lockouts

Network:
  - Active connections
  - Firewall blocks
  - Port scans detected
  - Data transfer volumes

System:
  - Process anomalies
  - File integrity changes
  - Service failures
  - Resource usage spikes

Performance Optimization

• 📊 Pre-aggregate common queries
• 💾 Limit data retention periods
• 🔄 Use caching for static data
• ⚡ Optimize database indexes

Integration Ideas

• 📧 Email dashboard snapshots daily
• 📱 Mobile-responsive dashboards
• 🔔 Slack/Teams notifications
• 📺 TV display mode for SOC

🏆 What You Learned

Incredible work! You’ve built a professional security monitoring center! 🎊

• ✅ Installed and configured Grafana stack
• ✅ Set up Prometheus metrics collection
• ✅ Configured Loki log aggregation
• ✅ Built custom security metrics exporter
• ✅ Created beautiful security dashboards
• ✅ Implemented real-time threat visualization
• ✅ Built executive summary dashboards
• ✅ Set up alerting and notifications
• ✅ Optimized dashboard performance
• ✅ Created enterprise-grade monitoring

🎯 Why This Matters

You’ve transformed security monitoring from reactive to proactive! 👁️ No more staring at log files or missing critical events. Your dashboards provide instant situational awareness, making it impossible for threats to hide.

This isn’t just pretty pictures - it’s actionable intelligence. You can now spot attack patterns, track security metrics, prove compliance, and respond to incidents faster. Your security posture is visible to everyone from analysts to executives.

Your AlmaLinux infrastructure now has enterprise-grade security visualization that rivals commercial SOC platforms. You built this with open-source tools and your own creativity! 🎨

Keep visualizing, keep monitoring, and remember - a picture is worth a thousand log lines! You’ve got this! ⭐

Happy dashboarding, AlmaLinux security artist! 🙌