Managing System Logs with journalctl in AlmaLinux

Introduction

The systemd journal, managed through journalctl, has revolutionized log management in modern Linux distributions like AlmaLinux. Unlike traditional text-based log files, the systemd journal provides a structured, indexed, and centralized logging system that captures everything from kernel messages to application logs. This comprehensive guide explores journalctl’s powerful features, from basic queries to advanced filtering and analysis techniques.

Understanding the systemd Journal

Architecture Overview

The systemd journal collects and stores log data in a structured format:

Binary Storage: Logs stored in binary format for efficiency
Structured Data: Metadata-rich entries with indexed fields
Centralized Collection: All system logs in one place
Automatic Rotation: Built-in log rotation and size management
Cryptographic Sealing: Optional log authentication

Journal Components

# View journal configuration
cat /etc/systemd/journald.conf

# Check journal service status
systemctl status systemd-journald

# View journal disk usage
journalctl --disk-usage

# Verify journal integrity
journalctl --verify

Basic journalctl Usage

Viewing Logs

# View all logs (oldest first)
journalctl

# View logs in reverse order (newest first)
journalctl -r

# Follow log output in real-time
journalctl -f

# Show most recent entries
journalctl -n 50

# View logs from current boot
journalctl -b

# View logs from previous boot
journalctl -b -1

# List all boots
journalctl --list-boots

Output Formatting

# Different output formats
journalctl -o short          # Default syslog style
journalctl -o short-iso      # ISO 8601 timestamps
journalctl -o short-precise  # Microsecond precision
journalctl -o short-monotonic # Monotonic timestamps
journalctl -o verbose        # Full structured data
journalctl -o export         # Export format
journalctl -o json           # JSON format
journalctl -o json-pretty    # Pretty-printed JSON
journalctl -o cat            # Message only

# Disable pager
journalctl --no-pager

# Output to file
journalctl > system_logs.txt

# Export in binary format
journalctl -o export > journal_export.log

Time-Based Filtering

Date and Time Filters

# Logs since specific date
journalctl --since "2024-07-19"

# Logs until specific date
journalctl --until "2024-07-20"

# Logs within date range
journalctl --since "2024-07-19" --until "2024-07-20"

# Logs from last hour
journalctl --since "1 hour ago"

# Various time formats
journalctl --since "2024-07-19 10:00:00"
journalctl --since "10:00" --until "11:00"
journalctl --since yesterday
journalctl --since today
journalctl --since "5 minutes ago"
journalctl --since "2 days ago" --until "1 day ago"

# Combining with other filters
journalctl --since "2024-07-19" -u sshd.service

Boot-Based Filtering

# Current boot logs
journalctl -b 0
journalctl -b

# Previous boot
journalctl -b -1

# Specific boot by ID
journalctl -b 5d4851c6b0f74c17962e4685826e0c49

# List boots with timestamps
journalctl --list-boots

# Logs from all boots
journalctl --boot=all

# Filter by boot and time
journalctl -b -1 --since "10:00" --until "11:00"

Unit and Service Filtering

Service-Specific Logs

# View logs for specific service
journalctl -u nginx.service
journalctl -u sshd

# Multiple services
journalctl -u nginx.service -u php-fpm.service

# Service logs with follows
journalctl -u httpd -f

# Service logs from current boot
journalctl -b -u NetworkManager

# Pattern matching for units
journalctl -u "systemd-*"

# View available units
systemctl list-units --type=service

System Component Logs

# Kernel messages
journalctl -k
journalctl --dmesg

# Logs from specific executable
journalctl /usr/sbin/sshd
journalctl /usr/bin/sudo

# Logs by systemd facility
journalctl -t kernel
journalctl -t systemd
journalctl -t audit

Priority and Severity Filtering

Log Priorities

# Filter by priority (and above)
journalctl -p err         # Error and above
journalctl -p warning     # Warning and above
journalctl -p notice      # Notice and above
journalctl -p info        # Info and above
journalctl -p debug       # All messages

# Priority levels:
# 0: emerg    - System is unusable
# 1: alert    - Action must be taken immediately
# 2: crit     - Critical conditions
# 3: err      - Error conditions
# 4: warning  - Warning conditions
# 5: notice   - Normal but significant condition
# 6: info     - Informational messages
# 7: debug    - Debug-level messages

# Specific priority only
journalctl -p warning..warning

# Priority range
journalctl -p err..warning

# Combine with service
journalctl -u httpd -p err

Examples by Severity

# Critical system errors
journalctl -p 0..3 --since today

# All warnings from specific service
journalctl -u mysql.service -p warning

# Debug messages for troubleshooting
journalctl -u NetworkManager -p debug --since "10 minutes ago"

# Emergency and alert messages only
journalctl -p 0..1

Advanced Filtering

Field-Based Filtering

# Filter by PID
journalctl _PID=1234

# Filter by UID
journalctl _UID=1000

# Filter by GID
journalctl _GID=100

# Filter by hostname
journalctl _HOSTNAME=server01

# Filter by command
journalctl _COMM=sshd

# Filter by executable path
journalctl _EXE=/usr/bin/ssh

# Filter by systemd unit
journalctl _SYSTEMD_UNIT=sshd.service

# Multiple field filters
journalctl _SYSTEMD_UNIT=httpd.service _PID=5678

# Filter by transport
journalctl _TRANSPORT=kernel
journalctl _TRANSPORT=syslog
journalctl _TRANSPORT=journal

Custom Field Queries

# Show available fields
journalctl --fields

# Show values for specific field
journalctl -F _SYSTEMD_UNIT
journalctl -F _COMM
journalctl -F PRIORITY

# Complex field matching
journalctl SYSLOG_FACILITY=4 # auth facility
journalctl SYSLOG_FACILITY=10 # authpriv facility

# Message content filtering
journalctl MESSAGE="*failed*"

# Custom application fields
journalctl MY_APP_FIELD=value

Using grep with journalctl

# Basic grep filtering
journalctl | grep -i error
journalctl | grep -i "failed password"

# Grep with context
journalctl | grep -A 5 -B 5 "kernel panic"

# Multiple patterns
journalctl | grep -E "(error|warning|critical)"

# Inverse matching
journalctl | grep -v debug

# Count occurrences
journalctl -u sshd | grep -c "Failed password"

User and Session Logs

User-Specific Logs

# Current user logs
journalctl --user

# Specific user logs (requires privileges)
journalctl _UID=1000

# User session logs
loginctl list-sessions
journalctl --user-unit=session-3.scope

# User service logs
journalctl --user -u myapp.service

# All user units
journalctl --user --all

# Login/logout events
journalctl -u systemd-logind

# SSH sessions
journalctl -u sshd | grep "session opened"

# Failed login attempts
journalctl _SYSTEMD_UNIT=sshd.service | grep "Failed"

# Authentication logs
journalctl -t sshd -t sudo -t su

# User switching events
journalctl _COMM=su
journalctl _COMM=sudo

Log Persistence and Rotation

Configuring Persistence

# Edit journal configuration
sudo vim /etc/systemd/journald.conf

# Key settings:
[Journal]
Storage=persistent        # Options: volatile, persistent, auto, none
Compress=yes             # Compress journal files
Seal=yes                 # Enable FSS sealing
SplitMode=uid           # Split by uid

# Size limits
SystemMaxUse=1G         # Maximum disk usage for system
SystemKeepFree=1G       # Keep free space
SystemMaxFileSize=100M  # Maximum individual file size
RuntimeMaxUse=100M      # Maximum runtime storage
RuntimeKeepFree=100M    # Runtime free space
RuntimeMaxFileSize=10M  # Runtime file size

# Time limits
MaxRetentionSec=1month  # Maximum retention time
MaxFileSec=1week        # Maximum file age

# Apply changes
sudo systemctl restart systemd-journald

Manual Maintenance

# Rotate journal files
sudo journalctl --rotate

# Vacuum by size
sudo journalctl --vacuum-size=500M

# Vacuum by time
sudo journalctl --vacuum-time=1month

# Vacuum by number of files
sudo journalctl --vacuum-files=10

# Verify and clean corrupted files
sudo journalctl --verify
sudo journalctl --sync

# Check disk usage
journalctl --disk-usage
du -sh /var/log/journal/

Real-Time Monitoring

Following Logs

# Follow all logs
journalctl -f

# Follow specific unit
journalctl -u nginx -f

# Follow with number of lines
journalctl -f -n 100

# Follow kernel messages
journalctl -k -f

# Follow with priority filter
journalctl -f -p err

# Follow multiple units
journalctl -f -u httpd -u php-fpm

# Follow with grep
journalctl -f | grep -i error

Monitoring Scripts

# Create monitoring script
cat > /usr/local/bin/monitor-errors.sh << 'EOF'
#!/bin/bash
# Monitor system errors in real-time

echo "Monitoring system errors..."
journalctl -f -p err --no-pager | while read line; do
    echo "[ERROR] $line"
    # Send alert (example)
    # echo "$line" | mail -s "System Error on $(hostname)" [email protected]
done
EOF

chmod +x /usr/local/bin/monitor-errors.sh

# Monitor specific patterns
cat > /usr/local/bin/monitor-failures.sh << 'EOF'
#!/bin/bash
# Monitor authentication failures

journalctl -f | grep -i "failed\|error\|denied" | while read line; do
    if echo "$line" | grep -q "authentication failure"; then
        echo "[AUTH FAILURE] $line"
        logger -p auth.warning "Authentication failure detected: $line"
    fi
done
EOF

chmod +x /usr/local/bin/monitor-failures.sh

Analyzing Logs

Statistical Analysis

# Count messages by priority
for p in {0..7}; do
    count=$(journalctl -p $p..$p --no-pager | wc -l)
    echo "Priority $p: $count messages"
done

# Top services by log volume
journalctl --no-pager | grep -oP '_SYSTEMD_UNIT=\K[^ ]+' | sort | uniq -c | sort -rn | head -20

# Errors per hour
for hour in {00..23}; do
    count=$(journalctl --since "today $hour:00" --until "today $hour:59" -p err --no-pager | wc -l)
    echo "Hour $hour: $count errors"
done

# Most common error messages
journalctl -p err --no-pager | grep -oP 'MESSAGE=\K.*' | sort | uniq -c | sort -rn | head -20

Pattern Detection

# Find recurring issues
cat > /usr/local/bin/analyze-patterns.sh << 'EOF'
#!/bin/bash
# Analyze log patterns

echo "=== System Error Patterns ==="
journalctl -p err --since "1 week ago" --no-pager | \
    grep -oP 'MESSAGE=\K.*' | \
    sed 's/[0-9]\+/NUM/g' | \
    sort | uniq -c | sort -rn | head -20

echo -e "\n=== Failed Services ==="
journalctl --since "1 week ago" --no-pager | \
    grep "Failed to start" | \
    grep -oP 'Failed to start \K.*' | \
    sort | uniq -c | sort -rn

echo -e "\n=== Authentication Failures by IP ==="
journalctl _SYSTEMD_UNIT=sshd.service --since "1 week ago" --no-pager | \
    grep "Failed password" | \
    grep -oP 'from \K[0-9.]+' | \
    sort | uniq -c | sort -rn | head -20
EOF

chmod +x /usr/local/bin/analyze-patterns.sh

Integration with Other Tools

Exporting for Analysis

# Export to CSV for analysis
cat > /usr/local/bin/export-logs-csv.sh << 'EOF'
#!/bin/bash
# Export logs to CSV format

echo "timestamp,priority,unit,message" > logs.csv
journalctl -o json --since today --no-pager | \
    jq -r '. | [.__REALTIME_TIMESTAMP, .PRIORITY, ._SYSTEMD_UNIT, .MESSAGE] | @csv' >> logs.csv
EOF

chmod +x /usr/local/bin/export-logs-csv.sh

# Export specific timeframe
journalctl --since "2024-07-19 00:00" --until "2024-07-19 23:59" -o export > daily_logs.export

# Convert to syslog format
journalctl -o short --no-pager > syslog_format.log

Remote Logging

# Configure remote logging
cat >> /etc/systemd/journald.conf << 'EOF'
[Journal]
ForwardToSyslog=yes
MaxLevelSyslog=debug
EOF

# Configure rsyslog forwarding
cat > /etc/rsyslog.d/remote.conf << 'EOF'
# Forward all logs to remote server
*.* @@remote-log-server:514

# Forward only errors
*.err @@remote-log-server:514
EOF

# Systemd-journal-remote setup
sudo dnf install systemd-journal-gateway

# Configure journal upload
cat > /etc/systemd/journal-upload.conf << 'EOF'
[Upload]
URL=https://journal-server:19532
ServerKeyFile=/etc/ssl/private/journal-upload.key
ServerCertificateFile=/etc/ssl/certs/journal-upload.pem
TrustedCertificateFile=/etc/ssl/certs/ca.pem
EOF

Troubleshooting with journalctl

Boot Issues

# Analyze boot time
systemd-analyze
systemd-analyze blame
systemd-analyze critical-chain

# Check for boot errors
journalctl -b -p err

# Failed services during boot
journalctl -b -u systemd-modules-load
systemctl --failed

# Kernel issues during boot
journalctl -b -k -p err

# Boot process timeline
journalctl -b -o short-monotonic

Service Troubleshooting

# Why service failed
systemctl status problematic.service
journalctl -u problematic.service -p err

# Service start attempts
journalctl -u nginx.service | grep -E "(Starting|Started|Failed)"

# Service dependencies
systemctl list-dependencies nginx.service
journalctl -u nginx.service --catalog

# Detailed service debugging
systemctl edit problematic.service
# Add under [Service]:
# Environment="SYSTEMD_LOG_LEVEL=debug"

# View service logs with context
journalctl -u httpd.service --since "5 minutes ago" -p debug

Performance Issues

# High CPU usage investigation
journalctl -f _TRANSPORT=kernel | grep -i cpu

# Memory issues
journalctl | grep -i "out of memory"
journalctl -k | grep -i "oom-killer"

# Disk I/O issues
journalctl -u systemd-journald | grep -i "failed to write"
journalctl | grep -i "i/o error"

# Network issues
journalctl -u NetworkManager -p err
journalctl | grep -i "link down\|link up"

Security Auditing

Authentication Monitoring

# All authentication attempts
journalctl -t sshd -t login -t systemd-logind

# Failed SSH attempts
journalctl _SYSTEMD_UNIT=sshd.service | grep "Failed password"

# Successful logins
journalctl _SYSTEMD_UNIT=sshd.service | grep "Accepted"

# Sudo usage
journalctl _COMM=sudo

# User account changes
journalctl -u systemd-logind | grep -E "(New user|User logged|Session opened)"

# SELinux denials
journalctl _TRANSPORT=audit | grep -i denied

Security Event Analysis

# Create security monitoring script
cat > /usr/local/bin/security-audit.sh << 'EOF'
#!/bin/bash
# Security audit script using journalctl

REPORT="/var/log/security-audit-$(date +%Y%m%d).log"

echo "Security Audit Report - $(date)" > $REPORT
echo "=================================" >> $REPORT

echo -e "\n[Failed Login Attempts]" >> $REPORT
journalctl --since today _SYSTEMD_UNIT=sshd.service | \
    grep "Failed password" | wc -l >> $REPORT

echo -e "\n[Successful Root Logins]" >> $REPORT
journalctl --since today _UID=0 | \
    grep "session opened" >> $REPORT

echo -e "\n[Sudo Commands Executed]" >> $REPORT
journalctl --since today _COMM=sudo | \
    grep COMMAND >> $REPORT

echo -e "\n[SELinux Denials]" >> $REPORT
journalctl --since today | \
    grep -i "avc:.*denied" >> $REPORT

echo -e "\n[System Reboots]" >> $REPORT
journalctl --list-boots | tail -5 >> $REPORT

echo -e "\n[Service Failures]" >> $REPORT
journalctl --since today -p err _TRANSPORT=systemd >> $REPORT

mail -s "Daily Security Audit - $(hostname)" [email protected] < $REPORT
EOF

chmod +x /usr/local/bin/security-audit.sh

Custom Log Integration

Application Logging

# Send custom logs to journal
echo "Application started" | systemd-cat -t myapp -p info

# With priority
echo "Critical error occurred" | systemd-cat -t myapp -p err

# From script
cat > /usr/local/bin/myapp.sh << 'EOF'
#!/bin/bash
# Application with journal logging

APP_NAME="myapp"

log_info() {
    echo "$1" | systemd-cat -t $APP_NAME -p info
}

log_error() {
    echo "$1" | systemd-cat -t $APP_NAME -p err
}

log_debug() {
    echo "$1" | systemd-cat -t $APP_NAME -p debug
}

# Application logic
log_info "Application starting..."
if ! do_something; then
    log_error "Failed to do something"
    exit 1
fi
log_info "Application completed successfully"
EOF

# Python logging to journal
cat > log_to_journal.py << 'EOF'
#!/usr/bin/env python3
from systemd import journal

# Simple logging
journal.write("Hello from Python app")

# With priority and fields
journal.send(
    "Application error occurred",
    PRIORITY=journal.Priority.ERROR,
    MY_APP_FIELD="custom_value",
    SYSLOG_IDENTIFIER="python_app"
)
EOF

Structured Logging

# Send structured data
systemd-cat -t myapp << EOF
MESSAGE=Application event occurred
PRIORITY=6
MY_CUSTOM_FIELD=value123
REQUEST_ID=abc-123-def
USER_ID=1000
EOF

# From application
logger -t myapp --journal "MESSAGE=User login" "USER_ID=1000" "SESSION_ID=xyz789"

# Query structured data
journalctl MY_CUSTOM_FIELD=value123
journalctl REQUEST_ID=abc-123-def

Performance Optimization

Query Optimization

# Use specific time ranges
journalctl --since "10 minutes ago" --until now

# Limit output size
journalctl -n 1000 --no-pager

# Use indexes efficiently
journalctl -u specific.service  # Faster than grep

# Export for external processing
journalctl -o export | process_logs

# Avoid excessive regex
journalctl _SYSTEMD_UNIT=sshd.service  # Better than grep

Resource Management

# Monitor journal performance
systemctl status systemd-journald
journalctl -u systemd-journald

# Check journal I/O
iotop -p $(pgrep systemd-journal)

# Limit journal CPU usage
systemctl edit systemd-journald
# Add:
# [Service]
# CPUQuota=20%

# Optimize for specific use cases
# For high-volume logging:
cat >> /etc/systemd/journald.conf << 'EOF'
RateLimitIntervalSec=0
RateLimitBurst=0
SyncIntervalSec=5m
EOF

Best Practices

1. Regular Maintenance

# Create maintenance script
cat > /usr/local/bin/journal-maintenance.sh << 'EOF'
#!/bin/bash
# Journal maintenance script

echo "Starting journal maintenance - $(date)"

# Rotate logs
journalctl --rotate

# Vacuum old logs
journalctl --vacuum-time=1month

# Verify integrity
journalctl --verify

# Check disk usage
echo "Current journal disk usage:"
journalctl --disk-usage

# Archive old logs if needed
if [ -d /var/log/journal ]; then
    find /var/log/journal -name "*.journal" -mtime +90 -exec gzip {} \;
fi

echo "Journal maintenance completed - $(date)"
EOF

chmod +x /usr/local/bin/journal-maintenance.sh

# Add to cron
echo "0 2 * * 0 /usr/local/bin/journal-maintenance.sh" | crontab -

2. Monitoring Setup

# Create comprehensive monitoring
cat > /usr/local/bin/log-monitor.sh << 'EOF'
#!/bin/bash
# Continuous log monitoring

monitor_errors() {
    journalctl -f -p err --no-pager | while read line; do
        echo "[ERROR] $line" | tee -a /var/log/monitor-errors.log
        # Alert on critical errors
        if echo "$line" | grep -q "CRITICAL\|FATAL\|PANIC"; then
            echo "$line" | mail -s "Critical Error on $(hostname)" [email protected]
        fi
    done
}

monitor_security() {
    journalctl -f | grep -E "(Failed password|authentication failure|unauthorized)" | \
    while read line; do
        echo "[SECURITY] $line" | tee -a /var/log/monitor-security.log
    done
}

# Run monitors in background
monitor_errors &
monitor_security &

wait
EOF

chmod +x /usr/local/bin/log-monitor.sh

3. Documentation

# Document important queries
cat > /etc/journalctl-queries.md << 'EOF'
# Common journalctl Queries

## System Health
- Recent errors: `journalctl -p err --since "1 hour ago"`
- Failed services: `systemctl --failed`
- Boot issues: `journalctl -b -p err`

## Security
- Failed logins: `journalctl _SYSTEMD_UNIT=sshd.service | grep Failed`
- Sudo usage: `journalctl _COMM=sudo`
- SELinux denials: `journalctl | grep "avc:.*denied"`

## Performance
- Slow boot: `systemd-analyze blame`
- High load: `journalctl --since "1 hour ago" | grep -i "high load"`
- OOM events: `journalctl -k | grep -i oom-killer`

## Application Specific
- Web server: `journalctl -u httpd -u nginx --since today`
- Database: `journalctl -u mariadb -u postgresql --since today`
- Custom app: `journalctl -t myapp --since today`
EOF

Conclusion

The journalctl command and systemd journal provide powerful capabilities for log management in AlmaLinux. From basic log viewing to advanced filtering and analysis, mastering journalctl is essential for effective system administration, troubleshooting, and security monitoring.

Key takeaways:

Use appropriate filters to find relevant information quickly
Configure persistence and rotation based on your needs
Implement regular monitoring and maintenance procedures
Leverage structured logging for better analysis
Combine journalctl with other tools for comprehensive monitoring

With these skills, you can efficiently manage system logs, quickly diagnose issues, and maintain comprehensive audit trails for your AlmaLinux systems.