Introduction
SSH (Secure Shell) is the cornerstone of secure remote system administration in Linux environments. Properly configuring SSH on AlmaLinux is crucial for maintaining system security while ensuring reliable remote access. This comprehensive guide covers everything from basic SSH setup to advanced security configurations, including key-based authentication, port knocking, intrusion prevention, and enterprise-grade hardening techniques.
Understanding SSH Architecture
SSH Components and Protocol
SSH operates using a client-server architecture with several key components:
- SSH Server (sshd): Daemon listening for connections
- SSH Client: Program connecting to SSH servers
- Authentication Methods: Password, public key, GSSAPI
- Encryption: Symmetric and asymmetric encryption
- Port Forwarding: TCP tunneling capabilities
- File Transfer: SFTP and SCP protocols
SSH Protocol Versions
# Check SSH version
ssh -V
sshd -V
# Verify protocol support
grep "Protocol" /etc/ssh/sshd_config
# Ensure only Protocol 2 is enabled (Protocol 1 is deprecated)
echo "Protocol 2" >> /etc/ssh/sshd_config.d/01-protocol.conf
Basic SSH Installation and Setup
Installing SSH Server
# Install OpenSSH server and client
dnf install -y openssh openssh-server openssh-clients
# Install additional SSH utilities
dnf install -y openssh-askpass openssh-cavs openssh-ldap
# Enable and start SSH service
systemctl enable --now sshd
# Verify SSH service status
systemctl status sshd
# Check SSH port binding
ss -tlnp | grep :22
netstat -tlnp | grep :22
Initial SSH Configuration
# Backup original configuration
cp /etc/ssh/sshd_config /etc/ssh/sshd_config.original
cp -r /etc/ssh/sshd_config.d /etc/ssh/sshd_config.d.original
# Create modular configuration directory
mkdir -p /etc/ssh/sshd_config.d
# Basic secure configuration
cat > /etc/ssh/sshd_config.d/10-basic-security.conf << 'EOF'
# Basic Security Configuration
# Disable root login
PermitRootLogin no
# Enable public key authentication
PubkeyAuthentication yes
# Set authentication methods
AuthenticationMethods publickey
# Disable password authentication after setting up keys
PasswordAuthentication yes
PermitEmptyPasswords no
# Challenge-response authentication
ChallengeResponseAuthentication no
# Kerberos authentication
KerberosAuthentication no
GSSAPIAuthentication no
# Host-based authentication
HostbasedAuthentication no
IgnoreRhosts yes
IgnoreUserKnownHosts yes
# PAM authentication
UsePAM yes
# Disable forwarding
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
PermitTunnel no
# Disable user environment
PermitUserEnvironment no
# Session settings
MaxAuthTries 3
MaxSessions 10
ClientAliveInterval 300
ClientAliveCountMax 2
LoginGraceTime 60
# Logging
SyslogFacility AUTH
LogLevel INFO
EOF
# Test configuration
sshd -t
# Reload SSH service
systemctl reload sshd
SSH Key-Based Authentication
Generating SSH Keys
# Generate SSH key pairs for users
# RSA key (4096 bits)
ssh-keygen -t rsa -b 4096 -C "user@almalinux-server" -f ~/.ssh/id_rsa
# Ed25519 key (recommended)
ssh-keygen -t ed25519 -C "user@almalinux-server" -f ~/.ssh/id_ed25519
# ECDSA key
ssh-keygen -t ecdsa -b 521 -C "user@almalinux-server" -f ~/.ssh/id_ecdsa
# Generate key with custom options
ssh-keygen -t ed25519 \
-C "$(whoami)@$(hostname)-$(date +%Y%m%d)" \
-f ~/.ssh/id_ed25519_almalinux \
-N "passphrase_here"
# Set proper permissions
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_*
chmod 644 ~/.ssh/id_*.pub
Managing Authorized Keys
# Create authorized keys file
touch ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
# Add public key to authorized_keys
cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
# Script to manage SSH keys
cat > /usr/local/bin/ssh-key-manager.sh << 'EOF'
#!/bin/bash
# SSH Key Management Script
ACTION=$1
USERNAME=$2
KEYFILE=$3
case $ACTION in
add)
if [[ -z "$USERNAME" ]] || [[ -z "$KEYFILE" ]]; then
echo "Usage: $0 add <username> <public-key-file>"
exit 1
fi
# Create .ssh directory if it doesn't exist
USER_HOME=$(getent passwd "$USERNAME" | cut -d: -f6)
mkdir -p "$USER_HOME/.ssh"
touch "$USER_HOME/.ssh/authorized_keys"
# Add key
cat "$KEYFILE" >> "$USER_HOME/.ssh/authorized_keys"
# Set permissions
chown -R "$USERNAME:$USERNAME" "$USER_HOME/.ssh"
chmod 700 "$USER_HOME/.ssh"
chmod 600 "$USER_HOME/.ssh/authorized_keys"
echo "Key added for user $USERNAME"
;;
remove)
if [[ -z "$USERNAME" ]] || [[ -z "$KEYFILE" ]]; then
echo "Usage: $0 remove <username> <key-comment>"
exit 1
fi
USER_HOME=$(getent passwd "$USERNAME" | cut -d: -f6)
sed -i "/$KEYFILE/d" "$USER_HOME/.ssh/authorized_keys"
echo "Key removed for user $USERNAME"
;;
list)
if [[ -z "$USERNAME" ]]; then
echo "Usage: $0 list <username>"
exit 1
fi
USER_HOME=$(getent passwd "$USERNAME" | cut -d: -f6)
echo "SSH keys for $USERNAME:"
cat "$USER_HOME/.ssh/authorized_keys" 2>/dev/null || echo "No keys found"
;;
*)
echo "Usage: $0 {add|remove|list} <username> [keyfile|comment]"
exit 1
;;
esac
EOF
chmod +x /usr/local/bin/ssh-key-manager.sh
SSH Agent Configuration
# Configure SSH agent
cat > ~/.bash_profile << 'EOF'
# SSH Agent Configuration
if [ -z "$SSH_AUTH_SOCK" ]; then
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_ed25519
fi
EOF
# Systemd user service for SSH agent
mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/ssh-agent.service << 'EOF'
[Unit]
Description=SSH key agent
Before=graphical-session-pre.target
Wants=graphical-session-pre.target
[Service]
Type=simple
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket
ExecStart=/usr/bin/ssh-agent -D -a $SSH_AUTH_SOCK
ExecStop=/usr/bin/ssh-agent -k
[Install]
WantedBy=default.target
EOF
# Enable SSH agent service
systemctl --user enable ssh-agent
systemctl --user start ssh-agent
Advanced SSH Security Configuration
Hardened SSH Configuration
# Create comprehensive security configuration
cat > /etc/ssh/sshd_config.d/20-hardening.conf << 'EOF'
# Advanced SSH Hardening Configuration
# Supported host key algorithms
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
# Key exchange algorithms
KexAlgorithms curve25519-sha256,[email protected],diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
# Ciphers
Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
# Message authentication codes
MACs [email protected],[email protected],[email protected]
# Host key algorithms
HostKeyAlgorithms ssh-ed25519,[email protected],rsa-sha2-512,rsa-sha2-256
# Pubkey accepted algorithms
PubkeyAcceptedAlgorithms ssh-ed25519,[email protected],rsa-sha2-512,rsa-sha2-256
# Compression
Compression no
# DNS
UseDNS no
# Strict modes
StrictModes yes
# Privilege separation
UsePrivilegeSeparation sandbox
# Subsystems
Subsystem sftp /usr/libexec/openssh/sftp-server -f AUTHPRIV -l INFO
# Chroot directory (for SFTP)
#ChrootDirectory /home/%u
#ForceCommand internal-sftp
# Allow/Deny directives
AllowUsers *@10.0.0.0/8 *@172.16.0.0/12 *@192.168.0.0/16
DenyUsers root daemon bin sys sync games man lp mail news uucp proxy
AllowGroups sshusers admins
DenyGroups root
# Banner
Banner /etc/ssh/banner.txt
# Debian banner
DebianBanner no
EOF
# Create SSH banner
cat > /etc/ssh/banner.txt << 'EOF'
*******************************************************************
* AUTHORIZED ACCESS ONLY *
*******************************************************************
* This system is for the use of authorized users only. *
* Individuals using this computer system without authority, or in *
* excess of their authority, are subject to having all of their *
* activities on this system monitored and recorded by system *
* personnel. *
* *
* By accessing this system, you consent to this monitoring. *
* Evidence of unauthorized access or criminal activity will be *
* provided to law enforcement officials. *
*******************************************************************
EOF
# Set banner permissions
chmod 644 /etc/ssh/banner.txt
Implementing Port Knocking
# Install knockd
dnf install -y epel-release
dnf install -y knock-server
# Configure knockd
cat > /etc/knockd.conf << 'EOF'
[options]
UseSyslog
LogFile = /var/log/knockd.log
Interface = enp0s3
[openSSH]
sequence = 7000,8000,9000
seq_timeout = 30
command = /usr/sbin/iptables -A INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
tcpflags = syn
[closeSSH]
sequence = 9000,8000,7000
seq_timeout = 30
command = /usr/sbin/iptables -D INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
tcpflags = syn
[openCustom]
sequence = 2222,3333,4444
seq_timeout = 30
command = /usr/sbin/iptables -A INPUT -s %IP% -p tcp --dport 2222 -j ACCEPT
tcpflags = syn
cmd_timeout = 30
stop_command = /usr/sbin/iptables -D INPUT -s %IP% -p tcp --dport 2222 -j ACCEPT
EOF
# Create systemd service for knockd
cat > /etc/systemd/system/knockd.service << 'EOF'
[Unit]
Description=Port-Knock Daemon
After=network.target
Wants=network.target
[Service]
Type=simple
ExecStart=/usr/sbin/knockd -D
ExecReload=/bin/kill -HUP $MAINPID
KillMode=mixed
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
# Enable and start knockd
systemctl enable --now knockd
# Configure firewall for port knocking
firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -p tcp --dport 22 -j REJECT
firewall-cmd --reload
# Client-side knock script
cat > /usr/local/bin/knock-ssh.sh << 'EOF'
#!/bin/bash
# SSH port knocking client script
SERVER=$1
if [[ -z "$SERVER" ]]; then
echo "Usage: $0 <server-ip>"
exit 1
fi
echo "Knocking on $SERVER..."
for port in 7000 8000 9000; do
nc -z -w 1 $SERVER $port
sleep 0.5
done
echo "Connecting to SSH..."
ssh $SERVER
EOF
chmod +x /usr/local/bin/knock-ssh.sh
Two-Factor Authentication Setup
# Install Google Authenticator
dnf install -y google-authenticator qrencode
# Configure PAM for 2FA
cat > /etc/pam.d/sshd-2fa << 'EOF'
#%PAM-1.0
auth required pam_google_authenticator.so
auth required pam_sepermit.so
auth substack password-auth
auth include postlogin
# Used with polkit to reauthorize users in remote sessions
-auth optional pam_reauthorize.so prepare
account required pam_nologin.so
account include password-auth
password include password-auth
# pam_selinux.so close should be the first session rule
session required pam_selinux.so close
session required pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session required pam_selinux.so open env_params
session required pam_namespace.so
session optional pam_keyinit.so force revoke
session include password-auth
session include postlogin
# Used with polkit to reauthorize users in remote sessions
-session optional pam_reauthorize.so prepare
EOF
# Update SSH configuration for 2FA
cat > /etc/ssh/sshd_config.d/30-two-factor.conf << 'EOF'
# Two-Factor Authentication Configuration
# Enable challenge-response for 2FA
ChallengeResponseAuthentication yes
# Authentication methods for 2FA
# Require both publickey and keyboard-interactive
AuthenticationMethods publickey,keyboard-interactive
# Use custom PAM configuration
UsePAM yes
EOF
# Setup script for users
cat > /usr/local/bin/setup-2fa.sh << 'EOF'
#!/bin/bash
# Setup 2FA for SSH users
USER=$1
if [[ -z "$USER" ]]; then
echo "Usage: $0 <username>"
exit 1
fi
echo "Setting up 2FA for user: $USER"
su - $USER -c "google-authenticator -t -d -f -r 3 -R 30 -w 3"
echo "2FA setup complete for $USER"
echo "User should scan the QR code with their authenticator app"
EOF
chmod +x /usr/local/bin/setup-2fa.sh
Intrusion Prevention with Fail2ban
Installing and Configuring Fail2ban
# Install fail2ban
dnf install -y fail2ban fail2ban-systemd
# Create local configuration
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
# Configure fail2ban for SSH
cat > /etc/fail2ban/jail.d/sshd.local << 'EOF'
[DEFAULT]
# Ban IP for 1 hour
bantime = 3600
# Consider 5 failures in 10 minutes
findtime = 600
maxretry = 5
# Email notifications
destemail = root@localhost
sender = fail2ban@localhost
mta = sendmail
# Action
action = %(action_mwl)s
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/secure
maxretry = 3
bantime = 7200
findtime = 600
[sshd-ddos]
enabled = true
port = ssh
filter = sshd-ddos
logpath = /var/log/secure
maxretry = 10
bantime = 3600
findtime = 600
[sshd-aggressive]
enabled = true
port = ssh
filter = sshd[mode=aggressive]
logpath = /var/log/secure
maxretry = 2
bantime = 86400
findtime = 3600
EOF
# Create custom SSH filter
cat > /etc/fail2ban/filter.d/sshd-custom.conf << 'EOF'
[Definition]
failregex = ^%(__prefix_line)sFailed password for .* from <HOST> port \d+ ssh2$
^%(__prefix_line)sConnection closed by <HOST> port \d+ \[preauth\]$
^%(__prefix_line)sReceived disconnect from <HOST> port \d+:\d+: .*: Auth fail$
^%(__prefix_line)sDisconnecting: Too many authentication failures for .* from <HOST> port \d+$
^%(__prefix_line)sUnable to negotiate with <HOST> port \d+: no matching .* found$
^%(__prefix_line)sConnection reset by <HOST> port \d+$
^%(__prefix_line)sDid not receive identification string from <HOST>$
ignoreregex =
[Init]
maxlines = 10
EOF
# Configure fail2ban service
systemctl enable --now fail2ban
# Check fail2ban status
fail2ban-client status
fail2ban-client status sshd
Advanced Fail2ban Actions
# Create custom action for persistent bans
cat > /etc/fail2ban/action.d/iptables-persistent.conf << 'EOF'
[Definition]
actionstart = <iptables> -N f2b-<name>
<iptables> -A f2b-<name> -j <returntype>
<iptables> -I <chain> -p <protocol> --dport <port> -j f2b-<name>
actionstop = <iptables> -D <chain> -p <protocol> --dport <port> -j f2b-<name>
<actionflush>
<iptables> -X f2b-<name>
actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]'
actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>
echo "<ip> # banned on $(date)" >> /etc/fail2ban/persistent-bans.txt
actionunban = <iptables> -D f2b-<name> -s <ip> -j <blocktype>
sed -i "/<ip>/d" /etc/fail2ban/persistent-bans.txt
EOF
# Create script to restore persistent bans
cat > /usr/local/bin/restore-fail2ban-bans.sh << 'EOF'
#!/bin/bash
# Restore persistent fail2ban bans
BANFILE="/etc/fail2ban/persistent-bans.txt"
if [[ -f "$BANFILE" ]]; then
while read -r line; do
IP=$(echo "$line" | awk '{print $1}')
if [[ "$IP" =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
iptables -I f2b-sshd 1 -s "$IP" -j REJECT
fi
done < "$BANFILE"
fi
EOF
chmod +x /usr/local/bin/restore-fail2ban-bans.sh
# Add to systemd service
mkdir -p /etc/systemd/system/fail2ban.service.d
cat > /etc/systemd/system/fail2ban.service.d/override.conf << 'EOF'
[Service]
ExecStartPost=/usr/local/bin/restore-fail2ban-bans.sh
EOF
systemctl daemon-reload
systemctl restart fail2ban
SSH Tunneling and Port Forwarding
Secure Tunneling Configuration
# Configure SSH tunneling options
cat > /etc/ssh/sshd_config.d/40-tunneling.conf << 'EOF'
# SSH Tunneling Configuration
# Allow specific forwarding
AllowTcpForwarding yes
AllowAgentForwarding no
AllowStreamLocalForwarding yes
GatewayPorts no
# Permit tunnel
PermitTunnel no
# X11 forwarding (disabled for security)
X11Forwarding no
X11DisplayOffset 10
X11UseLocalhost yes
# Port forwarding restrictions
PermitOpen localhost:* 10.0.0.0/8:* 172.16.0.0/12:* 192.168.0.0/16:*
# Match specific users for forwarding
Match Group sftpusers
AllowTcpForwarding no
X11Forwarding no
ForceCommand internal-sftp
Match Group tunnelers
AllowTcpForwarding yes
PermitTunnel yes
EOF
# Create groups for different access levels
groupadd -r sftpusers
groupadd -r tunnelers
# Example secure tunnel script
cat > /usr/local/bin/ssh-tunnel.sh << 'EOF'
#!/bin/bash
# Secure SSH tunnel creation script
ACTION=$1
TYPE=$2
LOCAL_PORT=$3
REMOTE_HOST=$4
REMOTE_PORT=$5
SSH_HOST=$6
case $ACTION in
create)
case $TYPE in
local)
ssh -f -N -L ${LOCAL_PORT}:${REMOTE_HOST}:${REMOTE_PORT} ${SSH_HOST}
echo "Local tunnel created: localhost:${LOCAL_PORT} -> ${REMOTE_HOST}:${REMOTE_PORT}"
;;
remote)
ssh -f -N -R ${REMOTE_PORT}:localhost:${LOCAL_PORT} ${SSH_HOST}
echo "Remote tunnel created: ${SSH_HOST}:${REMOTE_PORT} -> localhost:${LOCAL_PORT}"
;;
dynamic)
ssh -f -N -D ${LOCAL_PORT} ${SSH_HOST}
echo "Dynamic tunnel (SOCKS) created on port ${LOCAL_PORT}"
;;
esac
;;
list)
ps aux | grep "ssh -f -N" | grep -v grep
;;
kill)
pkill -f "ssh -f -N"
echo "All SSH tunnels terminated"
;;
*)
echo "Usage: $0 {create|list|kill} {local|remote|dynamic} [options]"
exit 1
;;
esac
EOF
chmod +x /usr/local/bin/ssh-tunnel.sh
SSH Session Management
Session Recording and Monitoring
# Install session recording tools
dnf install -y tlog
# Configure tlog for SSH session recording
cat > /etc/sssd/conf.d/sssd-session-recording.conf << 'EOF'
[session_recording]
scope = all
users =
groups = admins, sshusers
EOF
# Create tlog configuration
cat > /etc/tlog/tlog-rec-session.conf << 'EOF'
{
"shell": "/bin/bash",
"notice": "ATTENTION: Your session is being recorded!",
"writer": "journal",
"journal": {
"priority": "info",
"augment": true
},
"limit": {
"rate": 16384,
"burst": 32768,
"action": "drop"
},
"file": {
"path": "/var/log/tlog-rec"
}
}
EOF
# Configure shell for recording
cat >> /etc/profile.d/session-recording.sh << 'EOF'
# Session recording notice
if [[ -n "$SSH_CLIENT" ]]; then
echo "WARNING: This SSH session is being recorded for security purposes."
fi
EOF
# Create session monitoring script
cat > /usr/local/bin/ssh-monitor.sh << 'EOF'
#!/bin/bash
# SSH session monitoring script
LOGFILE="/var/log/ssh-monitor.log"
monitor_sessions() {
echo "=== SSH Session Monitor - $(date) ===" >> "$LOGFILE"
# Active SSH sessions
who | grep pts | while read line; do
USER=$(echo $line | awk '{print $1}')
TTY=$(echo $line | awk '{print $2}')
FROM=$(echo $line | awk '{print $5}' | tr -d '()')
LOGIN_TIME=$(echo $line | awk '{print $3, $4}')
echo "User: $USER, TTY: $TTY, From: $FROM, Login: $LOGIN_TIME" >> "$LOGFILE"
# Get process information
PID=$(ps aux | grep "sshd: $USER@$TTY" | grep -v grep | awk '{print $2}')
if [[ -n "$PID" ]]; then
echo " PID: $PID" >> "$LOGFILE"
echo " Commands:" >> "$LOGFILE"
ps --ppid $PID -o comm,pid,etime >> "$LOGFILE"
fi
echo "" >> "$LOGFILE"
done
}
case $1 in
start)
while true; do
monitor_sessions
sleep 60
done
;;
once)
monitor_sessions
;;
*)
echo "Usage: $0 {start|once}"
exit 1
;;
esac
EOF
chmod +x /usr/local/bin/ssh-monitor.sh
# Create systemd service for monitoring
cat > /etc/systemd/system/ssh-monitor.service << 'EOF'
[Unit]
Description=SSH Session Monitor
After=network.target sshd.service
[Service]
Type=simple
ExecStart=/usr/local/bin/ssh-monitor.sh start
Restart=always
RestartSec=60
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now ssh-monitor
SSH Connection Limits
# Configure connection limits
cat > /etc/ssh/sshd_config.d/50-limits.conf << 'EOF'
# SSH Connection Limits
# Maximum number of concurrent unauthenticated connections
MaxStartups 10:30:60
# Maximum number of open sessions per connection
MaxSessions 10
# Maximum number of authentication attempts
MaxAuthTries 3
# Client alive settings
ClientAliveInterval 300
ClientAliveCountMax 2
# Login grace time
LoginGraceTime 60
# Limit connections per user
Match User *
MaxSessions 5
# Limit connections per group
Match Group limited
MaxSessions 2
EOF
# Create PAM limits for SSH
cat > /etc/security/limits.d/ssh-limits.conf << 'EOF'
# SSH connection limits
* hard maxlogins 5
* soft maxlogins 3
@admins hard maxlogins 10
@admins soft maxlogins 10
EOF
SSH Key Management at Scale
Centralized Key Management
# Create SSH key management system
mkdir -p /opt/ssh-key-mgmt/{keys,scripts,logs}
# Key rotation script
cat > /opt/ssh-key-mgmt/scripts/rotate-keys.sh << 'EOF'
#!/bin/bash
# SSH Key Rotation Script
KEY_DIR="/opt/ssh-key-mgmt/keys"
LOG_FILE="/opt/ssh-key-mgmt/logs/rotation-$(date +%Y%m%d).log"
log_message() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}
rotate_user_key() {
local USER=$1
local USER_HOME=$(getent passwd "$USER" | cut -d: -f6)
if [[ ! -d "$USER_HOME/.ssh" ]]; then
log_message "No .ssh directory for user $USER"
return 1
fi
# Backup existing keys
local BACKUP_DIR="$KEY_DIR/backup/$USER-$(date +%Y%m%d%H%M%S)"
mkdir -p "$BACKUP_DIR"
cp -r "$USER_HOME/.ssh" "$BACKUP_DIR/"
# Generate new key
local NEW_KEY="$KEY_DIR/active/$USER-$(date +%Y%m%d)"
ssh-keygen -t ed25519 -f "$NEW_KEY" -C "$USER@$(hostname)-rotated-$(date +%Y%m%d)" -N ""
# Deploy new key
cat "$NEW_KEY.pub" > "$USER_HOME/.ssh/authorized_keys.new"
cat "$USER_HOME/.ssh/authorized_keys" >> "$USER_HOME/.ssh/authorized_keys.new"
mv "$USER_HOME/.ssh/authorized_keys.new" "$USER_HOME/.ssh/authorized_keys"
# Set permissions
chown $USER:$USER "$USER_HOME/.ssh/authorized_keys"
chmod 600 "$USER_HOME/.ssh/authorized_keys"
log_message "Key rotated for user $USER"
}
# Main rotation logic
log_message "Starting SSH key rotation"
for USER in $(getent group sshusers | cut -d: -f4 | tr ',' ' '); do
rotate_user_key "$USER"
done
log_message "SSH key rotation completed"
EOF
chmod +x /opt/ssh-key-mgmt/scripts/rotate-keys.sh
# Key audit script
cat > /opt/ssh-key-mgmt/scripts/audit-keys.sh << 'EOF'
#!/bin/bash
# SSH Key Audit Script
REPORT_FILE="/opt/ssh-key-mgmt/logs/key-audit-$(date +%Y%m%d).txt"
echo "SSH Key Audit Report - $(date)" > "$REPORT_FILE"
echo "================================" >> "$REPORT_FILE"
echo "" >> "$REPORT_FILE"
# Check all users with SSH access
for USER_HOME in /home/*; do
if [[ -d "$USER_HOME/.ssh" ]]; then
USER=$(basename "$USER_HOME")
echo "User: $USER" >> "$REPORT_FILE"
if [[ -f "$USER_HOME/.ssh/authorized_keys" ]]; then
KEY_COUNT=$(grep -c "ssh-" "$USER_HOME/.ssh/authorized_keys" 2>/dev/null || echo 0)
echo " Authorized keys: $KEY_COUNT" >> "$REPORT_FILE"
# Check key types and ages
while IFS= read -r line; do
if [[ "$line" =~ ^ssh- ]]; then
KEY_TYPE=$(echo "$line" | awk '{print $1}')
KEY_COMMENT=$(echo "$line" | awk '{print $NF}')
echo " - Type: $KEY_TYPE, Comment: $KEY_COMMENT" >> "$REPORT_FILE"
fi
done < "$USER_HOME/.ssh/authorized_keys"
else
echo " No authorized_keys file" >> "$REPORT_FILE"
fi
echo "" >> "$REPORT_FILE"
fi
done
# Check for weak keys
echo "Checking for weak keys..." >> "$REPORT_FILE"
find /home -name "authorized_keys" -exec sh -c '
for file do
while IFS= read -r line; do
if [[ "$line" =~ ssh-rsa.*[0-9]{3}\ .* ]]; then
if [[ $(echo "$line" | awk "{print \$2}" | wc -c) -lt 372 ]]; then
echo "WEAK KEY FOUND in $file" >> "$REPORT_FILE"
fi
fi
done < "$file"
done
' sh {} \;
echo "Audit completed: $REPORT_FILE"
EOF
chmod +x /opt/ssh-key-mgmt/scripts/audit-keys.sh
SSH Certificate Authority
# Setup SSH Certificate Authority
mkdir -p /opt/ssh-ca/{ca,certs,scripts}
cd /opt/ssh-ca
# Generate CA keys
ssh-keygen -t ed25519 -f ca/ssh-ca-key -C "SSH CA Root Key" -N ""
# Create CA signing script
cat > scripts/sign-user-key.sh << 'EOF'
#!/bin/bash
# SSH User Key Signing Script
CA_KEY="/opt/ssh-ca/ca/ssh-ca-key"
CERT_DIR="/opt/ssh-ca/certs"
USER=$1
PUBKEY_FILE=$2
VALIDITY=${3:-"+52w"}
if [[ -z "$USER" ]] || [[ -z "$PUBKEY_FILE" ]]; then
echo "Usage: $0 <username> <public-key-file> [validity]"
exit 1
fi
# Create certificate
CERT_FILE="$CERT_DIR/${USER}-cert.pub"
ssh-keygen -s "$CA_KEY" \
-I "${USER}@$(hostname)" \
-n "$USER" \
-V "$VALIDITY" \
-z $(date +%s) \
"$PUBKEY_FILE"
mv "${PUBKEY_FILE}-cert.pub" "$CERT_FILE"
echo "Certificate created: $CERT_FILE"
echo "Valid for: $VALIDITY"
EOF
chmod +x scripts/sign-user-key.sh
# Configure SSH to trust CA
echo "TrustedUserCAKeys /opt/ssh-ca/ca/ssh-ca-key.pub" >> /etc/ssh/sshd_config.d/60-ca.conf
# Create host certificate
ssh-keygen -s ca/ssh-ca-key \
-I "$(hostname)" \
-h \
-n "$(hostname),$(hostname -f),$(hostname -I | tr ' ' ',')" \
-V +52w \
/etc/ssh/ssh_host_ed25519_key.pub
# Configure host certificate
echo "HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub" >> /etc/ssh/sshd_config.d/60-ca.conf
systemctl reload sshd
SSH Monitoring and Alerting
Real-time SSH Monitoring
# Create SSH monitoring system
cat > /usr/local/bin/ssh-monitor-rt.sh << 'EOF'
#!/bin/bash
# Real-time SSH monitoring script
ALERT_EMAIL="[email protected]"
WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
send_alert() {
local LEVEL=$1
local MESSAGE=$2
# Log to syslog
logger -t ssh-monitor -p auth.$LEVEL "$MESSAGE"
# Send email
echo "$MESSAGE" | mail -s "SSH Alert: $LEVEL" "$ALERT_EMAIL"
# Send to webhook
curl -X POST -H 'Content-type: application/json' \
--data "{\"text\":\"SSH Alert ($LEVEL): $MESSAGE\"}" \
"$WEBHOOK_URL" 2>/dev/null
}
monitor_auth_log() {
tail -F /var/log/secure | while read line; do
# Failed login attempts
if echo "$line" | grep -q "Failed password"; then
IP=$(echo "$line" | grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}")
USER=$(echo "$line" | grep -oE "for [^ ]+" | cut -d' ' -f2)
send_alert "warning" "Failed login attempt for $USER from $IP"
fi
# Successful login
if echo "$line" | grep -q "Accepted publickey"; then
IP=$(echo "$line" | grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}")
USER=$(echo "$line" | grep -oE "for [^ ]+" | cut -d' ' -f2)
send_alert "info" "Successful login for $USER from $IP"
fi
# Root login attempts
if echo "$line" | grep -q "Failed password for root"; then
IP=$(echo "$line" | grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}")
send_alert "critical" "Root login attempt from $IP"
fi
# Invalid users
if echo "$line" | grep -q "Invalid user"; then
USER=$(echo "$line" | grep -oE "Invalid user [^ ]+" | cut -d' ' -f3)
IP=$(echo "$line" | grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}")
send_alert "warning" "Invalid user $USER from $IP"
fi
done
}
# Start monitoring
monitor_auth_log
EOF
chmod +x /usr/local/bin/ssh-monitor-rt.sh
# Create systemd service
cat > /etc/systemd/system/ssh-monitor-rt.service << 'EOF'
[Unit]
Description=Real-time SSH Monitor
After=network.target
[Service]
Type=simple
ExecStart=/usr/local/bin/ssh-monitor-rt.sh
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now ssh-monitor-rt
SSH Usage Statistics
# Create SSH statistics script
cat > /usr/local/bin/ssh-stats.sh << 'EOF'
#!/bin/bash
# SSH Usage Statistics Script
STATS_FILE="/var/log/ssh-stats-$(date +%Y%m).log"
generate_stats() {
echo "=== SSH Usage Statistics ==="
echo "Generated: $(date)"
echo ""
# Login statistics
echo "Top 10 Users by Login Count:"
grep "Accepted publickey" /var/log/secure* | \
awk '{print $(NF-5)}' | sort | uniq -c | sort -rn | head -10
echo ""
# Source IP statistics
echo "Top 10 Source IPs:"
grep "Accepted publickey" /var/log/secure* | \
grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" | \
sort | uniq -c | sort -rn | head -10
echo ""
# Failed login statistics
echo "Failed Login Attempts by IP:"
grep "Failed password" /var/log/secure* | \
grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" | \
sort | uniq -c | sort -rn | head -10
echo ""
# Login times
echo "Login Distribution by Hour:"
grep "Accepted publickey" /var/log/secure* | \
awk '{print $3}' | cut -d: -f1 | sort | uniq -c
echo ""
# Current sessions
echo "Current Active Sessions:"
who | grep pts
echo ""
# Session duration
echo "Average Session Duration:"
last -F | grep -v "still logged in" | grep pts | \
awk '{print $NF}' | grep -E "^\(" | \
sed 's/[()]//g' | \
awk -F: '{ total += $1*60 + $2 } END { print total/NR " minutes" }'
}
# Generate and save statistics
generate_stats | tee "$STATS_FILE"
# Create HTML report
cat > /var/www/html/ssh-stats.html << EOF
<!DOCTYPE html>
<html>
<head>
<title>SSH Usage Statistics</title>
<style>
body { font-family: Arial, sans-serif; margin: 20px; }
pre { background-color: #f4f4f4; padding: 10px; border-radius: 5px; }
h2 { color: #333; }
</style>
</head>
<body>
<h1>SSH Usage Statistics</h1>
<pre>$(generate_stats)</pre>
<p>Generated: $(date)</p>
</body>
</html>
EOF
EOF
chmod +x /usr/local/bin/ssh-stats.sh
# Schedule statistics generation
echo "0 0 * * * root /usr/local/bin/ssh-stats.sh" >> /etc/crontab
SSH Compliance and Auditing
Security Compliance Checks
# Create SSH compliance checker
cat > /usr/local/bin/ssh-compliance-check.sh << 'EOF'
#!/bin/bash
# SSH Security Compliance Checker
REPORT_FILE="/var/log/ssh-compliance-$(date +%Y%m%d).txt"
PASS="✓"
FAIL="✗"
check_config() {
local CONFIG=$1
local VALUE=$2
local EXPECTED=$3
local DESCRIPTION=$4
CURRENT=$(sshd -T | grep "^$CONFIG" | awk '{print $2}')
if [[ "$CURRENT" == "$EXPECTED" ]]; then
echo "$PASS $DESCRIPTION ($CONFIG = $EXPECTED)"
else
echo "$FAIL $DESCRIPTION (Expected: $EXPECTED, Found: $CURRENT)"
fi
}
echo "SSH Security Compliance Check - $(date)" > "$REPORT_FILE"
echo "==========================================" >> "$REPORT_FILE"
echo "" >> "$REPORT_FILE"
{
echo "Configuration Checks:"
check_config "permitrootlogin" "no" "no" "Root login disabled"
check_config "passwordauthentication" "no" "no" "Password authentication disabled"
check_config "pubkeyauthentication" "yes" "yes" "Public key authentication enabled"
check_config "protocol" "2" "2" "SSH Protocol 2 only"
check_config "x11forwarding" "no" "no" "X11 forwarding disabled"
check_config "maxauthtries" "3" "3" "Maximum auth tries limited"
check_config "clientaliveinterval" "300" "300" "Client alive interval set"
check_config "loglevel" "INFO" "INFO" "Appropriate log level"
echo ""
echo "Cryptography Checks:"
# Check ciphers
CIPHERS=$(sshd -T | grep "^ciphers" | cut -d' ' -f2-)
if echo "$CIPHERS" | grep -q "3des\|arcfour\|blowfish"; then
echo "$FAIL Weak ciphers detected"
else
echo "$PASS No weak ciphers found"
fi
# Check key exchange algorithms
KEX=$(sshd -T | grep "^kexalgorithms" | cut -d' ' -f2-)
if echo "$KEX" | grep -q "diffie-hellman-group1\|diffie-hellman-group14-sha1"; then
echo "$FAIL Weak key exchange algorithms detected"
else
echo "$PASS Strong key exchange algorithms only"
fi
echo ""
echo "File Permission Checks:"
# Check SSH config permissions
if [[ $(stat -c %a /etc/ssh/sshd_config) == "600" ]]; then
echo "$PASS /etc/ssh/sshd_config permissions correct (600)"
else
echo "$FAIL /etc/ssh/sshd_config permissions incorrect"
fi
# Check host key permissions
for KEY in /etc/ssh/ssh_host_*_key; do
if [[ -f "$KEY" ]]; then
PERM=$(stat -c %a "$KEY")
if [[ "$PERM" == "600" ]] || [[ "$PERM" == "640" ]]; then
echo "$PASS $KEY permissions correct ($PERM)"
else
echo "$FAIL $KEY permissions incorrect ($PERM)"
fi
fi
done
} | tee -a "$REPORT_FILE"
echo "" >> "$REPORT_FILE"
echo "Compliance check completed. Report saved to: $REPORT_FILE"
EOF
chmod +x /usr/local/bin/ssh-compliance-check.sh
# Schedule compliance checks
echo "0 2 * * 1 root /usr/local/bin/ssh-compliance-check.sh" >> /etc/crontab
Best Practices Summary
SSH Security Checklist
# Create SSH security checklist
cat > /root/ssh-security-checklist.md << 'EOF'
# SSH Security Checklist for AlmaLinux
## Initial Configuration
- [ ] Disable root login (PermitRootLogin no)
- [ ] Use SSH Protocol 2 only
- [ ] Change default SSH port (optional)
- [ ] Configure strong ciphers and MACs
- [ ] Set up SSH banner warning
## Authentication
- [ ] Enable public key authentication
- [ ] Disable password authentication (after keys setup)
- [ ] Implement two-factor authentication
- [ ] Configure SSH certificates (optional)
- [ ] Set up port knocking (optional)
## Access Control
- [ ] Use AllowUsers/AllowGroups directives
- [ ] Implement IP-based restrictions
- [ ] Configure MaxAuthTries and LoginGraceTime
- [ ] Set up connection rate limiting
- [ ] Use Match blocks for specific requirements
## Monitoring and Logging
- [ ] Enable comprehensive logging (LogLevel INFO)
- [ ] Implement fail2ban or similar IPS
- [ ] Set up real-time monitoring alerts
- [ ] Configure session recording (if required)
- [ ] Regular log analysis and reporting
## Key Management
- [ ] Use strong key algorithms (Ed25519 preferred)
- [ ] Implement key rotation policy
- [ ] Regular key audits
- [ ] Secure key storage
- [ ] Document key management procedures
## System Hardening
- [ ] Keep SSH and system packages updated
- [ ] Configure SELinux policies for SSH
- [ ] Set appropriate file permissions
- [ ] Disable unnecessary SSH features
- [ ] Regular security audits
## Operational Procedures
- [ ] Document SSH configuration
- [ ] Train administrators on SSH security
- [ ] Establish incident response procedures
- [ ] Regular compliance checks
- [ ] Backup SSH configurations
EOF
# Create quick reference card
cat > /root/ssh-quick-reference.txt << 'EOF'
SSH Security Quick Reference
===========================
Essential Commands:
-------------------
# Check SSH configuration
sshd -T
# Test configuration changes
sshd -t
# Generate SSH keys
ssh-keygen -t ed25519 -C "comment"
# Copy SSH key to server
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server
# Check fail2ban status
fail2ban-client status sshd
# View SSH logs
journalctl -u sshd -f
tail -f /var/log/secure
# Active SSH connections
ss -tnp | grep :22
who | grep pts
Security Settings:
------------------
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
ClientAliveInterval 300
Protocol 2
Troubleshooting:
----------------
# Debug SSH connection
ssh -vvv user@server
# Check SELinux denials
ausearch -m avc -ts recent | grep ssh
# Reset fail2ban bans
fail2ban-client unban <IP>
# Check SSH daemon status
systemctl status sshd
EOF
Conclusion
Properly securing SSH on AlmaLinux requires a multi-layered approach combining strong authentication, access controls, monitoring, and regular maintenance. By implementing the configurations and best practices outlined in this guide, you can significantly enhance your system’s security posture while maintaining operational efficiency.
Key takeaways:
- Always use key-based authentication
- Implement defense in depth with multiple security layers
- Monitor and log all SSH activity
- Regularly audit and update configurations
- Maintain comprehensive documentation
- Train users on SSH security best practices
Remember that security is an ongoing process. Regularly review and update your SSH configuration to address new threats and maintain compliance with security standards.