Setting Up SSH Server in Alpine Linux: Remote Access Done Right

I’ll walk you through setting up SSH on Alpine Linux. Trust me, after dealing with broken SSH configs and locked-out servers, I’ve learned the hard way how to do this properly. Let’s get you connected securely.

Introduction

So here’s the thing - SSH is probably the most important service you’ll run on any Linux server. It’s your lifeline for remote access. Alpine Linux doesn’t come with SSH server running by default, which is actually smart from a security standpoint.

I remember the first time I tried to SSH into a fresh Alpine install and got “connection refused.” That’s when I realized Alpine keeps things minimal - you gotta enable what you need.

Why You Need This

• Connect to your Alpine server from anywhere
• Manage servers without physical access
• Transfer files securely over the network
• Run commands remotely with confidence

Prerequisites

You’ll need these things first:

• Alpine Linux system with root access
• Network connectivity
• Basic understanding of terminal commands
• Another computer to test SSH connection from

What we’re doing: Let’s check what’s currently running and update our system.

# Check if SSH is already running (probably not)
ps aux | grep ssh

# Update package database
sudo apk update

# Check current network status
ip addr show

Code explanation:

• ps aux | grep ssh: Shows if any SSH processes are running
• sudo apk update: Gets latest package information
• ip addr show: Displays network interfaces and IP addresses

Installing OpenSSH Server

Basic Installation

What we’re doing: Installing the SSH server package and getting it ready to run.

# Install OpenSSH server
sudo apk add openssh

# Check what we got
which sshd
sshd -V

Code explanation:

• sudo apk add openssh: Installs the OpenSSH server package
• which sshd: Shows where the SSH daemon is installed
• sshd -V: Displays SSH server version information

Starting SSH Service

What we’re doing: Starting the SSH service and making it start automatically on boot.

# Start SSH service now
sudo service sshd start

# Make SSH start automatically on boot
sudo rc-update add sshd

# Check if it's running
sudo service sshd status
ps aux | grep sshd

Code explanation:

• sudo service sshd start: Starts the SSH daemon immediately
• sudo rc-update add sshd: Adds SSH to startup services
• sudo service sshd status: Shows current SSH service status
• ps aux | grep sshd: Confirms SSH processes are running

Checking SSH is Working

What we’re doing: Making sure SSH is listening for connections.

# Check what ports SSH is listening on
sudo netstat -tlnp | grep :22

# Alternative way to check
sudo ss -tlnp | grep :22

# Test local connection
ssh localhost

Code explanation:

• netstat -tlnp | grep :22: Shows if SSH is listening on port 22
• ss -tlnp | grep :22: Modern alternative to netstat
• ssh localhost: Tests SSH connection to the same machine

Basic SSH Configuration

Understanding SSH Config File

What we’re doing: Looking at the main SSH configuration file.

# View SSH config file
sudo cat /etc/ssh/sshd_config

# Make a backup before changing anything
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup

# Edit SSH config
sudo nano /etc/ssh/sshd_config

Code explanation:

• /etc/ssh/sshd_config: Main SSH server configuration file
• Making backup prevents losing working config
• nano: Simple text editor for config changes

Essential Configuration Changes

What we’re doing: Making basic security improvements to SSH config.

# Edit the SSH config file
sudo nano /etc/ssh/sshd_config

Key settings to change:

# Change default SSH port for security (optional)
Port 22

# Only allow SSH protocol version 2
Protocol 2

# Don't allow root login (create regular user first)
PermitRootLogin no

# Use key-based authentication (more secure)
PubkeyAuthentication yes

# Disable password authentication (after setting up keys)
PasswordAuthentication yes

# Don't allow empty passwords
PermitEmptyPasswords no

# Limit login attempts
MaxAuthTries 3

# Set login timeout
LoginGraceTime 60

Configuration explanation:

• Port 22: Default SSH port (change to random port for security)
• PermitRootLogin no: Prevents direct root login attempts
• PubkeyAuthentication yes: Allows SSH key authentication
• PasswordAuthentication yes: Keep enabled until keys are set up
• MaxAuthTries 3: Limits failed login attempts

Applying Configuration Changes

What we’re doing: Restarting SSH to apply our configuration changes.

# Test config file syntax first
sudo sshd -t

# If test passes, restart SSH service
sudo service sshd restart

# Check service status
sudo service sshd status

Code explanation:

• sshd -t: Tests configuration file for syntax errors
• Always test config before restarting to avoid lockouts
• Restart applies the new configuration

Setting Up User Access

Creating SSH User

What we’re doing: Creating a regular user account for SSH access.

# Create new user for SSH access
sudo adduser sshuser

# Add user to wheel group for sudo access
sudo adduser sshuser wheel

# Set a strong password
sudo passwd sshuser

Code explanation:

• adduser sshuser: Creates new user account
• wheel group: Allows sudo access on Alpine Linux
• Strong passwords are important until key auth is set up

Testing User Login

What we’re doing: Making sure our new user can log in via SSH.

# Test SSH login with new user (from another machine or terminal)
ssh sshuser@your_server_ip

# Or test locally first
ssh sshuser@localhost

Code explanation:

• Replace your_server_ip with actual server IP address
• Testing locally first helps debug config issues
• Should prompt for password we just set

SSH Key Authentication Setup

Generating SSH Keys

What we’re doing: Creating SSH key pair for secure authentication.

# Generate SSH key pair (run on client machine)
ssh-keygen -t rsa -b 4096 -C "[email protected]"

# Or generate ed25519 key (more modern)
ssh-keygen -t ed25519 -C "[email protected]"

# Keys are saved in ~/.ssh/
ls -la ~/.ssh/

Code explanation:

• ssh-keygen: Creates public/private key pair
• -t rsa -b 4096: RSA key with 4096 bits (very secure)
• -t ed25519: Modern key type (smaller, faster)
• Private key stays on client, public key goes to server

Copying Public Key to Server

What we’re doing: Installing the public key on the server for authentication.

# Easy way using ssh-copy-id
ssh-copy-id sshuser@your_server_ip

# Manual way if ssh-copy-id isn't available
cat ~/.ssh/id_rsa.pub | ssh sshuser@your_server_ip "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

# Set proper permissions on server
ssh sshuser@your_server_ip "chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys"

Code explanation:

• ssh-copy-id: Automatically copies public key to server
• ~/.ssh/authorized_keys: File where server stores allowed public keys
• Proper permissions are critical for SSH security

Testing Key Authentication

What we’re doing: Making sure SSH key authentication works properly.

# Test SSH with key authentication
ssh sshuser@your_server_ip

# Should log in without password prompt
# If prompted for password, key auth isn't working

Code explanation:

• Successful key auth logs in without password
• If still prompting for password, check permissions and config

Disabling Password Authentication

What we’re doing: Turning off password authentication for better security.

# Edit SSH config to disable passwords
sudo nano /etc/ssh/sshd_config

Change this setting:

# Disable password authentication (only after key auth works!)
PasswordAuthentication no

What we’re doing: Testing the change and applying it.

# Test SSH config
sudo sshd -t

# Restart SSH service
sudo service sshd restart

# Test SSH still works with keys
ssh sshuser@your_server_ip

Code explanation:

• Only disable passwords after confirming key auth works
• This prevents password-based attacks
• Always test before fully applying changes

Security Hardening

Changing Default Port

What we’re doing: Moving SSH off the default port to reduce automated attacks.

# Edit SSH config
sudo nano /etc/ssh/sshd_config

Change port setting:

# Change from default port 22 to something else
Port 2222

What we’re doing: Opening the new port in firewall and restarting SSH.

# If using iptables/firewall, allow new port
sudo iptables -A INPUT -p tcp --dport 2222 -j ACCEPT

# Test config and restart
sudo sshd -t
sudo service sshd restart

# Test connection on new port
ssh -p 2222 sshuser@your_server_ip

Code explanation:

• Non-standard ports reduce automated attack attempts
• Must update firewall rules for new port
• -p 2222: Specifies custom SSH port

Setting Up Fail2Ban

What we’re doing: Installing fail2ban to block repeated failed login attempts.

# Install fail2ban
sudo apk add fail2ban

# Start and enable fail2ban
sudo service fail2ban start
sudo rc-update add fail2ban

# Check if it's working
sudo service fail2ban status

Code explanation:

• fail2ban: Automatically blocks IPs with failed login attempts
• Essential protection against brute force attacks
• Works by monitoring SSH logs

Configuring SSH Client Restrictions

What we’re doing: Adding more restrictions to SSH connections.

# Edit SSH config for additional security
sudo nano /etc/ssh/sshd_config

Add these security settings:

# Limit which users can SSH
AllowUsers sshuser

# Limit connection attempts
MaxStartups 3

# Disconnect idle sessions
ClientAliveInterval 300
ClientAliveCountMax 2

# Disable X11 forwarding if not needed
X11Forwarding no

# Disable TCP forwarding if not needed
AllowTcpForwarding no

Configuration explanation:

• AllowUsers: Only listed users can SSH
• MaxStartups: Limits concurrent connection attempts
• ClientAliveInterval: Disconnects idle sessions
• Disabling forwarding reduces attack surface

Troubleshooting Common Issues

SSH Connection Refused

What’s wrong: Can’t connect to SSH server at all.

What we’re doing: Checking if SSH service is actually running.

# Check if SSH is running
sudo service sshd status
ps aux | grep sshd

# Check if SSH is listening
sudo netstat -tlnp | grep :22

# Check firewall settings
sudo iptables -L

# Restart SSH if needed
sudo service sshd restart

Code explanation:

• Service might not be running
• Firewall might be blocking connections
• Wrong port configuration

Permission Denied Errors

What’s wrong: SSH connects but won’t let you log in.

What we’re doing: Checking permissions and authentication settings.

# Check SSH key permissions
ls -la ~/.ssh/
ls -la ~/.ssh/authorized_keys

# Fix permissions if wrong
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

# Check SSH config allows key auth
grep -i pubkey /etc/ssh/sshd_config
grep -i passwordauth /etc/ssh/sshd_config

Code explanation:

• Wrong permissions break SSH key auth
• Config might disable the auth method you’re using
• SSH is very picky about file permissions

Can’t Login After Config Changes

What’s wrong: Locked out after changing SSH config.

What we’re doing: Getting back in and fixing the config.

# If you have physical/console access
# Log in directly and fix SSH config
sudo nano /etc/ssh/sshd_config

# Restore from backup
sudo cp /etc/ssh/sshd_config.backup /etc/ssh/sshd_config

# Test and restart
sudo sshd -t
sudo service sshd restart

Code explanation:

• Always keep backup of working config
• Physical access bypasses SSH issues
• Test config before restarting in production

Best Practices

SSH Key Management

What we’re doing: Organizing SSH keys properly for multiple servers.

# Create specific keys for different servers
ssh-keygen -t ed25519 -f ~/.ssh/server1_key -C "server1-access"
ssh-keygen -t ed25519 -f ~/.ssh/server2_key -C "server2-access"

# Use SSH config file to manage connections
nano ~/.ssh/config

SSH client config example:

# Server 1 configuration
Host server1
    HostName 192.168.1.100
    User sshuser
    Port 2222
    IdentityFile ~/.ssh/server1_key

# Server 2 configuration  
Host server2
    HostName 192.168.1.101
    User admin
    Port 22
    IdentityFile ~/.ssh/server2_key

Configuration explanation:

• Separate keys for different servers improves security
• SSH config simplifies connection commands
• Can use easy names instead of IPs

Regular Security Maintenance

What we’re doing: Keeping SSH secure with regular maintenance.

# Check SSH logs for suspicious activity
sudo tail -f /var/log/auth.log

# Update SSH regularly
sudo apk update
sudo apk upgrade openssh

# Review SSH config periodically
sudo cat /etc/ssh/sshd_config | grep -v '^#' | grep -v '^$'

# Check fail2ban status
sudo fail2ban-client status sshd

Code explanation:

• Monitor logs for attack attempts
• Keep SSH updated for security patches
• Review config to remove unused features
• Check that fail2ban is blocking attackers

Multiple User Setup

What we’re doing: Setting up SSH access for multiple users safely.

# Create additional SSH users
sudo adduser developer
sudo adduser admin

# Set up SSH access for each user
# (copy public keys to each user's ~/.ssh/authorized_keys)

# Configure different access levels
sudo nano /etc/ssh/sshd_config

Multi-user SSH config:

# Different settings for different users
Match User developer
    AllowTcpForwarding no
    X11Forwarding no
    PermitTunnel no

Match User admin
    AllowTcpForwarding yes
    X11Forwarding yes

Configuration explanation:

• Different users can have different permissions
• Developers might not need forwarding capabilities
• Admins might need full access

Advanced SSH Features

SSH Tunneling

What we’re doing: Setting up secure tunnels through SSH.

# Local port forwarding (access remote service locally)
ssh -L 8080:localhost:80 sshuser@your_server_ip

# Remote port forwarding (expose local service remotely)  
ssh -R 9090:localhost:3000 sshuser@your_server_ip

# SOCKS proxy tunnel
ssh -D 1080 sshuser@your_server_ip

Code explanation:

• -L: Forward local port to remote service
• -R: Forward remote port to local service
• -D: Create SOCKS proxy for web browsing

SSH Agent Setup

What we’re doing: Using SSH agent to manage keys automatically.

# Start SSH agent
eval $(ssh-agent)

# Add keys to agent
ssh-add ~/.ssh/id_rsa
ssh-add ~/.ssh/server1_key

# List loaded keys
ssh-add -l

# Remove all keys from agent
ssh-add -D

Code explanation:

• SSH agent remembers passphrases
• No need to type passphrase for each connection
• More secure than unprotected keys

Wrapping Up

You just learned how to:

• Install and configure SSH server on Alpine Linux
• Set up secure key-based authentication
• Harden SSH security with best practices
• Troubleshoot common SSH problems

That’s it! You now have a secure SSH server running on Alpine Linux. I use this exact setup on all my Alpine servers - it’s rock solid and secure. The key is taking it step by step and testing everything before locking down security.

Remember to always keep a backup of your SSH config and never lock yourself out. Physical access is your safety net when SSH goes wrong.

These SSH settings will keep the bad guys out while letting you manage your servers easily. Most attacks are automated scripts looking for default configs - once you change the basics, you’re way ahead of the game.