💾 Longhorn Distributed Storage on AlmaLinux 9: Complete Guide

Ready to revolutionize your Kubernetes storage? 🚀 Today we’ll set up Longhorn on AlmaLinux 9, creating bulletproof distributed storage that never loses data! Let’s build something amazing! ✨💪

🤔 Why is Longhorn Important?

Imagine storage that heals itself and never fails! 🎯 That’s Longhorn’s superpower! Here’s why it’s game-changing:

• 🔄 Self-Healing Storage - Automatically recovers from node failures!
• 📦 Distributed Replicas - Your data lives on multiple nodes safely
• 🛡️ Automatic Backups - Schedule backups to S3 or NFS automatically
• 📸 Instant Snapshots - Create point-in-time copies instantly
• 🎨 Beautiful UI - Manage everything through an intuitive dashboard
• 🚀 Zero Downtime Upgrades - Upgrade storage without stopping apps
• 🌍 Disaster Recovery - Restore data across clusters easily
• 💡 Simple Management - No PhD required to manage storage!

🎯 What You Need

Before we start this storage adventure, gather these:

• ✅ AlmaLinux 9 server (8GB RAM minimum, 16GB recommended)
• ✅ Kubernetes cluster running (K3s or K8s)
• ✅ At least 3 nodes for production (1 for testing)
• ✅ 50GB+ free disk space per node
• ✅ kubectl configured and working
• ✅ iSCSI client installed
• ✅ Open ports 9500-9504
• ✅ Excitement for bulletproof storage! 🎉

📝 Step 1: Prepare AlmaLinux for Longhorn

Let’s prepare your system for distributed storage! 🛠️

Install Required Packages

# Update system packages
sudo dnf update -y  # Get latest updates

# Install iSCSI initiator (required for Longhorn)
sudo dnf install -y iscsi-initiator-utils

# Install NFS utilities (for backup support)
sudo dnf install -y nfs-utils

# Install other dependencies
sudo dnf install -y util-linux e2fsprogs xfsprogs

# Enable and start iSCSI service
sudo systemctl enable --now iscsid

# Verify iSCSI is running
sudo systemctl status iscsid  # Should show active

Configure Kernel Modules

# Load required kernel modules
sudo modprobe iscsi_tcp
sudo modprobe dm_snapshot
sudo modprobe dm_thin_pool

# Make modules persistent
cat <<EOF | sudo tee /etc/modules-load.d/longhorn.conf
iscsi_tcp
dm_snapshot
dm_thin_pool
EOF

# Verify modules are loaded
lsmod | grep iscsi  # Should show iscsi_tcp

Create Longhorn Data Directory

# Create dedicated partition mount point
sudo mkdir -p /var/lib/longhorn

# If you have a dedicated disk (recommended)
# Replace /dev/sdb with your disk
sudo mkfs.ext4 /dev/sdb  # Format disk
sudo mount /dev/sdb /var/lib/longhorn  # Mount it

# Add to /etc/fstab for persistent mount
echo "/dev/sdb /var/lib/longhorn ext4 defaults 0 0" | sudo tee -a /etc/fstab

# Set permissions
sudo chmod 755 /var/lib/longhorn

# Verify mount
df -h /var/lib/longhorn  # Should show your disk

🔧 Step 2: Check Environment Compatibility

Let’s verify everything is ready! 🔍

Run Longhorn Environment Check

# Download and run environment check script
curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.6.0/scripts/environment_check.sh | bash

# You should see all checks passing!
# If any fail, fix them before continuing

Check Kubernetes Cluster

# Verify nodes are ready
kubectl get nodes  # All should be Ready

# Check available storage on nodes
kubectl get nodes -o json | jq '.items[].status.allocatable.storage'

# Verify default storage class
kubectl get storageclass  # Note current default

🌟 Step 3: Install Longhorn

Time to install the magic! 🎊 Choose your favorite method!

Method 1: Install with kubectl (Simple)

# Install Longhorn with single command!
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.0/deploy/longhorn.yaml

# Watch the installation progress
kubectl get pods -n longhorn-system -w  # Ctrl+C when all Running

# This creates namespace and all resources
kubectl get all -n longhorn-system  # See everything!

Method 2: Install with Helm (Flexible)

# Add Longhorn Helm repository
helm repo add longhorn https://charts.longhorn.io
helm repo update

# Install Longhorn with custom values
cat <<EOF > longhorn-values.yaml
persistence:
  defaultClass: true
  defaultClassReplicaCount: 3
  reclaimPolicy: Retain
defaultSettings:
  defaultDataPath: /var/lib/longhorn
  replicaSoftAntiAffinity: true
  storageMinimalAvailablePercentage: 10
  upgradeChecker: true
  defaultReplicaCount: 3
  backupTarget: s3://your-bucket@us-east-1/longhorn-backups
  backupTargetCredentialSecret: backup-secret
ingress:
  enabled: true
  host: longhorn.yourdomain.com
  tls: true
EOF

# Install with Helm
helm install longhorn longhorn/longhorn \
  --namespace longhorn-system \
  --create-namespace \
  --values longhorn-values.yaml

# Check installation
helm list -n longhorn-system

Verify Installation

# Check all pods are running
kubectl get pods -n longhorn-system

# Check Longhorn components
kubectl get daemonset -n longhorn-system  # Should show nodes
kubectl get deployment -n longhorn-system  # Should be ready
kubectl get service -n longhorn-system  # Should show services

✅ Step 4: Access Longhorn UI

Let’s explore the beautiful dashboard! 🎨

Method 1: Port Forwarding

# Forward Longhorn UI port
kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80 &

# Access UI at http://localhost:8080
echo "🎉 Longhorn UI is ready at http://localhost:8080"

Method 2: NodePort Service

# Create NodePort for external access
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: longhorn-frontend-nodeport
  namespace: longhorn-system
spec:
  type: NodePort
  selector:
    app: longhorn-ui
  ports:
  - port: 80
    targetPort: 8000
    nodePort: 30080
EOF

# Access at http://YOUR_NODE_IP:30080

Method 3: Ingress (Production)

# Create Ingress for Longhorn UI
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: longhorn-ingress
  namespace: longhorn-system
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - host: longhorn.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: longhorn-frontend
            port:
              number: 80
EOF

# Access at http://longhorn.yourdomain.com

🎮 Quick Examples

Let’s create some persistent storage! 🚀

Example 1: Create Your First Volume

# Create a PersistentVolumeClaim
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-first-longhorn-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
EOF

# Check PVC status
kubectl get pvc my-first-longhorn-pvc  # Should be Bound

# See the volume in Longhorn UI!

Example 2: Deploy Application with Persistent Storage

# Deploy PostgreSQL with Longhorn storage
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:15
        env:
        - name: POSTGRES_PASSWORD
          value: "mysecretpassword"
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: postgres-storage
        persistentVolumeClaim:
          claimName: postgres-pvc
EOF

# Check deployment
kubectl get pods | grep postgres  # Should be Running
kubectl exec -it deploy/postgres -- df -h  # See mounted volume!

Example 3: Create Snapshot and Backup

# Create a snapshot using kubectl
cat <<EOF | kubectl apply -f -
apiVersion: longhorn.io/v1beta2
kind: Snapshot
metadata:
  name: postgres-snapshot-1
  namespace: longhorn-system
spec:
  volume: pvc-xxxxx  # Replace with your volume name
EOF

# Configure S3 backup target
kubectl create secret generic backup-secret \
  -n longhorn-system \
  --from-literal=AWS_ACCESS_KEY_ID=your-key \
  --from-literal=AWS_SECRET_ACCESS_KEY=your-secret

# Create backup from UI or CLI
# The backup will be stored in S3!

🚨 Fix Common Problems

Don’t panic! Here are solutions! 💪

Problem 1: Pods Stuck in Pending

# Check PVC status
kubectl describe pvc <pvc-name>

# Check Longhorn nodes
kubectl get nodes.longhorn.io -n longhorn-system

# Check available disk space
kubectl get nodes.longhorn.io -n longhorn-system -o yaml | grep -A5 disk

# Fix: Add more disk space or adjust settings
kubectl edit settings.longhorn.io -n longhorn-system
# Change storageMinimalAvailablePercentage to 5

Problem 2: Volume Attachment Failed

# Check volume status
kubectl get volumes.longhorn.io -n longhorn-system

# Check engine and replica
kubectl get engines.longhorn.io -n longhorn-system
kubectl get replicas.longhorn.io -n longhorn-system

# Force detach and reattach
kubectl delete volumeattachment <attachment-name>

# Restart Longhorn manager
kubectl rollout restart deployment/longhorn-manager -n longhorn-system

Problem 3: Backup Not Working

# Check backup target settings
kubectl get settings.longhorn.io backup-target -n longhorn-system -o yaml

# Verify credentials
kubectl get secret backup-secret -n longhorn-system -o yaml

# Test S3 connection manually
aws s3 ls s3://your-bucket/longhorn-backups/ --endpoint-url=https://s3.amazonaws.com

# Check backup jobs
kubectl get backups.longhorn.io -n longhorn-system

📋 Simple Commands Summary

Your Longhorn command toolkit! 📚

💡 Tips for Success

Become a storage expert with these tips! 🏆

Storage Best Practices

• 🔢 Use 3 replicas for production data
• 💾 Keep 20% disk space free for operations
• 📸 Schedule daily snapshots for important data
• 🌍 Configure S3 backups for disaster recovery
• 🔄 Test restore procedures regularly

Performance Optimization

• ⚡ Use SSDs for better performance
• 🎯 Set node affinity for critical workloads
• 📊 Monitor disk I/O with Longhorn UI
• 🚀 Adjust replica counts based on importance
• 💡 Use separate disks for OS and Longhorn

Monitoring Tips

• 📈 Watch volume health in UI dashboard
• 🔍 Set up alerts for low disk space
• 📊 Monitor replica rebuild progress
• ⚠️ Check for degraded volumes daily
• 📝 Keep backup logs for audit

🏆 What You Learned

Amazing job! You’re now a storage wizard! 🎉 You can:

• ✅ Install Longhorn on AlmaLinux 9
• ✅ Create persistent volumes for applications
• ✅ Access and navigate Longhorn UI
• ✅ Deploy applications with distributed storage
• ✅ Create snapshots and backups
• ✅ Troubleshoot common storage issues
• ✅ Configure disaster recovery
• ✅ Optimize storage performance

🎯 Why This Matters

You’ve just built enterprise-grade storage! 🚀 With Longhorn:

• No More Data Loss - Multiple replicas protect your data
• Automatic Recovery - Self-healing from node failures
• Easy Backups - Automated backups to cloud storage
• Simple Management - Beautiful UI for all operations
• Cost Effective - Use commodity hardware for enterprise storage
• Cloud Native - Perfect for Kubernetes workloads
• Peace of Mind - Your data is always safe and available

Your Kubernetes cluster now has bulletproof storage that rivals expensive SAN solutions! Data persistence is no longer a worry - Longhorn handles everything automatically.

Keep exploring advanced features like recurring backups, disaster recovery volumes, and storage network isolation. Your storage game is now top-tier! 🌟

Remember: Your data is your treasure - Longhorn is your vault! Happy storing! 🎊💾

P.S. - Explore Longhorn’s advanced features like encryption, RAID configurations, and multi-cluster replication. Join the community and share your storage success stories! ⭐🙌