🛟 Velero Backup & Disaster Recovery on AlmaLinux 9: Complete Guide

Ready to make your Kubernetes bulletproof? 🚀 Today we’ll deploy Velero on AlmaLinux 9, creating powerful backup and disaster recovery that saves the day! Let’s protect everything! ✨🛡️

🤔 Why is Velero Important?

Imagine restoring your entire cluster in minutes! 💫 That’s Velero’s magic! Here’s why it’s essential:

• 🔄 Complete Backups - Resources and persistent volumes together!
• 📦 Disaster Recovery - Restore entire clusters from scratch
• 🚀 Migration Tool - Move workloads between clusters
• 🎯 Scheduled Backups - Automatic protection on schedule
• 📸 Point-in-Time Recovery - Go back to any snapshot
• 🛡️ Application Consistency - Pre/post backup hooks
• 🌍 Multi-Cloud - Works with S3, Azure, GCP, MinIO
• 💡 Selective Restore - Restore only what you need

🎯 What You Need

Before we backup everything, gather these:

• ✅ AlmaLinux 9 server (4GB RAM minimum)
• ✅ Kubernetes cluster 1.16+ running
• ✅ kubectl configured and working
• ✅ Object storage (MinIO, S3, Azure, or GCP)
• ✅ Storage credentials ready
• ✅ 10GB+ storage space for backups
• ✅ Basic Kubernetes knowledge
• ✅ Ready for disaster recovery! 🎉

📝 Step 1: Prepare AlmaLinux Environment

Let’s prepare your system for Velero! 🛠️

Install Prerequisites

# Update system
sudo dnf update -y  # Keep everything current

# Install required tools
sudo dnf install -y wget curl git

# Verify Kubernetes cluster
kubectl get nodes  # All should be Ready
kubectl version --short  # Check version

# Create velero namespace
kubectl create namespace velero

# Check cluster resources
kubectl top nodes  # Ensure sufficient resources

Install Velero CLI

# Download latest Velero CLI
VELERO_VERSION="v1.13.0"  # Check latest at github.com/vmware-tanzu/velero
wget https://github.com/vmware-tanzu/velero/releases/download/${VELERO_VERSION}/velero-${VELERO_VERSION}-linux-amd64.tar.gz

# Extract and install
tar -xzf velero-${VELERO_VERSION}-linux-amd64.tar.gz
sudo mv velero-${VELERO_VERSION}-linux-amd64/velero /usr/local/bin/
sudo chmod +x /usr/local/bin/velero

# Verify installation
velero version --client-only  # Shows CLI version

# Enable bash completion
velero completion bash | sudo tee /etc/bash_completion.d/velero > /dev/null
source /etc/bash_completion.d/velero

🔧 Step 2: Set Up Object Storage

Let’s configure storage for backups! 🎊

Option 1: MinIO Setup (Self-Hosted)

# Deploy MinIO in Kubernetes
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: minio
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
  namespace: minio
spec:
  replicas: 1
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
      - name: minio
        image: minio/minio:latest
        args:
        - server
        - /storage
        - --console-address
        - ":9001"
        env:
        - name: MINIO_ROOT_USER
          value: "minioadmin"
        - name: MINIO_ROOT_PASSWORD
          value: "minioadmin123"  # Change this!
        ports:
        - containerPort: 9000
          name: api
        - containerPort: 9001
          name: console
        volumeMounts:
        - name: storage
          mountPath: /storage
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: minio-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-pvc
  namespace: minio
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
---
apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: minio
spec:
  type: NodePort
  ports:
  - port: 9000
    targetPort: 9000
    nodePort: 30900
    name: api
  - port: 9001
    targetPort: 9001
    nodePort: 30901
    name: console
  selector:
    app: minio
EOF

# Wait for MinIO to be ready
kubectl wait --for=condition=ready pod -l app=minio -n minio --timeout=120s

# Create bucket for Velero
kubectl exec -n minio deploy/minio -- mc alias set local http://localhost:9000 minioadmin minioadmin123
kubectl exec -n minio deploy/minio -- mc mb local/velero-backups

Option 2: AWS S3 Setup

# Create AWS credentials file
cat > credentials-velero <<EOF
[default]
aws_access_key_id=YOUR_AWS_ACCESS_KEY
aws_secret_access_key=YOUR_AWS_SECRET_KEY
EOF

# Create S3 bucket (using AWS CLI)
aws s3api create-bucket \
  --bucket velero-backups-$(date +%s) \
  --region us-west-2 \
  --create-bucket-configuration LocationConstraint=us-west-2

🌟 Step 3: Install Velero

Time to deploy Velero with all features! 🚀

Install with MinIO

# Create credentials for MinIO
cat > credentials-minio <<EOF
[default]
aws_access_key_id=minioadmin
aws_secret_access_key=minioadmin123
EOF

# Get MinIO endpoint
MINIO_ENDPOINT=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[0].address}'):30900

# Install Velero with MinIO
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.9.0 \
  --bucket velero-backups \
  --secret-file ./credentials-minio \
  --use-node-agent \
  --backup-location-config \
    region=minio,s3ForcePathStyle="true",s3Url=http://${MINIO_ENDPOINT} \
  --snapshot-location-config \
    region=minio \
  --namespace velero

# Clean up credentials
rm credentials-minio

Install with AWS S3

# Install Velero with S3
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.9.0 \
  --bucket velero-backups-YOUR_BUCKET \
  --secret-file ./credentials-velero \
  --backup-location-config region=us-west-2 \
  --snapshot-location-config region=us-west-2 \
  --use-node-agent \
  --namespace velero

# Verify installation
kubectl get pods -n velero  # All should be Running
velero backup-location get  # Should show Available

✅ Step 4: Configure Backup Strategies

Let’s create comprehensive backup plans! 📋

Create Backup Schedule

# Daily backup of all namespaces
velero schedule create daily-backup \
  --schedule="0 2 * * *" \
  --ttl 720h0m0s \
  --include-namespaces "*" \
  --exclude-namespaces kube-system,kube-public,kube-node-lease

# Hourly backup of critical namespace
velero schedule create critical-hourly \
  --schedule="0 * * * *" \
  --ttl 168h0m0s \
  --include-namespaces production \
  --default-volumes-to-fs-backup

# Weekly full cluster backup
velero schedule create weekly-full \
  --schedule="0 0 * * 0" \
  --ttl 720h0m0s

# List schedules
velero schedule get

Configure Backup Hooks

# Create backup with pre/post hooks
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: database-pod
  namespace: production
  annotations:
    backup.velero.io/backup-volumes: data-volume
    pre.hook.backup.velero.io/container: database
    pre.hook.backup.velero.io/command: '["sh", "-c", "pg_dump -U postgres mydb > /backup/dump.sql"]'
    post.hook.backup.velero.io/container: database
    post.hook.backup.velero.io/command: '["sh", "-c", "echo Backup completed"]'
spec:
  containers:
  - name: database
    image: postgres:15
    volumeMounts:
    - name: data-volume
      mountPath: /var/lib/postgresql/data
  volumes:
  - name: data-volume
    persistentVolumeClaim:
      claimName: database-pvc
EOF

🎮 Quick Examples

Let’s master backup and restore! 🎬

Example 1: On-Demand Backup

# Backup entire cluster
velero backup create full-backup-$(date +%Y%m%d)

# Backup specific namespace
velero backup create app-backup \
  --include-namespaces production \
  --default-volumes-to-fs-backup

# Backup with label selector
velero backup create critical-backup \
  --selector app=critical \
  --default-volumes-to-fs-backup

# Monitor backup progress
velero backup describe full-backup-20240101
velero backup logs full-backup-20240101

Example 2: Disaster Recovery

# Simulate disaster - delete namespace
kubectl delete namespace production

# List available backups
velero backup get

# Restore from backup
velero restore create --from-backup app-backup

# Monitor restore
velero restore describe app-backup-20240101
velero restore logs app-backup-20240101

# Verify restoration
kubectl get all -n production

Example 3: Cluster Migration

# On source cluster - create backup
velero backup create migration-backup \
  --default-volumes-to-fs-backup \
  --ttl 720h0m0s

# Wait for completion
velero backup describe migration-backup --details

# On destination cluster - install Velero with same storage
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.9.0 \
  --bucket velero-backups \
  --secret-file ./credentials-minio \
  --backup-location-config \
    region=minio,s3ForcePathStyle="true",s3Url=http://minio.example.com:9000 \
  --use-node-agent

# Restore on destination
velero restore create --from-backup migration-backup

# Verify migration
kubectl get all --all-namespaces

🚨 Fix Common Problems

Don’t panic! We’ve got solutions! 💪

Problem 1: Backup Fails

# Check backup status
velero backup describe failed-backup --details

# Common fixes:
# Check node-agent/restic pods
kubectl get pods -n velero
kubectl logs -n velero -l name=node-agent

# Verify storage access
velero backup-location get

# Check credentials
kubectl get secret -n velero cloud-credentials -o yaml

# Force new backup after fix
velero backup delete failed-backup
velero backup create new-backup

Problem 2: Restore Issues

# Check restore status
velero restore describe failed-restore --details

# Partial restore
velero restore create partial-restore \
  --from-backup my-backup \
  --include-namespaces app-namespace \
  --include-resources deployments,services

# Skip problem resources
velero restore create skip-restore \
  --from-backup my-backup \
  --exclude-resources persistentvolumes,persistentvolumeclaims

Problem 3: Volume Backup Not Working

# Verify node-agent is running
kubectl get daemonset -n velero node-agent

# Add annotation to pod
kubectl annotate pod my-pod backup.velero.io/backup-volumes=my-volume

# Or set default for all volumes
velero backup create volume-backup \
  --default-volumes-to-fs-backup

# Check node-agent logs
kubectl logs -n velero -l name=node-agent --tail=100

📋 Simple Commands Summary

Your Velero command toolkit! 📚

💡 Tips for Success

Master disaster recovery with these tips! 🏆

Backup Strategy

• 📅 Schedule regular backups (daily minimum)
• 🎯 Use namespaces to organize workloads
• 📝 Label resources for selective backup
• 🔄 Test restore procedures regularly
• 💾 Keep multiple backup versions

Performance Optimization

• ⚡ Use parallel backup/restore
• 🎯 Exclude unnecessary resources
• 📊 Monitor backup sizes
• 🚀 Optimize storage backend
• 💡 Use incremental backups when possible

Best Practices

• 🛡️ Encrypt backups at rest
• 📋 Document restore procedures
• 🔍 Monitor backup success/failure
• ⚠️ Set up alerting for failures
• 🌍 Store backups in multiple locations
• 🔐 Rotate credentials regularly
• 📈 Track backup metrics

🏆 What You Learned

Incredible work! You’re now a disaster recovery expert! 🎉 You can:

• ✅ Install Velero on AlmaLinux 9
• ✅ Configure multiple storage backends
• ✅ Create comprehensive backup strategies
• ✅ Schedule automatic backups
• ✅ Perform disaster recovery
• ✅ Migrate clusters
• ✅ Use backup hooks
• ✅ Troubleshoot common issues

🎯 Why This Matters

You’ve achieved true resilience! 🚀 With Velero:

• Zero Data Loss - Everything is backed up
• Fast Recovery - Restore in minutes, not hours
• Easy Migration - Move workloads between clusters
• Compliance Ready - Meet backup requirements
• Peace of Mind - Sleep better knowing you’re protected
• Cost Savings - Avoid downtime costs
• Cloud Agnostic - Works everywhere

Your Kubernetes clusters are now disaster-proof! No more fear of data loss, no more lengthy recovery procedures. Everything is automated and tested.

Keep exploring features like cross-region replication, backup encryption, and custom resource definitions. You’re running enterprise-grade disaster recovery! 🌟

Remember: Hope is not a strategy - Velero is your insurance! Stay protected! 🎊🛟

P.S. - Join the Velero community, share your backup strategies, and help others achieve resilience! Together we’re making Kubernetes bulletproof! ⭐🙌