🚀 AlmaLinux DevOps CI/CD Pipelines Complete Guide

Ready to transform your software delivery from manual chaos to automated excellence? ⚡ This comprehensive guide will turn you into a CI/CD pipeline master, covering everything from basic automation concepts to enterprise-scale deployment strategies that deliver code from commit to production in minutes!

DevOps CI/CD isn’t just about automation – it’s about creating a culture of continuous improvement where every code change is automatically tested, validated, and deployed with confidence. Let’s build pipelines that never sleep and always deliver! 🌟

🤔 Why are CI/CD Pipelines Important?

Imagine deploying to production 100 times a day with zero fear – that’s the power of CI/CD! 🎯 Here’s why continuous integration and delivery are absolutely revolutionary:

• ⚡ Lightning-Fast Delivery: Deploy features in minutes instead of months
• 🛡️ Risk Reduction: Catch bugs early with automated testing at every stage
• 📈 Increased Productivity: Developers focus on coding, not manual deployments
• 🔄 Continuous Feedback: Instant feedback on code quality and functionality
• 💰 Cost Savings: Automated processes reduce manual effort and human errors
• 🎯 Better Quality: Consistent, repeatable processes ensure high-quality releases
• 🌍 Faster Time-to-Market: Beat competitors with rapid feature delivery
• 🤝 Team Collaboration: Break down silos between development and operations

🎯 What You Need

Before we embark on this DevOps journey, let’s make sure you have everything ready:

✅ AlmaLinux server(s) (your CI/CD command center!) ✅ Git repository (GitHub, GitLab, or Bitbucket for source control) ✅ Docker installed (for containerized applications) ✅ At least 4GB RAM (CI/CD tools need resources) ✅ 20GB+ disk space (for builds, artifacts, and containers) ✅ Basic command line knowledge (we’ll handle the rest!) ✅ A sample application (or we’ll create one together) ✅ Enthusiasm for automation (we’re eliminating manual work forever!)

📝 Step 1: Setting Up Jenkins CI/CD Server

Let’s start with Jenkins, the open-source automation powerhouse! 🏗️ Jenkins will be the heart of our CI/CD infrastructure.

# Update system packages
sudo dnf update -y
# Ensures you have the latest security patches

# Install Java (Jenkins requirement)
sudo dnf install -y java-11-openjdk java-11-openjdk-devel
# Jenkins runs on Java

# Add Jenkins repository
sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key

# Install Jenkins
sudo dnf install -y jenkins

# Start and enable Jenkins
sudo systemctl enable --now jenkins

# Check Jenkins status
sudo systemctl status jenkins
# Should show active (running)

Configure firewall and initial setup:

# Open Jenkins port in firewall
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --reload

# Get initial admin password
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
# Save this password - you'll need it for web setup!

echo "🌐 Access Jenkins at: http://$(hostname -I | awk '{print $1}'):8080"
echo "🔑 Use the admin password shown above"

Create Jenkins pipeline configuration:

# Create directory for Jenkins configurations
mkdir -p ~/jenkins-configs

# Create a sample Jenkinsfile for pipeline as code
cat << 'EOF' > ~/jenkins-configs/Jenkinsfile
pipeline {
    agent any

    environment {
        DOCKER_IMAGE = "myapp"
        DOCKER_TAG = "${BUILD_NUMBER}"
        REGISTRY = "localhost:5000"
    }

    stages {
        stage('Checkout') {
            steps {
                echo '📥 Checking out code from repository...'
                checkout scm
            }
        }

        stage('Build') {
            steps {
                echo '🔨 Building application...'
                sh '''
                    # Build your application here
                    echo "Building version ${BUILD_NUMBER}"
                    # Example for Node.js app
                    # npm install
                    # npm run build
                '''
            }
        }

        stage('Test') {
            parallel {
                stage('Unit Tests') {
                    steps {
                        echo '🧪 Running unit tests...'
                        sh '''
                            # Run unit tests
                            echo "Running unit tests"
                            # npm test
                        '''
                    }
                }

                stage('Integration Tests') {
                    steps {
                        echo '🔗 Running integration tests...'
                        sh '''
                            # Run integration tests
                            echo "Running integration tests"
                            # npm run test:integration
                        '''
                    }
                }

                stage('Security Scan') {
                    steps {
                        echo '🔒 Running security scan...'
                        sh '''
                            # Security scanning
                            echo "Scanning for vulnerabilities"
                            # Example: npm audit
                        '''
                    }
                }
            }
        }

        stage('Build Docker Image') {
            steps {
                echo '🐳 Building Docker image...'
                sh '''
                    docker build -t ${REGISTRY}/${DOCKER_IMAGE}:${DOCKER_TAG} .
                    docker tag ${REGISTRY}/${DOCKER_IMAGE}:${DOCKER_TAG} ${REGISTRY}/${DOCKER_IMAGE}:latest
                '''
            }
        }

        stage('Push to Registry') {
            steps {
                echo '📤 Pushing image to registry...'
                sh '''
                    docker push ${REGISTRY}/${DOCKER_IMAGE}:${DOCKER_TAG}
                    docker push ${REGISTRY}/${DOCKER_IMAGE}:latest
                '''
            }
        }

        stage('Deploy to Staging') {
            steps {
                echo '🚀 Deploying to staging environment...'
                sh '''
                    # Deploy to staging
                    echo "Deploying version ${BUILD_NUMBER} to staging"
                    # kubectl apply -f k8s/staging/
                    # docker-compose -f docker-compose.staging.yml up -d
                '''
            }
        }

        stage('Smoke Tests') {
            steps {
                echo '💨 Running smoke tests...'
                sh '''
                    # Verify deployment
                    echo "Running smoke tests on staging"
                    # curl http://staging.example.com/health
                '''
            }
        }

        stage('Deploy to Production') {
            when {
                branch 'main'
            }
            input {
                message "Deploy to production?"
                ok "Deploy"
            }
            steps {
                echo '🎯 Deploying to production...'
                sh '''
                    # Deploy to production
                    echo "Deploying version ${BUILD_NUMBER} to production"
                    # kubectl apply -f k8s/production/
                '''
            }
        }
    }

    post {
        success {
            echo '✅ Pipeline completed successfully!'
            // Send success notification
        }
        failure {
            echo '❌ Pipeline failed!'
            // Send failure notification
        }
        always {
            echo '🧹 Cleaning up...'
            sh 'docker system prune -f'
        }
    }
}
EOF

Create Jenkins automation script:

# Create Jenkins management script
cat << 'EOF' > ~/jenkins-manager.sh
#!/bin/bash
echo "🔧 JENKINS CI/CD MANAGER"
echo "========================"

install_plugins() {
    echo "📦 Installing essential Jenkins plugins..."

    JENKINS_URL="http://localhost:8080"
    JENKINS_CLI="/var/lib/jenkins/jenkins-cli.jar"

    # Download Jenkins CLI
    sudo wget -O $JENKINS_CLI "$JENKINS_URL/jnlpJars/jenkins-cli.jar"

    # List of essential plugins
    PLUGINS=(
        "git"
        "github"
        "docker-workflow"
        "kubernetes"
        "pipeline-stage-view"
        "blueocean"
        "credentials-binding"
        "email-ext"
        "slack"
        "prometheus"
    )

    for plugin in "${PLUGINS[@]}"; do
        echo "Installing plugin: $plugin"
        java -jar $JENKINS_CLI -s $JENKINS_URL install-plugin $plugin
    done

    echo "✅ Plugins installed. Restart Jenkins to activate."
}

create_pipeline_job() {
    local job_name="$1"
    local repo_url="$2"

    echo "🔨 Creating pipeline job: $job_name"

    cat > /tmp/job-config.xml << XML
<?xml version='1.1' encoding='UTF-8'?>
<flow-definition plugin="workflow-job">
    <description>CI/CD Pipeline for $job_name</description>
    <keepDependencies>false</keepDependencies>
    <properties>
        <org.jenkinsci.plugins.workflow.job.properties.PipelineTriggersJobProperty>
            <triggers>
                <hudson.triggers.SCMTrigger>
                    <spec>H/5 * * * *</spec>
                </hudson.triggers.SCMTrigger>
            </triggers>
        </org.jenkinsci.plugins.workflow.job.properties.PipelineTriggersJobProperty>
    </properties>
    <definition class="org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition">
        <scm class="hudson.plugins.git.GitSCM">
            <configVersion>2</configVersion>
            <userRemoteConfigs>
                <hudson.plugins.git.UserRemoteConfig>
                    <url>$repo_url</url>
                </hudson.plugins.git.UserRemoteConfig>
            </userRemoteConfigs>
            <branches>
                <hudson.plugins.git.BranchSpec>
                    <name>*/main</name>
                </hudson.plugins.git.BranchSpec>
            </branches>
        </scm>
        <scriptPath>Jenkinsfile</scriptPath>
    </definition>
    <triggers/>
</flow-definition>
XML

    # Create job via Jenkins API
    curl -X POST "http://localhost:8080/createItem?name=$job_name" \
        --data-binary @/tmp/job-config.xml \
        -H "Content-Type: application/xml"

    echo "✅ Pipeline job created: $job_name"
}

backup_jenkins() {
    echo "💾 Backing up Jenkins configuration..."

    BACKUP_DIR="/backup/jenkins-$(date +%Y%m%d_%H%M%S)"
    sudo mkdir -p $BACKUP_DIR

    # Backup Jenkins home directory
    sudo tar -czf $BACKUP_DIR/jenkins-home.tar.gz \
        --exclude='*/workspace/*' \
        --exclude='*/jobs/*/builds/*' \
        /var/lib/jenkins/

    echo "✅ Backup saved to: $BACKUP_DIR"
}

monitor_builds() {
    echo "📊 Jenkins Build Statistics:"
    echo "============================"

    # Get build statistics via API
    curl -s "http://localhost:8080/api/json?tree=jobs[name,lastBuild[number,result,timestamp,duration]]" | \
        python3 -m json.tool
}

case "$1" in
    plugins)
        install_plugins
        ;;
    create-job)
        create_pipeline_job "$2" "$3"
        ;;
    backup)
        backup_jenkins
        ;;
    monitor)
        monitor_builds
        ;;
    *)
        echo "Usage: $0 {plugins|create-job|backup|monitor}"
        echo "  plugins                    - Install essential plugins"
        echo "  create-job <name> <repo>   - Create pipeline job"
        echo "  backup                     - Backup Jenkins configuration"
        echo "  monitor                    - Show build statistics"
        ;;
esac
EOF

chmod +x ~/jenkins-manager.sh

🔧 Step 2: GitLab CI/CD Setup

Now let’s configure GitLab CI/CD for powerful, integrated pipelines! 🦊 GitLab provides everything you need in one platform.

# Install GitLab Runner
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh | sudo bash
sudo dnf install -y gitlab-runner

# Register runner with GitLab
sudo gitlab-runner register
# You'll be prompted for:
# - GitLab instance URL
# - Registration token (from GitLab project settings)
# - Description
# - Tags
# - Executor (choose docker or shell)

Create GitLab CI configuration:

# Create .gitlab-ci.yml for your project
cat << 'EOF' > ~/gitlab-ci-example.yml
# GitLab CI/CD Pipeline Configuration
stages:
  - build
  - test
  - security
  - package
  - deploy

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""
  IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  LATEST_TAG: $CI_REGISTRY_IMAGE:latest

# Cache dependencies
cache:
  paths:
    - node_modules/
    - .npm/
    - vendor/

# Build stage
build:app:
  stage: build
  image: node:16-alpine
  script:
    - echo "🔨 Building application..."
    - npm ci --cache .npm --prefer-offline
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour
  only:
    - branches
  tags:
    - docker

# Test stage - parallel execution
test:unit:
  stage: test
  image: node:16-alpine
  script:
    - echo "🧪 Running unit tests..."
    - npm ci --cache .npm --prefer-offline
    - npm run test:unit
  coverage: '/Coverage: \d+\.\d+%/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
      junit: junit.xml
  tags:
    - docker

test:integration:
  stage: test
  image: node:16-alpine
  services:
    - postgres:13
    - redis:6
  variables:
    POSTGRES_DB: test_db
    POSTGRES_USER: test_user
    POSTGRES_PASSWORD: test_pass
  script:
    - echo "🔗 Running integration tests..."
    - npm ci --cache .npm --prefer-offline
    - npm run test:integration
  tags:
    - docker

test:e2e:
  stage: test
  image: cypress/browsers:node16.14.2-slim-chrome100-ff99-edge
  script:
    - echo "🌐 Running end-to-end tests..."
    - npm ci --cache .npm --prefer-offline
    - npm run test:e2e:ci
  artifacts:
    when: always
    paths:
      - cypress/videos/
      - cypress/screenshots/
    expire_in: 1 week
  tags:
    - docker

# Security scanning
security:dependency-check:
  stage: security
  image: node:16-alpine
  script:
    - echo "🔒 Checking dependencies for vulnerabilities..."
    - npm audit --audit-level=moderate
  allow_failure: true
  tags:
    - docker

security:sast:
  stage: security
  image: returntocorp/semgrep:latest
  script:
    - echo "🔍 Running static analysis..."
    - semgrep --config=auto .
  allow_failure: true
  tags:
    - docker

security:container-scan:
  stage: security
  image: aquasec/trivy:latest
  script:
    - echo "🐳 Scanning container image..."
    - trivy image --no-progress --severity HIGH,CRITICAL $IMAGE_TAG
  only:
    - main
    - develop
  tags:
    - docker

# Package stage
package:docker:
  stage: package
  image: docker:20.10
  services:
    - docker:20.10-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - echo "🐳 Building Docker image..."
    - docker build -t $IMAGE_TAG -t $LATEST_TAG .
    - docker push $IMAGE_TAG
    - docker push $LATEST_TAG
  only:
    - main
    - develop
  tags:
    - docker

# Deploy stages
deploy:staging:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: staging
    url: https://staging.example.com
  before_script:
    - kubectl config set-cluster k8s --server="$KUBE_URL" --insecure-skip-tls-verify=true
    - kubectl config set-credentials admin --token="$KUBE_TOKEN"
    - kubectl config set-context default --cluster=k8s --user=admin
    - kubectl config use-context default
  script:
    - echo "🚀 Deploying to staging..."
    - kubectl set image deployment/app app=$IMAGE_TAG -n staging
    - kubectl rollout status deployment/app -n staging
  only:
    - develop
  tags:
    - docker

deploy:production:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: production
    url: https://example.com
  before_script:
    - kubectl config set-cluster k8s --server="$KUBE_URL" --insecure-skip-tls-verify=true
    - kubectl config set-credentials admin --token="$KUBE_TOKEN"
    - kubectl config set-context default --cluster=k8s --user=admin
    - kubectl config use-context default
  script:
    - echo "🎯 Deploying to production..."
    - kubectl set image deployment/app app=$IMAGE_TAG -n production
    - kubectl rollout status deployment/app -n production
  only:
    - main
  when: manual
  tags:
    - docker

# Rollback job
rollback:production:
  stage: deploy
  image: bitnami/kubectl:latest
  environment:
    name: production
    url: https://example.com
    action: rollback
  script:
    - echo "⏮️ Rolling back production deployment..."
    - kubectl rollout undo deployment/app -n production
    - kubectl rollout status deployment/app -n production
  when: manual
  only:
    - main
  tags:
    - docker
EOF

🌟 Step 3: GitHub Actions Configuration

Let’s set up GitHub Actions for seamless CI/CD! 🐙 GitHub Actions provides powerful, integrated workflows.

# Create GitHub Actions workflow directory structure
mkdir -p .github/workflows

# Create main CI/CD workflow
cat << 'EOF' > .github/workflows/cicd.yml
name: CI/CD Pipeline

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]
  workflow_dispatch:

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  # Build and test job
  build-and-test:
    name: Build and Test
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [14.x, 16.x, 18.x]

    steps:
    - name: 📥 Checkout code
      uses: actions/checkout@v3
      with:
        fetch-depth: 0

    - name: 🟢 Setup Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v3
      with:
        node-version: ${{ matrix.node-version }}
        cache: 'npm'

    - name: 📦 Install dependencies
      run: npm ci

    - name: 🔨 Build application
      run: npm run build

    - name: 🧪 Run tests
      run: |
        npm run test:unit
        npm run test:integration

    - name: 📊 Generate coverage report
      run: npm run coverage

    - name: 📤 Upload coverage to Codecov
      uses: codecov/codecov-action@v3
      with:
        file: ./coverage/coverage.xml
        flags: unittests
        name: codecov-umbrella

    - name: 📦 Upload artifacts
      uses: actions/upload-artifact@v3
      with:
        name: build-artifacts-${{ matrix.node-version }}
        path: dist/

  # Security scanning job
  security-scan:
    name: Security Scanning
    runs-on: ubuntu-latest
    needs: build-and-test

    steps:
    - name: 📥 Checkout code
      uses: actions/checkout@v3

    - name: 🔒 Run Trivy vulnerability scanner
      uses: aquasecurity/trivy-action@master
      with:
        scan-type: 'fs'
        scan-ref: '.'
        format: 'sarif'
        output: 'trivy-results.sarif'

    - name: 📤 Upload Trivy results to GitHub Security
      uses: github/codeql-action/upload-sarif@v2
      with:
        sarif_file: 'trivy-results.sarif'

    - name: 🔍 SonarQube scan
      uses: sonarsource/sonarqube-scan-action@master
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
      with:
        args: >
          -Dsonar.projectKey=my-project
          -Dsonar.sources=.

  # Docker build and push job
  docker:
    name: Build and Push Docker Image
    runs-on: ubuntu-latest
    needs: [build-and-test, security-scan]

    permissions:
      contents: read
      packages: write

    steps:
    - name: 📥 Checkout code
      uses: actions/checkout@v3

    - name: 🔧 Set up Docker Buildx
      uses: docker/setup-buildx-action@v2

    - name: 🔑 Login to GitHub Container Registry
      uses: docker/login-action@v2
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: 📝 Extract metadata
      id: meta
      uses: docker/metadata-action@v4
      with:
        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
        tags: |
          type=ref,event=branch
          type=ref,event=pr
          type=semver,pattern={{version}}
          type=semver,pattern={{major}}.{{minor}}
          type=sha

    - name: 🐳 Build and push Docker image
      uses: docker/build-push-action@v4
      with:
        context: .
        push: true
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        cache-from: type=gha
        cache-to: type=gha,mode=max

  # Deploy to staging
  deploy-staging:
    name: Deploy to Staging
    runs-on: ubuntu-latest
    needs: docker
    if: github.ref == 'refs/heads/develop'
    environment:
      name: staging
      url: https://staging.example.com

    steps:
    - name: 📥 Checkout code
      uses: actions/checkout@v3

    - name: 🔧 Setup kubectl
      uses: azure/setup-kubectl@v3

    - name: 🔑 Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v2
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1

    - name: 📦 Update kubeconfig
      run: aws eks update-kubeconfig --name staging-cluster

    - name: 🚀 Deploy to Kubernetes
      run: |
        kubectl set image deployment/app app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:sha-${GITHUB_SHA::7} -n staging
        kubectl rollout status deployment/app -n staging

  # Deploy to production
  deploy-production:
    name: Deploy to Production
    runs-on: ubuntu-latest
    needs: docker
    if: github.ref == 'refs/heads/main'
    environment:
      name: production
      url: https://example.com

    steps:
    - name: 📥 Checkout code
      uses: actions/checkout@v3

    - name: 🔧 Setup kubectl
      uses: azure/setup-kubectl@v3

    - name: 🔑 Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v2
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1

    - name: 📦 Update kubeconfig
      run: aws eks update-kubeconfig --name production-cluster

    - name: 🎯 Deploy to Kubernetes
      run: |
        kubectl set image deployment/app app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:sha-${GITHUB_SHA::7} -n production
        kubectl rollout status deployment/app -n production

    - name: 💨 Run smoke tests
      run: |
        ./scripts/smoke-tests.sh production

  # Notification job
  notify:
    name: Send Notifications
    runs-on: ubuntu-latest
    needs: [deploy-staging, deploy-production]
    if: always()

    steps:
    - name: 📨 Send Slack notification
      uses: 8398a7/action-slack@v3
      with:
        status: ${{ job.status }}
        text: |
          Pipeline Status: ${{ job.status }}
          Commit: ${{ github.sha }}
          Author: ${{ github.actor }}
          Branch: ${{ github.ref }}
        webhook_url: ${{ secrets.SLACK_WEBHOOK }}
      if: always()
EOF

✅ Step 4: Docker Registry and Artifact Management

Let’s set up a private Docker registry for storing container images! 🐳

# Install Docker (if not already installed)
sudo dnf install -y docker-ce docker-ce-cli containerd.io
sudo systemctl enable --now docker

# Add current user to docker group
sudo usermod -aG docker $USER

# Create Docker registry with authentication
mkdir -p ~/docker-registry/{data,auth,certs}

# Generate self-signed certificates for HTTPS
openssl req -newkey rsa:4096 -nodes -sha256 -x509 -days 365 \
  -keyout ~/docker-registry/certs/domain.key \
  -out ~/docker-registry/certs/domain.crt \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=registry.local"

# Create htpasswd file for authentication
sudo dnf install -y httpd-tools
htpasswd -Bc ~/docker-registry/auth/htpasswd admin
# Enter password when prompted

# Run Docker registry container
docker run -d \
  --restart=always \
  --name registry \
  -v ~/docker-registry/data:/var/lib/registry \
  -v ~/docker-registry/auth:/auth \
  -v ~/docker-registry/certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  -e REGISTRY_AUTH=htpasswd \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
  -p 5000:5000 \
  registry:2

# Test registry
docker login localhost:5000
# Enter username: admin and the password you created

Create artifact management script:

# Create artifact manager script
cat << 'EOF' > ~/artifact-manager.sh
#!/bin/bash
echo "📦 ARTIFACT MANAGER"
echo "==================="

setup_nexus() {
    echo "🔧 Setting up Nexus Repository Manager..."

    # Run Nexus container
    docker run -d \
        --name nexus \
        --restart always \
        -p 8081:8081 \
        -v nexus-data:/nexus-data \
        sonatype/nexus3

    echo "⏳ Waiting for Nexus to start..."
    sleep 30

    # Get initial admin password
    NEXUS_PASSWORD=$(docker exec nexus cat /nexus-data/admin.password)

    echo "✅ Nexus is running at http://localhost:8081"
    echo "👤 Username: admin"
    echo "🔑 Password: $NEXUS_PASSWORD"
}

cleanup_artifacts() {
    echo "🧹 Cleaning up old artifacts..."

    # Clean Docker images older than 30 days
    docker image prune -a --filter "until=720h" -f

    # Clean Docker volumes
    docker volume prune -f

    # Clean build caches
    docker builder prune -a -f

    echo "✅ Cleanup complete"
}

list_images() {
    echo "🐳 Docker Registry Images:"
    echo "=========================="

    # List images in local registry
    curl -X GET https://localhost:5000/v2/_catalog \
        -u admin:password \
        --insecure 2>/dev/null | python3 -m json.tool

    echo ""
    echo "📊 Image sizes:"
    docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"
}

backup_artifacts() {
    echo "💾 Backing up artifacts..."

    BACKUP_DIR="/backup/artifacts-$(date +%Y%m%d_%H%M%S)"
    mkdir -p $BACKUP_DIR

    # Backup Docker registry data
    tar -czf $BACKUP_DIR/registry-data.tar.gz ~/docker-registry/data/

    # Export important Docker images
    docker save $(docker images -q) -o $BACKUP_DIR/docker-images.tar

    echo "✅ Backup saved to: $BACKUP_DIR"
}

promote_image() {
    local image="$1"
    local from_tag="$2"
    local to_tag="$3"

    echo "📤 Promoting image from $from_tag to $to_tag..."

    docker pull localhost:5000/$image:$from_tag
    docker tag localhost:5000/$image:$from_tag localhost:5000/$image:$to_tag
    docker push localhost:5000/$image:$to_tag

    echo "✅ Image promoted successfully"
}

case "$1" in
    setup-nexus)
        setup_nexus
        ;;
    cleanup)
        cleanup_artifacts
        ;;
    list)
        list_images
        ;;
    backup)
        backup_artifacts
        ;;
    promote)
        promote_image "$2" "$3" "$4"
        ;;
    *)
        echo "Usage: $0 {setup-nexus|cleanup|list|backup|promote}"
        echo "  setup-nexus               - Set up Nexus repository"
        echo "  cleanup                   - Clean old artifacts"
        echo "  list                      - List registry images"
        echo "  backup                    - Backup artifacts"
        echo "  promote <img> <from> <to> - Promote image tag"
        ;;
esac
EOF

chmod +x ~/artifact-manager.sh

🌟 Step 5: Automated Testing and Quality Gates

Let’s implement comprehensive testing and quality gates! 🧪 Quality is non-negotiable in CI/CD.

Create automated testing framework:

# Create testing automation script
cat << 'EOF' > ~/test-automation.sh
#!/bin/bash
echo "🧪 AUTOMATED TESTING FRAMEWORK"
echo "=============================="

run_unit_tests() {
    echo "🔬 Running Unit Tests..."

    # Example for different languages
    case "$1" in
        node)
            npm test
            ;;
        python)
            pytest tests/unit --cov=app --cov-report=xml
            ;;
        java)
            mvn test
            ;;
        go)
            go test ./... -v -cover
            ;;
        *)
            echo "❌ Unknown language: $1"
            return 1
            ;;
    esac

    echo "✅ Unit tests completed"
}

run_integration_tests() {
    echo "🔗 Running Integration Tests..."

    # Start test containers
    docker-compose -f docker-compose.test.yml up -d

    # Wait for services
    sleep 10

    # Run tests
    npm run test:integration
    TEST_RESULT=$?

    # Cleanup
    docker-compose -f docker-compose.test.yml down

    return $TEST_RESULT
}

run_performance_tests() {
    echo "⚡ Running Performance Tests..."

    # Install k6 if not present
    if ! command -v k6 &> /dev/null; then
        sudo dnf install -y https://dl.k6.io/rpm/repo.rpm
        sudo dnf install -y k6
    fi

    # Run k6 performance test
    cat > performance-test.js << 'JS'
import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
    stages: [
        { duration: '2m', target: 100 },
        { duration: '5m', target: 100 },
        { duration: '2m', target: 200 },
        { duration: '5m', target: 200 },
        { duration: '2m', target: 0 },
    ],
    thresholds: {
        'http_req_duration': ['p(95)<500'],
        'http_req_failed': ['rate<0.1'],
    },
};

export default function() {
    let response = http.get('http://localhost:8080');
    check(response, {
        'status is 200': (r) => r.status === 200,
        'response time < 500ms': (r) => r.timings.duration < 500,
    });
    sleep(1);
}
JS

    k6 run performance-test.js

    echo "✅ Performance tests completed"
}

run_security_tests() {
    echo "🔒 Running Security Tests..."

    # OWASP ZAP security scanning
    docker run --rm \
        -v $(pwd):/zap/wrk/:rw \
        -t owasp/zap2docker-stable zap-baseline.py \
        -t http://localhost:8080 \
        -r security-report.html

    # Dependency vulnerability check
    npm audit --audit-level=moderate

    # Container security scan
    trivy image myapp:latest

    echo "✅ Security tests completed"
}

setup_sonarqube() {
    echo "📊 Setting up SonarQube..."

    # Run SonarQube container
    docker run -d \
        --name sonarqube \
        --restart always \
        -p 9000:9000 \
        -v sonarqube_data:/opt/sonarqube/data \
        -v sonarqube_extensions:/opt/sonarqube/extensions \
        -v sonarqube_logs:/opt/sonarqube/logs \
        sonarqube:lts

    echo "✅ SonarQube is running at http://localhost:9000"
    echo "Default credentials: admin/admin"
}

run_quality_gates() {
    echo "🎯 Checking Quality Gates..."

    # Run SonarQube analysis
    sonar-scanner \
        -Dsonar.projectKey=myproject \
        -Dsonar.sources=. \
        -Dsonar.host.url=http://localhost:9000 \
        -Dsonar.login=$SONAR_TOKEN

    # Check quality gate status
    QUALITY_GATE=$(curl -s -u $SONAR_TOKEN: \
        "http://localhost:9000/api/qualitygates/project_status?projectKey=myproject" | \
        python3 -c "import sys, json; print(json.load(sys.stdin)['projectStatus']['status'])")

    if [ "$QUALITY_GATE" != "OK" ]; then
        echo "❌ Quality gate failed: $QUALITY_GATE"
        return 1
    fi

    echo "✅ Quality gates passed"
}

generate_test_report() {
    echo "📈 Generating Test Report..."

    cat > test-report.html << 'HTML'
<!DOCTYPE html>
<html>
<head>
    <title>Test Report</title>
    <style>
        body { font-family: Arial; margin: 40px; }
        .header { background: #f0f0f0; padding: 20px; border-radius: 8px; }
        .success { color: #28a745; }
        .failure { color: #dc3545; }
        .metric { margin: 10px 0; padding: 10px; background: #f8f9fa; }
        table { width: 100%; border-collapse: collapse; }
        th, td { padding: 10px; text-align: left; border: 1px solid #ddd; }
    </style>
</head>
<body>
    <div class="header">
        <h1>🧪 Test Report</h1>
        <p>Generated: <script>document.write(new Date().toLocaleString());</script></p>
    </div>

    <div class="metric">
        <h2>Test Results Summary</h2>
        <p class="success">✅ Unit Tests: 95/100 passed</p>
        <p class="success">✅ Integration Tests: 48/50 passed</p>
        <p class="success">✅ Performance Tests: All thresholds met</p>
        <p class="failure">❌ Security Tests: 2 vulnerabilities found</p>
    </div>

    <div class="metric">
        <h2>Code Coverage</h2>
        <table>
            <tr><th>Module</th><th>Coverage</th></tr>
            <tr><td>Core</td><td>92%</td></tr>
            <tr><td>API</td><td>88%</td></tr>
            <tr><td>Utils</td><td>95%</td></tr>
            <tr><td>Overall</td><td><strong>91%</strong></td></tr>
        </table>
    </div>
</body>
</html>
HTML

    echo "✅ Test report generated: test-report.html"
}

case "$1" in
    unit)
        run_unit_tests "$2"
        ;;
    integration)
        run_integration_tests
        ;;
    performance)
        run_performance_tests
        ;;
    security)
        run_security_tests
        ;;
    sonarqube)
        setup_sonarqube
        ;;
    quality-gates)
        run_quality_gates
        ;;
    report)
        generate_test_report
        ;;
    all)
        run_unit_tests
        run_integration_tests
        run_performance_tests
        run_security_tests
        run_quality_gates
        generate_test_report
        ;;
    *)
        echo "Usage: $0 {unit|integration|performance|security|sonarqube|quality-gates|report|all}"
        ;;
esac
EOF

chmod +x ~/test-automation.sh

🎮 Quick Examples

Let’s see your CI/CD pipelines in action with real-world scenarios! 🎯

Example 1: Multi-Stage Pipeline with Rollback

# Create advanced pipeline with rollback capability
cat << 'EOF' > ~/advanced-pipeline.sh
#!/bin/bash

DEPLOYMENT_VERSION=$(git rev-parse --short HEAD)
PREVIOUS_VERSION=""

deploy_with_canary() {
    echo "🐤 Starting canary deployment..."

    # Deploy canary version (10% traffic)
    kubectl set image deployment/app-canary app=myapp:$DEPLOYMENT_VERSION -n production

    # Wait and monitor metrics
    sleep 60

    # Check canary health
    CANARY_ERRORS=$(kubectl logs -l version=canary -n production --since=1m | grep ERROR | wc -l)

    if [ $CANARY_ERRORS -gt 10 ]; then
        echo "❌ Canary failed with $CANARY_ERRORS errors"
        rollback_deployment
        return 1
    fi

    echo "✅ Canary healthy, proceeding with full deployment"

    # Full deployment
    kubectl set image deployment/app app=myapp:$DEPLOYMENT_VERSION -n production
    kubectl rollout status deployment/app -n production
}

rollback_deployment() {
    echo "⏮️ Rolling back to previous version..."
    kubectl rollout undo deployment/app -n production
    kubectl rollout status deployment/app -n production
    echo "✅ Rollback complete"
}

# Blue-green deployment
blue_green_deployment() {
    echo "🔵🟢 Starting blue-green deployment..."

    # Deploy to green environment
    kubectl apply -f k8s/green-deployment.yml
    kubectl set image deployment/app-green app=myapp:$DEPLOYMENT_VERSION -n production
    kubectl rollout status deployment/app-green -n production

    # Run smoke tests on green
    ./smoke-tests.sh green

    if [ $? -eq 0 ]; then
        echo "✅ Green deployment healthy, switching traffic..."
        # Switch service to green
        kubectl patch service app -n production -p '{"spec":{"selector":{"version":"green"}}}'

        # Keep blue for quick rollback
        sleep 300  # Monitor for 5 minutes

        # If stable, remove blue
        kubectl delete deployment app-blue -n production
    else
        echo "❌ Green deployment failed tests"
        kubectl delete deployment app-green -n production
        return 1
    fi
}

# Execute deployment strategy
case "$1" in
    canary)
        deploy_with_canary
        ;;
    blue-green)
        blue_green_deployment
        ;;
    rollback)
        rollback_deployment
        ;;
    *)
        echo "Usage: $0 {canary|blue-green|rollback}"
        ;;
esac
EOF

chmod +x ~/advanced-pipeline.sh

Example 2: GitOps with ArgoCD

# Install ArgoCD for GitOps
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Create ArgoCD application
cat << 'EOF' | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/youruser/yourapp
    targetRevision: HEAD
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
EOF

# Access ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443
echo "🌐 ArgoCD UI: https://localhost:8080"

Example 3: Pipeline Monitoring Dashboard

# Create monitoring dashboard
cat << 'EOF' > ~/pipeline-monitor.sh
#!/bin/bash

while true; do
    clear
    echo "📊 CI/CD PIPELINE DASHBOARD"
    echo "==========================="
    echo "🕒 $(date)"
    echo ""

    echo "🔧 Jenkins Status:"
    curl -s http://localhost:8080/api/json?tree=jobs[name,lastBuild[result]] | \
        python3 -c "import sys, json; data=json.load(sys.stdin); [print(f\"  {j['name']}: {j['lastBuild']['result'] if j['lastBuild'] else 'Never run'}\") for j in data.get('jobs', [])]" 2>/dev/null || echo "  Jenkins not accessible"

    echo ""
    echo "🦊 GitLab Pipelines:"
    curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
        "https://gitlab.com/api/v4/projects/$PROJECT_ID/pipelines?per_page=5" | \
        python3 -c "import sys, json; data=json.load(sys.stdin); [print(f\"  #{p['id']}: {p['status']} ({p['ref']})\") for p in data]" 2>/dev/null || echo "  GitLab not accessible"

    echo ""
    echo "🐳 Docker Registry:"
    echo "  Images: $(docker images | wc -l)"
    echo "  Size: $(docker system df | grep Images | awk '{print $4}')"

    echo ""
    echo "☸️ Kubernetes Deployments:"
    kubectl get deployments --all-namespaces --no-headers | \
        awk '{print "  " $2 " (" $1 "): " $3 "/" $4 " replicas"}' | head -5

    echo ""
    echo "📈 Build Statistics (Last 24h):"
    echo "  Successful: 42"
    echo "  Failed: 3"
    echo "  In Progress: 1"
    echo "  Average Duration: 8m 32s"

    echo ""
    echo "Press Ctrl+C to exit"
    sleep 10
done
EOF

chmod +x ~/pipeline-monitor.sh

🚨 Fix Common Problems

Don’t worry when CI/CD issues arise – here are solutions to common pipeline problems! 🛠️

Problem 1: Pipeline Fails with Permission Errors

Symptoms: “Permission denied”, “Cannot access repository” errors

# Fix Jenkins permissions
sudo usermod -aG docker jenkins
sudo systemctl restart jenkins

# Fix GitLab Runner permissions
sudo usermod -aG docker gitlab-runner
sudo systemctl restart gitlab-runner

# Fix GitHub Actions permissions
# In repository settings, check:
# Settings > Actions > General > Workflow permissions
# Select "Read and write permissions"

Problem 2: Docker Build Failures

Symptoms: “Cannot connect to Docker daemon”, build context issues

# Check Docker service
sudo systemctl status docker

# Fix Docker socket permissions
sudo chmod 666 /var/run/docker.sock

# Clean Docker cache
docker system prune -a -f

# Increase Docker storage
# Edit /etc/docker/daemon.json
{
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ],
  "data-root": "/var/lib/docker"
}

sudo systemctl restart docker

Problem 3: Slow Pipeline Performance

Symptoms: Builds taking too long, timeouts

# Enable Docker layer caching
# In Jenkinsfile or CI config:
docker build --cache-from myapp:latest -t myapp:latest .

# Parallelize tests
# Split tests across multiple jobs
parallel {
    stage('Unit Tests') { ... }
    stage('Integration Tests') { ... }
    stage('Security Scan') { ... }
}

# Use pipeline caching
# Cache dependencies between builds
cache:
  paths:
    - node_modules/
    - .m2/repository/
    - vendor/

Problem 4: Deployment Failures

Symptoms: “ImagePullBackOff”, deployment not updating

# Check image availability
docker pull myregistry.com/myapp:latest

# Fix registry authentication
kubectl create secret docker-registry regcred \
    --docker-server=myregistry.com \
    --docker-username=admin \
    --docker-password=password \
    [email protected]

# Use the secret in deployment
kubectl patch serviceaccount default \
    -p '{"imagePullSecrets": [{"name": "regcred"}]}'

# Force deployment update
kubectl rollout restart deployment/myapp

📋 Simple Commands Summary

Here’s your CI/CD quick reference guide! 📚

💡 Tips for Success

Follow these expert strategies to master CI/CD pipelines! 🌟

🎯 Pipeline Design Best Practices

• Keep pipelines fast – Aim for under 10 minutes total execution time
• Fail fast – Run quick checks first, expensive operations later
• Parallelize everything – Run independent tasks simultaneously
• Cache aggressively – Cache dependencies, Docker layers, and build artifacts

🔧 Testing Strategy

• Test pyramid – Many unit tests, fewer integration tests, minimal E2E tests
• Shift left – Run security and quality checks early in the pipeline
• Automate everything – If you do it twice, automate it
• Monitor test trends – Track test execution time and failure rates

🛡️ Security and Compliance

• Scan early and often – Security scanning in every pipeline stage
• Secret management – Never commit secrets, use vault or secret managers
• Audit trails – Log all deployments and configuration changes
• Compliance gates – Enforce security and quality standards

🚀 Advanced Techniques

• GitOps – Use Git as single source of truth for deployments
• Progressive delivery – Canary, blue-green, and feature flag deployments
• Self-healing – Automatic rollback on failure detection
• Multi-cloud deployments – Deploy across AWS, Azure, and GCP

🏆 What You Learned

Congratulations! You’ve mastered DevOps CI/CD pipelines on AlmaLinux! 🎉 Here’s your impressive achievement:

✅ Built complete CI/CD infrastructure with Jenkins, GitLab CI, and GitHub Actions ✅ Configured automated testing frameworks with unit, integration, and E2E tests ✅ Set up Docker registry and artifact management systems ✅ Implemented quality gates with SonarQube and security scanning ✅ Created deployment strategies including canary and blue-green deployments ✅ Built monitoring dashboards for pipeline visibility and metrics ✅ Configured GitOps workflows for declarative deployments ✅ Implemented rollback mechanisms for quick recovery ✅ Mastered pipeline optimization for fast, reliable builds ✅ Created automation scripts for pipeline management and monitoring

🎯 Why This Matters

DevOps CI/CD expertise is the backbone of modern software delivery! 💎

Every successful tech company relies on robust CI/CD pipelines to deliver features quickly and reliably. From startups shipping daily to enterprises deploying thousands of times per day, CI/CD enables the pace of innovation that drives business success.

These skills open doors to high-paying DevOps Engineer, Site Reliability Engineer, and Platform Engineer roles. Companies desperately need pipeline experts who can design, implement, and maintain CI/CD systems that scale with their growth while maintaining quality and security.

Remember, you haven’t just learned tools – you’ve mastered the art of continuous delivery. Your ability to automate the entire software lifecycle from code to production makes you invaluable in today’s fast-paced tech world.

Keep automating, keep delivering, and keep pushing the boundaries of what’s possible with CI/CD! Your expertise enables teams to focus on innovation while you ensure their code reaches users seamlessly! 🚀⚡🙌