Implementing GitOps Workflow with ArgoCD on Rocky Linux

GitOps has revolutionized how we deploy and manage applications in Kubernetes environments. This comprehensive guide will walk you through implementing a complete GitOps workflow using ArgoCD on Rocky Linux, transforming your deployment process into a declarative, version-controlled, and automated system.

Understanding GitOps Principles

GitOps extends DevOps practices by using Git as the single source of truth for declarative infrastructure and applications. Key principles include:

• Declarative Configuration: Everything is defined as code
• Version Control: All changes are tracked in Git
• Automated Synchronization: Desired state is automatically applied
• Continuous Reconciliation: Drift detection and correction
• Pull-based Deployments: Enhanced security and auditability

Prerequisites

Before implementing GitOps with ArgoCD, ensure you have:

• Rocky Linux 9 server (minimum 4 CPU, 8GB RAM)
• Kubernetes cluster (v1.24 or later)
• kubectl configured for cluster access
• Git repository (GitHub, GitLab, or Bitbucket)
• Basic understanding of Kubernetes concepts
• SSL certificates (for production)

Setting Up the Foundation

Installing Required Tools on Rocky Linux

Start by preparing your Rocky Linux system:

# Update system packages
sudo dnf update -y

# Install essential tools
sudo dnf install -y git curl wget vim jq

# Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Install helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Verify installations
kubectl version --client
helm version

Setting Up a Kubernetes Cluster

For this guide, we’ll use K3s for simplicity:

# Install K3s on Rocky Linux
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

# Wait for K3s to be ready
sudo systemctl status k3s

# Configure kubectl
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config

# Verify cluster is running
kubectl get nodes
kubectl get pods -A

Installing ArgoCD

Deploy ArgoCD to Kubernetes

Create a dedicated namespace and install ArgoCD:

# Create ArgoCD namespace
kubectl create namespace argocd

# Install ArgoCD
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for all pods to be ready
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s

# Check ArgoCD components
kubectl get all -n argocd

Configuring ArgoCD for Production

Create a production-ready configuration:

# argocd-server-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-server-config
  namespace: argocd
data:
  url: "https://argocd.example.com"
  application.instanceLabelKey: argocd.argoproj.io/instance
  server.insecure: "false"
  admin.enabled: "true"
  dex.config: |
    connectors:
    - type: github
      id: github
      name: GitHub
      config:
        clientID: $dex.github.clientID
        clientSecret: $dex.github.clientSecret
        orgs:
        - name: your-org
  resource.customizations: |
    admissionregistration.k8s.io/MutatingWebhookConfiguration:
      ignoreDifferences: |
        jsonPointers:
        - /webhooks/0/clientConfig/caBundle
  resource.compareoptions: |
    ignoreAggregatedRoles: true
---
apiVersion: v1
kind: Service
metadata:
  name: argocd-server-nodeport
  namespace: argocd
spec:
  type: NodePort
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8080
    nodePort: 30443
  selector:
    app.kubernetes.io/name: argocd-server

Apply the configuration:

kubectl apply -f argocd-server-config.yaml

Setting Up Ingress for ArgoCD

Configure NGINX ingress for ArgoCD:

# argocd-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - argocd.example.com
    secretName: argocd-server-tls
  rules:
  - host: argocd.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              number: 443

Configuring ArgoCD CLI

Installing ArgoCD CLI on Rocky Linux

# Download ArgoCD CLI
curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64

# Make it executable
sudo chmod +x /usr/local/bin/argocd

# Verify installation
argocd version

Initial Login and Configuration

# Get initial admin password
ARGOCD_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
echo "Initial password: $ARGOCD_PASSWORD"

# Login to ArgoCD
argocd login argocd.example.com --username admin --password $ARGOCD_PASSWORD

# Change admin password
argocd account update-password \
  --current-password $ARGOCD_PASSWORD \
  --new-password YourNewSecurePassword

# Delete initial admin secret
kubectl -n argocd delete secret argocd-initial-admin-secret

Creating Your First GitOps Repository

Repository Structure

Create a well-organized Git repository structure:

# Create repository structure
mkdir -p my-gitops-repo/{applications,environments,clusters}
cd my-gitops-repo

# Create base structure
cat > README.md << 'EOF'
# GitOps Repository

This repository contains all Kubernetes manifests for our applications.

## Structure

- `/applications` - Application definitions
- `/environments` - Environment-specific configurations
- `/clusters` - Cluster-specific configurations

## Usage

ArgoCD monitors this repository and automatically syncs changes to the cluster.
EOF

# Initialize git repository
git init
git add .
git commit -m "Initial commit"

Application Manifests

Create a sample application structure:

# applications/sample-app/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml

commonLabels:
  app: sample-app
  version: v1.0.0
---
# applications/sample-app/base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: sample-app
        image: nginx:1.21
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
        env:
        - name: ENVIRONMENT
          valueFrom:
            configMapKeyRef:
              name: sample-app-config
              key: environment
---
# applications/sample-app/base/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: sample-app
spec:
  selector:
    app: sample-app
  ports:
  - port: 80
    targetPort: 80
  type: ClusterIP
---
# applications/sample-app/base/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-app-config
data:
  environment: "base"
  log_level: "info"

Environment Overlays

Create environment-specific configurations:

# environments/development/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: development

resources:
  - namespace.yaml
  - ../../applications/sample-app/base

patchesStrategicMerge:
  - deployment-patch.yaml
  - configmap-patch.yaml

images:
  - name: nginx
    newTag: 1.21-alpine
---
# environments/development/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: development
---
# environments/development/deployment-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: sample-app
        resources:
          requests:
            memory: "32Mi"
            cpu: "100m"
          limits:
            memory: "64Mi"
            cpu: "200m"
---
# environments/development/configmap-patch.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-app-config
data:
  environment: "development"
  log_level: "debug"

Configuring ArgoCD Applications

Creating an App-of-Apps Pattern

Implement the app-of-apps pattern for better organization:

# clusters/production/apps/root-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: clusters/production/apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
    - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
---
# clusters/production/apps/sample-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sample-app-production
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: environments/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
  revisionHistoryLimit: 3

Registering Your Git Repository

# Add repository to ArgoCD
argocd repo add https://github.com/your-org/gitops-repo \
  --username your-username \
  --password your-token

# List registered repositories
argocd repo list

# Create the root application
kubectl apply -f clusters/production/apps/root-app.yaml

# Check application status
argocd app list
argocd app get root-app

Advanced ArgoCD Configuration

Multi-Cluster Management

Configure ArgoCD for multi-cluster deployments:

# Add external cluster
argocd cluster add production-cluster \
  --name production \
  --kubeconfig /path/to/kubeconfig

# List clusters
argocd cluster list

# Create cluster-specific application
cat > clusters/production/cluster-config.yaml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-cluster-config
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: clusters/production/config
  destination:
    name: production
    namespace: kube-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

RBAC Configuration

Implement fine-grained access control:

# argocd-rbac.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    p, role:admin, applications, *, */*, allow
    p, role:admin, clusters, *, *, allow
    p, role:admin, repositories, *, *, allow
    p, role:developer, applications, get, */*, allow
    p, role:developer, applications, sync, */*, allow
    p, role:developer, applications, action/*, */*, allow
    p, role:developer, logs, get, */*, allow
    p, role:developer, exec, create, */*, allow
    g, your-org:developers, role:developer
    g, your-org:admins, role:admin
  scopes: '[groups]'

Secrets Management

Implement secure secrets management with Sealed Secrets:

# Install Sealed Secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml

# Install kubeseal CLI
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/kubeseal-0.18.0-linux-amd64.tar.gz
tar -xvf kubeseal-0.18.0-linux-amd64.tar.gz
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

# Create a secret
echo -n mypassword | kubectl create secret generic mysecret \
  --dry-run=client \
  --from-file=password=/dev/stdin \
  -o yaml | kubeseal -o yaml > mysealedsecret.yaml

# Commit sealed secret to Git
git add mysealedsecret.yaml
git commit -m "Add sealed secret"
git push

Implementing CI/CD Integration

GitHub Actions Integration

Create a GitHub Actions workflow for automated deployments:

# .github/workflows/deploy.yaml
name: Deploy to ArgoCD

on:
  push:
    branches:
      - main
    paths:
      - 'applications/**'
      - 'environments/**'

jobs:
  update-manifests:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v3

    - name: Setup Kustomize
      uses: imranismail/setup-kustomize@v1

    - name: Update image tag
      run: |
        cd environments/production
        kustomize edit set image myapp=myapp:${{ github.sha }}

    - name: Commit and push changes
      run: |
        git config --local user.email "[email protected]"
        git config --local user.name "GitHub Action"
        git add .
        git commit -m "Update image tag to ${{ github.sha }}"
        git push

  sync-argocd:
    needs: update-manifests
    runs-on: ubuntu-latest
    steps:
    - name: Sync ArgoCD Application
      uses: omegion/[email protected]
      with:
        address: ${{ secrets.ARGOCD_SERVER }}
        token: ${{ secrets.ARGOCD_TOKEN }}
        appName: sample-app-production

GitLab CI Integration

Configure GitLab CI for GitOps:

# .gitlab-ci.yml
stages:
  - build
  - deploy

variables:
  DOCKER_REGISTRY: registry.gitlab.com
  IMAGE_TAG: $CI_COMMIT_SHORT_SHA

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker build -t $DOCKER_REGISTRY/$CI_PROJECT_PATH:$IMAGE_TAG .
    - docker push $DOCKER_REGISTRY/$CI_PROJECT_PATH:$IMAGE_TAG

deploy:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache git curl
    - git config --global user.email "[email protected]"
    - git config --global user.name "GitLab CI"
  script:
    - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/your-org/gitops-repo.git
    - cd gitops-repo/environments/production
    - sed -i "s|image:.*|image: $DOCKER_REGISTRY/$CI_PROJECT_PATH:$IMAGE_TAG|g" deployment.yaml
    - git add .
    - git commit -m "Update image to $IMAGE_TAG"
    - git push origin main
  only:
    - main

Monitoring and Observability

ArgoCD Metrics with Prometheus

Configure Prometheus to scrape ArgoCD metrics:

# prometheus-scrape-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
    - job_name: 'argocd-metrics'
      static_configs:
      - targets:
        - argocd-metrics.argocd.svc.cluster.local:8082
    - job_name: 'argocd-server-metrics'
      static_configs:
      - targets:
        - argocd-server-metrics.argocd.svc.cluster.local:8083
    - job_name: 'argocd-repo-server-metrics'
      static_configs:
      - targets:
        - argocd-repo-server.argocd.svc.cluster.local:8084

Grafana Dashboard for ArgoCD

Import ArgoCD dashboard to Grafana:

{
  "dashboard": {
    "title": "ArgoCD Dashboard",
    "panels": [
      {
        "title": "Application Sync Status",
        "targets": [
          {
            "expr": "sum by (sync_status) (argocd_app_info)"
          }
        ]
      },
      {
        "title": "Application Health Status",
        "targets": [
          {
            "expr": "sum by (health_status) (argocd_app_health_total)"
          }
        ]
      },
      {
        "title": "Sync Operations",
        "targets": [
          {
            "expr": "sum(rate(argocd_app_sync_total[5m])) by (name, phase)"
          }
        ]
      }
    ]
  }
}

Notification Configuration

Set up notifications for important events:

# argocd-notifications-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token
  template.app-deployed: |
    message: |
      {{if eq .serviceType "slack"}}:white_check_mark:{{end}} Application {{.app.metadata.name}} is now running new version.
  template.app-health-degraded: |
    message: |
      {{if eq .serviceType "slack"}}:exclamation:{{end}} Application {{.app.metadata.name}} has degraded health.
  template.app-sync-failed: |
    message: |
      {{if eq .serviceType "slack"}}:exclamation:{{end}} Application {{.app.metadata.name}} sync failed.
  trigger.on-deployed: |
    - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
      send: [app-deployed]
  trigger.on-health-degraded: |
    - when: app.status.health.status == 'Degraded'
      send: [app-health-degraded]
  trigger.on-sync-failed: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [app-sync-failed]
  subscriptions: |
    - recipients:
      - slack:devops-channel
      triggers:
      - on-deployed
      - on-health-degraded
      - on-sync-failed

Security Best Practices

Implementing Security Policies

Use Open Policy Agent (OPA) for policy enforcement:

# opa-policies.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-opa-policies
  namespace: argocd
data:
  policy.rego: |
    package argocd
    
    # Deny deployments without resource limits
    deny[msg] {
      input.kind == "Deployment"
      container := input.spec.template.spec.containers[_]
      not container.resources.limits.memory
      msg := sprintf("Container %v does not have memory limits", [container.name])
    }
    
    # Require specific labels
    deny[msg] {
      required_labels := ["app", "version", "environment"]
      provided_labels := input.metadata.labels
      missing := required_labels[_]
      not provided_labels[missing]
      msg := sprintf("Missing required label: %v", [missing])
    }
    
    # Prevent privilege escalation
    deny[msg] {
      input.kind == "Deployment"
      container := input.spec.template.spec.containers[_]
      container.securityContext.privileged == true
      msg := sprintf("Container %v has privileged access", [container.name])
    }

Network Policies

Implement network segmentation:

# network-policies.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: argocd-network-policy
  namespace: argocd
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: argocd
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: argocd
    - podSelector: {}
    ports:
    - protocol: TCP
      port: 8083
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: TCP
      port: 443
    - protocol: TCP
      port: 22

Troubleshooting Common Issues

Debugging Sync Failures

# Check application sync status
argocd app get sample-app-production

# View sync details
argocd app sync sample-app-production --dry-run

# Check application logs
kubectl logs -n argocd deployment/argocd-application-controller

# Force refresh
argocd app get sample-app-production --refresh

# Debug resource differences
argocd app diff sample-app-production

Performance Optimization

Configure ArgoCD for large-scale deployments:

# argocd-cm-performance.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  # Increase resource limits
  controller.status.processors: "50"
  controller.operation.processors: "25"
  controller.self.heal.timeout.seconds: "300"
  controller.repo.server.timeout.seconds: "300"
  
  # Enable caching
  reposerver.disable.helm.manifest.max.extracted.size: "false"
  reposerver.parallelism.limit: "10"
  
  # Optimize Git operations
  timeout.reconciliation: "180s"
  timeout.hard.reconciliation: "0s"

Disaster Recovery

Backup and Restore Procedures

Implement backup strategies:

#!/bin/bash
# backup-argocd.sh

# Backup ArgoCD resources
kubectl get applications -n argocd -o yaml > argocd-applications-backup.yaml
kubectl get appprojects -n argocd -o yaml > argocd-projects-backup.yaml
kubectl get repositories -n argocd -o yaml > argocd-repositories-backup.yaml

# Backup secrets (encrypted)
kubectl get secrets -n argocd -o yaml | \
  kubeseal -o yaml > argocd-sealed-secrets-backup.yaml

# Backup ConfigMaps
kubectl get configmaps -n argocd -o yaml > argocd-configmaps-backup.yaml

# Create backup archive
tar -czf argocd-backup-$(date +%Y%m%d-%H%M%S).tar.gz *.yaml

# Upload to S3 (optional)
aws s3 cp argocd-backup-*.tar.gz s3://backup-bucket/argocd/

Best Practices Summary

1. Repository Structure: Maintain clear separation between applications, environments, and clusters
2. Version Control: Tag releases and maintain proper branching strategies
3. Security: Use RBAC, sealed secrets, and network policies
4. Monitoring: Implement comprehensive monitoring and alerting
5. Documentation: Keep README files updated in your GitOps repository
6. Testing: Always test changes in development before production
7. Automation: Minimize manual interventions
8. Disaster Recovery: Regular backups and tested restore procedures

Conclusion

Implementing GitOps with ArgoCD on Rocky Linux provides a robust, scalable, and secure way to manage Kubernetes deployments. By following the practices outlined in this guide, you’ll have a production-ready GitOps workflow that enhances deployment reliability, improves security, and accelerates your development velocity.

Remember that GitOps is a journey, not a destination. Start simple, iterate often, and gradually expand your GitOps practices as your team becomes more comfortable with the workflow.