Building Secure CI/CD Pipeline with Tekton on Rocky Linux

Tekton provides a powerful, Kubernetes-native CI/CD framework that enables building secure, scalable pipelines on Rocky Linux. This comprehensive guide walks through implementing production-ready CI/CD pipelines with security best practices, automation, and enterprise features.

Understanding Tekton Architecture

Tekton is a cloud-native CI/CD solution that runs on Kubernetes, providing:

• Kubernetes Native: Leverages CRDs for pipeline definitions
• Declarative: YAML-based pipeline configuration
• Reusable: Shared tasks and pipeline components
• Scalable: Horizontal scaling with Kubernetes
• Secure: Built-in security features and RBAC integration

Core Components

1. Tasks: Individual units of work (build, test, deploy)
2. Pipelines: Ordered series of tasks
3. TaskRuns: Execution instances of tasks
4. PipelineRuns: Execution instances of pipelines
5. Triggers: Event-driven pipeline execution

Prerequisites

Before implementing Tekton pipelines:

• Rocky Linux 9 with Kubernetes cluster
• kubectl and helm installed
• Container registry access
• Git repository for source code
• Basic Kubernetes knowledge
• SSL certificates for secure endpoints

Setting Up Kubernetes on Rocky Linux

Installing Kubernetes

# Disable SELinux temporarily for installation
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

# Install required packages
sudo dnf install -y epel-release
sudo dnf install -y kubelet kubeadm kubectl

# Configure kernel modules
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# Configure sysctl parameters
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

# Initialize Kubernetes cluster
sudo kubeadm init --pod-network-cidr=10.244.0.0/16

# Configure kubectl for user
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Install network plugin (Flannel)
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Installing Tekton

Core Components Installation

# Install Tekton Pipelines
kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

# Install Tekton Triggers
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml

# Install Tekton Dashboard
kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/release-full.yaml

# Verify installation
kubectl get pods --namespace tekton-pipelines
kubectl get pods --namespace tekton-pipelines-resolvers

# Install Tekton CLI
curl -LO https://github.com/tektoncd/cli/releases/latest/download/tkn_0.35.0_Linux_x86_64.tar.gz
sudo tar xvzf tkn_0.35.0_Linux_x86_64.tar.gz -C /usr/local/bin/ tkn

Configuring Secure Access

# tekton-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tekton-dashboard
  namespace: tekton-pipelines
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: tekton-basic-auth
    nginx.ingress.kubernetes.io/auth-realm: "Tekton Dashboard"
spec:
  tls:
  - hosts:
    - tekton.example.com
    secretName: tekton-tls
  rules:
  - host: tekton.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: tekton-dashboard
            port:
              number: 9097

Creating Secure Pipeline Components

Service Account with RBAC

# tekton-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tekton-pipeline-sa
  namespace: default
secrets:
- name: docker-registry-creds
- name: git-credentials
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: tekton-pipeline-role
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps", "secrets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["tekton.dev"]
  resources: ["tasks", "pipelines", "taskruns", "pipelineruns"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tekton-pipeline-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: tekton-pipeline-role
subjects:
- kind: ServiceAccount
  name: tekton-pipeline-sa
  namespace: default

Secure Credentials Management

# docker-registry-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: docker-registry-creds
  namespace: default
  annotations:
    tekton.dev/docker-0: https://index.docker.io/v1/
type: kubernetes.io/basic-auth
stringData:
  username: your-registry-username
  password: your-registry-password
---
# git-credentials-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: git-credentials
  namespace: default
  annotations:
    tekton.dev/git-0: https://github.com
type: kubernetes.io/basic-auth
stringData:
  username: your-github-username
  password: your-github-token

Building Reusable Tasks

Source Code Scanning Task

# task-security-scan.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: security-scan
  namespace: default
spec:
  description: Scan source code for vulnerabilities
  params:
  - name: source-path
    type: string
    default: /workspace/source
  - name: severity-threshold
    type: string
    default: "HIGH"
  workspaces:
  - name: source
  steps:
  - name: trivy-scan
    image: aquasecurity/trivy:latest
    script: |
      #!/bin/bash
      set -e
      
      echo "Scanning filesystem for vulnerabilities..."
      trivy filesystem --severity $(params.severity-threshold) \
        --exit-code 1 \
        --no-progress \
        --format json \
        --output /workspace/scan-results.json \
        $(params.source-path)
      
      echo "Scan completed. Results:"
      cat /workspace/scan-results.json | jq .
  - name: sonarqube-scan
    image: sonarsource/sonar-scanner-cli:latest
    env:
    - name: SONAR_HOST_URL
      valueFrom:
        secretKeyRef:
          name: sonarqube-config
          key: url
    - name: SONAR_TOKEN
      valueFrom:
        secretKeyRef:
          name: sonarqube-config
          key: token
    script: |
      #!/bin/bash
      cd $(params.source-path)
      sonar-scanner \
        -Dsonar.projectKey=my-project \
        -Dsonar.sources=. \
        -Dsonar.host.url=$SONAR_HOST_URL \
        -Dsonar.token=$SONAR_TOKEN

Container Build Task with Security

# task-build-secure.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: build-secure
  namespace: default
spec:
  description: Build container image with security checks
  params:
  - name: IMAGE
    type: string
  - name: DOCKERFILE
    type: string
    default: ./Dockerfile
  workspaces:
  - name: source
  - name: docker-credentials
    mountPath: /kaniko/.docker
  results:
  - name: IMAGE_DIGEST
    description: Digest of the built image
  steps:
  - name: dockerfile-lint
    image: hadolint/hadolint:latest-alpine
    script: |
      #!/bin/sh
      echo "Linting Dockerfile..."
      hadolint $(params.DOCKERFILE) || exit 1
  - name: build-image
    image: gcr.io/kaniko-project/executor:latest
    args:
    - --context=$(workspaces.source.path)
    - --dockerfile=$(params.DOCKERFILE)
    - --destination=$(params.IMAGE)
    - --digest-file=/tekton/results/IMAGE_DIGEST
    - --cache=true
    - --cache-ttl=24h
    - --snapshotMode=redo
    - --use-new-run
  - name: scan-image
    image: aquasecurity/trivy:latest
    script: |
      #!/bin/bash
      echo "Scanning container image for vulnerabilities..."
      trivy image --severity CRITICAL,HIGH \
        --exit-code 1 \
        --no-progress \
        $(params.IMAGE)@$(cat /tekton/results/IMAGE_DIGEST)

Deployment Task with Verification

# task-deploy-verify.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: deploy-verify
  namespace: default
spec:
  description: Deploy application and verify deployment
  params:
  - name: deployment-name
    type: string
  - name: image
    type: string
  - name: namespace
    type: string
    default: default
  steps:
  - name: deploy
    image: bitnami/kubectl:latest
    script: |
      #!/bin/bash
      set -e
      
      # Update deployment image
      kubectl set image deployment/$(params.deployment-name) \
        app=$(params.image) \
        -n $(params.namespace)
      
      # Wait for rollout to complete
      kubectl rollout status deployment/$(params.deployment-name) \
        -n $(params.namespace) \
        --timeout=300s
  - name: verify-deployment
    image: curlimages/curl:latest
    script: |
      #!/bin/sh
      set -e
      
      # Get service endpoint
      SERVICE_IP=$(kubectl get svc $(params.deployment-name) \
        -n $(params.namespace) \
        -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
      
      # Health check
      for i in {1..30}; do
        if curl -f http://${SERVICE_IP}/health; then
          echo "Deployment verified successfully"
          exit 0
        fi
        echo "Waiting for application to be ready..."
        sleep 10
      done
      
      echo "Deployment verification failed"
      exit 1

Creating Secure CI/CD Pipeline

Complete Pipeline Definition

# pipeline-secure-cicd.yaml
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: secure-cicd-pipeline
  namespace: default
spec:
  description: Secure CI/CD pipeline with all checks
  params:
  - name: git-repo-url
    type: string
  - name: git-revision
    type: string
    default: main
  - name: image-name
    type: string
  - name: deployment-name
    type: string
  workspaces:
  - name: shared-workspace
  - name: docker-credentials
  - name: git-credentials
  tasks:
  - name: fetch-source
    taskRef:
      name: git-clone
      kind: ClusterTask
    workspaces:
    - name: output
      workspace: shared-workspace
    - name: basic-auth
      workspace: git-credentials
    params:
    - name: url
      value: $(params.git-repo-url)
    - name: revision
      value: $(params.git-revision)
  
  - name: security-scan
    taskRef:
      name: security-scan
    runAfter:
    - fetch-source
    workspaces:
    - name: source
      workspace: shared-workspace
    params:
    - name: source-path
      value: /workspace/source
  
  - name: run-tests
    taskRef:
      name: test-runner
    runAfter:
    - security-scan
    workspaces:
    - name: source
      workspace: shared-workspace
  
  - name: build-image
    taskRef:
      name: build-secure
    runAfter:
    - run-tests
    workspaces:
    - name: source
      workspace: shared-workspace
    - name: docker-credentials
      workspace: docker-credentials
    params:
    - name: IMAGE
      value: $(params.image-name):$(tasks.fetch-source.results.commit)
  
  - name: deploy-staging
    taskRef:
      name: deploy-verify
    runAfter:
    - build-image
    params:
    - name: deployment-name
      value: $(params.deployment-name)-staging
    - name: image
      value: $(params.image-name)@$(tasks.build-image.results.IMAGE_DIGEST)
    - name: namespace
      value: staging
  
  - name: integration-tests
    taskRef:
      name: integration-test-runner
    runAfter:
    - deploy-staging
    params:
    - name: endpoint
      value: http://$(params.deployment-name)-staging.staging.svc.cluster.local
  
  - name: deploy-production
    taskRef:
      name: deploy-verify
    runAfter:
    - integration-tests
    params:
    - name: deployment-name
      value: $(params.deployment-name)
    - name: image
      value: $(params.image-name)@$(tasks.build-image.results.IMAGE_DIGEST)
    - name: namespace
      value: production

Implementing Pipeline Triggers

GitHub Webhook Integration

# trigger-github-webhook.yaml
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
  name: github-pipeline-template
  namespace: default
spec:
  params:
  - name: git-repo-url
  - name: git-revision
  - name: git-repo-name
  resourcetemplates:
  - apiVersion: tekton.dev/v1beta1
    kind: PipelineRun
    metadata:
      generateName: $(tt.params.git-repo-name)-run-
    spec:
      pipelineRef:
        name: secure-cicd-pipeline
      params:
      - name: git-repo-url
        value: $(tt.params.git-repo-url)
      - name: git-revision
        value: $(tt.params.git-revision)
      - name: image-name
        value: registry.example.com/$(tt.params.git-repo-name)
      - name: deployment-name
        value: $(tt.params.git-repo-name)
      workspaces:
      - name: shared-workspace
        volumeClaimTemplate:
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
      - name: docker-credentials
        secret:
          secretName: docker-registry-creds
      - name: git-credentials
        secret:
          secretName: git-credentials
      serviceAccountName: tekton-pipeline-sa
---
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata:
  name: github-push-binding
  namespace: default
spec:
  params:
  - name: git-repo-url
    value: $(body.repository.clone_url)
  - name: git-revision
    value: $(body.after)
  - name: git-repo-name
    value: $(body.repository.name)
---
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
  name: github-listener
  namespace: default
spec:
  serviceAccountName: tekton-triggers-sa
  triggers:
  - name: github-push-events
    interceptors:
    - ref:
        name: github
      params:
      - name: secretRef
        value:
          secretName: github-webhook-secret
          secretKey: webhook-secret
      - name: eventTypes
        value: ["push", "pull_request"]
    bindings:
    - ref: github-push-binding
    template:
      ref: github-pipeline-template

Securing Webhook Endpoint

# Create webhook secret
kubectl create secret generic github-webhook-secret \
  --from-literal=webhook-secret=$(openssl rand -base64 32) \
  -n default

# Create ingress for webhook
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tekton-webhooks
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - webhooks.example.com
    secretName: webhooks-tls
  rules:
  - host: webhooks.example.com
    http:
      paths:
      - path: /hooks
        pathType: Prefix
        backend:
          service:
            name: el-github-listener
            port:
              number: 8080
EOF

Advanced Security Features

Secret Scanning Prevention

# task-secret-scan.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: secret-scan
  namespace: default
spec:
  description: Scan for exposed secrets and credentials
  workspaces:
  - name: source
  steps:
  - name: gitleaks-scan
    image: zricethezav/gitleaks:latest
    script: |
      #!/bin/bash
      set -e
      
      echo "Scanning for exposed secrets..."
      gitleaks detect --source=/workspace/source --verbose
      
      if [ $? -eq 0 ]; then
        echo "No secrets found"
      else
        echo "CRITICAL: Exposed secrets detected!"
        exit 1
      fi
  - name: trufflehog-scan
    image: trufflesecurity/trufflehog:latest
    script: |
      #!/bin/bash
      cd /workspace/source
      trufflehog filesystem . --fail

SBOM Generation and Signing

# task-sbom-generate.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: generate-sbom
  namespace: default
spec:
  description: Generate and sign Software Bill of Materials
  params:
  - name: image
    type: string
  workspaces:
  - name: source
  steps:
  - name: generate-sbom
    image: anchore/syft:latest
    script: |
      #!/bin/bash
      set -e
      
      echo "Generating SBOM..."
      syft $(params.image) -o spdx-json > /workspace/sbom.json
      
      echo "SBOM generated successfully"
      cat /workspace/sbom.json | jq '.name, .creationInfo'
  - name: sign-sbom
    image: gcr.io/projectsigstore/cosign:latest
    env:
    - name: COSIGN_EXPERIMENTAL
      value: "1"
    script: |
      #!/bin/bash
      set -e
      
      echo "Signing SBOM with Cosign..."
      cosign sign-blob /workspace/sbom.json \
        --output-signature /workspace/sbom.sig \
        --output-certificate /workspace/sbom.cert
      
      echo "Uploading SBOM to registry..."
      cosign attach sbom --sbom /workspace/sbom.json $(params.image)

Monitoring and Observability

Pipeline Metrics Collection

# tekton-metrics-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-observability
  namespace: tekton-pipelines
data:
  _example: |
    metrics.backend-destination: prometheus
    metrics.taskrun.level: "task"
    metrics.taskrun.duration-type: "histogram"
    metrics.pipelinerun.level: "pipeline"
    metrics.pipelinerun.duration-type: "histogram"
---
apiVersion: v1
kind: Service
metadata:
  name: tekton-pipelines-controller-metrics
  namespace: tekton-pipelines
  labels:
    app: tekton-pipelines-controller
spec:
  ports:
  - name: metrics
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app.kubernetes.io/name: controller
    app.kubernetes.io/component: controller
    app.kubernetes.io/part-of: tekton-pipelines

Grafana Dashboard Configuration

{
  "dashboard": {
    "title": "Tekton CI/CD Pipeline Metrics",
    "panels": [
      {
        "title": "Pipeline Success Rate",
        "targets": [
          {
            "expr": "sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_count{status=\"success\"}[5m])) / sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_count[5m])) * 100"
          }
        ]
      },
      {
        "title": "Average Pipeline Duration",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])) by (le))"
          }
        ]
      },
      {
        "title": "Task Failure Rate",
        "targets": [
          {
            "expr": "sum(rate(tekton_pipelines_controller_taskrun_duration_seconds_count{status=\"failed\"}[5m])) by (task)"
          }
        ]
      }
    ]
  }
}

Best Practices and Security Hardening

Pipeline Security Checklist

1. Access Control

   • Implement RBAC for pipeline resources
   • Use separate service accounts per environment
   • Rotate credentials regularly

2. Image Security

   • Scan all container images
   • Sign images with Cosign
   • Use admission controllers for policy enforcement

3. Secret Management

   • Use external secret stores (Vault, AWS Secrets Manager)
   • Never hardcode secrets in pipeline definitions
   • Implement secret rotation

4. Network Security

   • Use NetworkPolicies to restrict pod communication
   • Enable TLS for all endpoints
   • Implement webhook signature verification

5. Audit and Compliance

   • Enable audit logging for all pipeline activities
   • Generate SBOMs for all deployments
   • Implement compliance scanning

Resource Limits and Quotas

# pipeline-resource-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: pipeline-quota
  namespace: default
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    persistentvolumeclaims: "10"
    pods: "50"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: pipeline-limits
  namespace: default
spec:
  limits:
  - max:
      cpu: "2"
      memory: 4Gi
    min:
      cpu: 100m
      memory: 128Mi
    default:
      cpu: 500m
      memory: 1Gi
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    type: Container

Troubleshooting Common Issues

Debugging Failed Pipeline Runs

# List recent pipeline runs
tkn pipelinerun list

# Get detailed logs for a specific run
tkn pipelinerun logs my-pipeline-run-xyz -f

# Describe pipeline run for status
kubectl describe pipelinerun my-pipeline-run-xyz

# Check task run details
tkn taskrun list
tkn taskrun logs my-task-run-abc

# Debug pod issues
kubectl get pods -l tekton.dev/pipelineRun=my-pipeline-run-xyz
kubectl logs -l tekton.dev/pipelineRun=my-pipeline-run-xyz --all-containers

Performance Optimization

# parallel-tasks-pipeline.yaml
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: optimized-pipeline
spec:
  tasks:
  - name: unit-tests
    taskRef:
      name: run-tests
    params:
    - name: test-type
      value: unit
  - name: integration-tests
    taskRef:
      name: run-tests
    params:
    - name: test-type
      value: integration
  - name: security-scan
    taskRef:
      name: security-scan
  # These tasks run in parallel after source is fetched
  - name: build-app
    taskRef:
      name: build-secure
    runAfter:
    - unit-tests
    - integration-tests
    - security-scan

Conclusion

Building secure CI/CD pipelines with Tekton on Rocky Linux provides a robust, cloud-native solution for modern software delivery. By implementing security checks at every stage, using proper RBAC, and following best practices, you can create enterprise-grade pipelines that ensure both speed and security in your deployment process.

The combination of Tekton’s Kubernetes-native architecture with Rocky Linux’s stability creates a powerful platform for continuous integration and deployment that scales with your needs while maintaining security and compliance requirements.