🔍 Jaeger Distributed Tracing Setup on AlmaLinux 9: Complete Installation and Configuration Guide

Welcome to the exciting world of distributed tracing! 🎉 Today we’re going to learn how to set up Jaeger on AlmaLinux 9, the powerful tracing platform that helps you monitor and troubleshoot your microservices. Think of Jaeger as your detective tool that follows requests through your entire application stack! 🕵️‍♂️✨

🤔 Why is Jaeger Important?

Distributed tracing is like having X-ray vision for your applications! 👀 When you have multiple services talking to each other, it becomes really hard to understand what’s happening when something goes wrong. Here’s why Jaeger is awesome:

• 🔍 Track requests across multiple services and see the complete journey
• ⚡ Find performance bottlenecks and slow database queries instantly
• 🐛 Debug complex issues that span multiple microservices
• 📊 Visualize service dependencies and understand your architecture
• 🎯 Monitor SLA compliance and set up alerts for slow operations
• 💡 Optimize performance by identifying the slowest parts of your system

🎯 What You Need

Before we start our Jaeger adventure, let’s make sure you have everything ready:

✅ AlmaLinux 9 system (fresh installation recommended)
✅ Root or sudo access for installing packages
✅ At least 4GB RAM (8GB recommended for production)
✅ 10GB free disk space for traces storage
✅ Internet connection for downloading packages
✅ Basic terminal knowledge (don’t worry, we’ll explain everything!)
✅ Docker installed (we’ll cover this if you don’t have it)

📝 Step 1: Update Your AlmaLinux System

Let’s start by making sure your system is up to date! 🚀

# Update all packages to latest versions
sudo dnf update -y

# Install essential development tools
sudo dnf groupinstall "Development Tools" -y

# Install helpful utilities we'll need
sudo dnf install -y curl wget git vim htop

Perfect! Your system is now ready for Jaeger installation! ✨

🔧 Step 2: Install Docker and Docker Compose

Jaeger works great with Docker containers! Let’s set it up:

# Install Docker from official repository
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# Install Docker Engine
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Start and enable Docker service
sudo systemctl start docker
sudo systemctl enable docker

# Add your user to docker group (no more sudo needed!)
sudo usermod -aG docker $USER

# Apply group changes (or logout/login)
newgrp docker

# Test Docker installation
docker --version
docker compose version

Great! Docker is ready to run Jaeger! 🐳

🌟 Step 3: Deploy Jaeger All-in-One

Let’s start with Jaeger’s all-in-one deployment - perfect for learning and development:

# Create directory for Jaeger configuration
mkdir -p ~/jaeger-setup
cd ~/jaeger-setup

# Create docker-compose.yml file for Jaeger
cat > docker-compose.yml << 'EOF'
version: '3.8'

services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    container_name: jaeger-allinone
    ports:
      - "16686:16686"   # Jaeger UI
      - "14268:14268"   # HTTP collector
      - "6831:6831/udp" # UDP Thrift compact
      - "6832:6832/udp" # UDP Thrift binary  
      - "5778:5778"     # HTTP configuration
      - "14250:14250"   # gRPC
    environment:
      - COLLECTOR_OTLP_ENABLED=true
      - COLLECTOR_ZIPKIN_HOST_PORT=9411
    volumes:
      - jaeger-data:/tmp
    networks:
      - jaeger-net
    restart: unless-stopped

volumes:
  jaeger-data:

networks:
  jaeger-net:
    driver: bridge
EOF

# Start Jaeger services
docker compose up -d

# Check if services are running
docker compose ps

Awesome! Jaeger is now running! 🎉

✅ Step 4: Verify Jaeger Installation

Let’s make sure everything is working perfectly:

# Check container status
docker ps | grep jaeger

# View Jaeger logs to ensure it started correctly
docker compose logs jaeger

# Test if Jaeger UI is accessible
curl -s http://localhost:16686/api/services | head -n 5

# Check all exposed ports are listening
ss -tuln | grep -E "(16686|14268|6831|6832)"

You should see the Jaeger container running and all ports listening! 🚀

Open your web browser and visit: http://your-server-ip:16686 to see the beautiful Jaeger UI! ✨

🔧 Step 5: Configure Jaeger for Production

For production use, let’s create a more robust configuration:

# Create production docker-compose file
cat > docker-compose-prod.yml << 'EOF'
version: '3.8'

services:
  elasticsearch:
    image: elasticsearch:7.17.0
    container_name: jaeger-elasticsearch
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
      - xpack.security.enabled=false
    ports:
      - "9200:9200"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    networks:
      - jaeger-net

  jaeger-collector:
    image: jaegertracing/jaeger-collector:latest
    container_name: jaeger-collector
    ports:
      - "14269:14269"
      - "14268:14268"
      - "14250:14250"
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - ES_SERVER_URLS=http://elasticsearch:9200
      - COLLECTOR_OTLP_ENABLED=true
    depends_on:
      - elasticsearch
    networks:
      - jaeger-net
    restart: unless-stopped

  jaeger-agent:
    image: jaegertracing/jaeger-agent:latest
    container_name: jaeger-agent
    ports:
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778"
    environment:
      - REPORTER_GRPC_HOST_PORT=jaeger-collector:14250
    depends_on:
      - jaeger-collector
    networks:
      - jaeger-net
    restart: unless-stopped

  jaeger-query:
    image: jaegertracing/jaeger-query:latest
    container_name: jaeger-query
    ports:
      - "16686:16686"
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - ES_SERVER_URLS=http://elasticsearch:9200
    depends_on:
      - elasticsearch
    networks:
      - jaeger-net
    restart: unless-stopped

volumes:
  elasticsearch-data:

networks:
  jaeger-net:
    driver: bridge
EOF

# Deploy production setup (optional)
# docker compose -f docker-compose-prod.yml up -d

This gives you a scalable production setup with Elasticsearch backend! 💪

🎮 Quick Examples

Let’s try some practical examples to see Jaeger in action! 🚀

Example 1: Simple Python Application with Tracing

# Create a simple Python app directory
mkdir ~/python-jaeger-demo
cd ~/python-jaeger-demo

# Create requirements.txt
cat > requirements.txt << 'EOF'
opentelemetry-api==1.20.0
opentelemetry-sdk==1.20.0
opentelemetry-exporter-jaeger-thrift==1.20.0
opentelemetry-instrumentation-requests==0.41b0
flask==2.3.3
requests==2.31.0
EOF

# Install Python packages
pip3 install -r requirements.txt

# Create demo application
cat > app.py << 'EOF'
#!/usr/bin/env python3
import time
import requests
from flask import Flask
from opentelemetry import trace
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Configure tracing
resource = Resource(attributes={SERVICE_NAME: "python-demo-app"})
trace.set_tracer_provider(TracerProvider(resource=resource))
tracer = trace.get_tracer(__name__)

jaeger_exporter = JaegerExporter(
    agent_host_name="localhost",
    agent_port=6831,
)

span_processor = BatchSpanProcessor(jaeger_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

app = Flask(__name__)

@app.route('/')
def hello():
    with tracer.start_as_current_span("hello-operation"):
        with tracer.start_as_current_span("database-query"):
            time.sleep(0.1)  # Simulate database query
        with tracer.start_as_current_span("external-api-call"):
            time.sleep(0.05)  # Simulate API call
        return "Hello from traced Python app! 🚀"

@app.route('/slow')
def slow_endpoint():
    with tracer.start_as_current_span("slow-operation"):
        time.sleep(2)  # Simulate slow operation
        return "This was slow! ⏰"

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)
EOF

# Run the demo app
python3 app.py &

# Generate some traces
curl http://localhost:5000/
curl http://localhost:5000/slow
curl http://localhost:5000/

Now check the Jaeger UI at http://localhost:16686 and search for the python-demo-app service! 🎯

Example 2: Node.js Microservice with Tracing

# Create Node.js demo
mkdir ~/nodejs-jaeger-demo
cd ~/nodejs-jaeger-demo

# Initialize npm project
npm init -y

# Install dependencies
npm install express @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-jaeger

# Create traced Node.js app
cat > server.js << 'EOF'
const express = require('express');
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');

// Initialize tracing
const jaegerExporter = new JaegerExporter({
  endpoint: 'http://localhost:14268/api/traces',
});

const sdk = new NodeSDK({
  serviceName: 'nodejs-demo-app',
  traceExporter: jaegerExporter,
});

sdk.start();

const app = express();

app.get('/', (req, res) => {
  // Simulate some processing
  setTimeout(() => {
    res.json({ 
      message: 'Hello from traced Node.js app! 🚀',
      timestamp: new Date().toISOString()
    });
  }, 100);
});

app.get('/api/users', (req, res) => {
  // Simulate database query
  setTimeout(() => {
    res.json([
      { id: 1, name: 'Alice' },
      { id: 2, name: 'Bob' }
    ]);
  }, 200);
});

app.listen(3000, () => {
  console.log('Traced Node.js app running on port 3000! 🚀');
});
EOF

# Run the Node.js app
node server.js &

# Test the endpoints
curl http://localhost:3000/
curl http://localhost:3000/api/users

Check Jaeger UI for the nodejs-demo-app service traces! 📊

Example 3: Kubernetes Deployment with Jaeger

# Create Kubernetes manifests directory
mkdir ~/k8s-jaeger
cd ~/k8s-jaeger

# Create Jaeger Operator deployment
cat > jaeger-operator.yaml << 'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: observability
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-operator
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      name: jaeger-operator
  template:
    metadata:
      labels:
        name: jaeger-operator
    spec:
      containers:
      - name: jaeger-operator
        image: jaegertracing/jaeger-operator:latest
        ports:
        - containerPort: 8383
        env:
        - name: WATCH_NAMESPACE
          value: ""
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger-operator-metrics
  namespace: observability
spec:
  ports:
  - name: http-metrics
    port: 8383
    targetPort: 8383
  selector:
    name: jaeger-operator
EOF

# Apply if you have Kubernetes cluster
# kubectl apply -f jaeger-operator.yaml

This shows you how to deploy Jaeger in Kubernetes! ⚓

🚨 Fix Common Problems

Here are solutions to the most common Jaeger issues you might encounter:

Problem 1: Jaeger UI Not Accessible 🔒

Symptoms: Can’t access http://localhost:16686

Solutions:

# Check if container is running
docker ps | grep jaeger

# Check port binding
docker port jaeger-allinone

# Restart Jaeger if needed
docker compose restart jaeger

# Check firewall (if applicable)
sudo firewall-cmd --permanent --add-port=16686/tcp
sudo firewall-cmd --reload

Problem 2: No Traces Appearing 📊

Symptoms: Services sending traces but nothing shows in UI

Solutions:

# Check collector endpoint is reachable
curl -v http://localhost:14268/api/traces

# Verify agent ports are listening
ss -tuln | grep 6831

# Check application tracing configuration
# Make sure span_processor is added correctly

# View Jaeger logs for errors
docker compose logs jaeger | grep -i error

Problem 3: High Memory Usage 💾

Symptoms: Jaeger consuming too much memory

Solutions:

# Limit memory in docker-compose.yml
cat >> docker-compose.yml << 'EOF'
services:
  jaeger:
    mem_limit: 1g
    mem_reservation: 512m
EOF

# Configure sampling rate to reduce traces
docker run -d \
  -e JAEGER_SAMPLER_TYPE=probabilistic \
  -e JAEGER_SAMPLER_PARAM=0.1 \
  jaegertracing/all-in-one

# Clean up old traces periodically
# (This happens automatically with TTL settings)

Problem 4: Application Not Sending Traces 📤

Symptoms: App runs but no traces in Jaeger

Solutions:

# Verify tracing initialization in your code
# Check this Python example:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

# Make sure you set the tracer provider!
trace.set_tracer_provider(TracerProvider())

# Test connectivity to Jaeger agent
telnet localhost 6831  # Should connect

# Check for firewall blocking UDP ports
sudo iptables -L | grep 6831

📋 Simple Commands Summary

Here’s your quick reference guide for managing Jaeger:

💡 Tips for Success

Here are some pro tips to get the most out of Jaeger! 🌟

🎯 Configure Smart Sampling: Don’t trace every request in production - use probabilistic sampling (10-20%) to reduce overhead while capturing enough data for analysis.

⚡ Use Consistent Service Names: Make sure all instances of the same service use identical service names in their tracing configuration for proper aggregation.

📊 Tag Important Operations: Add custom tags to spans for better filtering - like user_id, region, or feature flags.

🔍 Monitor Jaeger Itself: Set up monitoring for Jaeger components using Prometheus metrics endpoints.

💾 Plan Storage Strategy: Traces can grow large quickly - implement proper retention policies and consider using Elasticsearch or Cassandra for production.

🚀 Start Simple: Begin with all-in-one deployment for learning, then graduate to production architecture with separate components.

🔗 Integrate with APM: Combine Jaeger with metrics (Prometheus) and logs (ELK stack) for complete observability.

🏆 What You Learned

Congratulations! You’ve successfully mastered Jaeger distributed tracing! 🎉 Here’s everything you accomplished:

✅ Installed Docker and Docker Compose on AlmaLinux 9
✅ Deployed Jaeger all-in-one for development use
✅ Configured production setup with Elasticsearch backend
✅ Created traced applications in Python and Node.js
✅ Explored the Jaeger UI and learned to analyze traces
✅ Set up Kubernetes deployment with Jaeger Operator
✅ Troubleshot common issues and optimized performance
✅ Learned best practices for production deployments

🎯 Why This Matters

Distributed tracing with Jaeger is a game-changer for modern applications! 🚀 You can now:

🔍 Debug Complex Issues: Track requests across microservices and find exactly where problems occur
⚡ Optimize Performance: Identify slow database queries, API calls, and bottlenecks instantly
📊 Understand Dependencies: Visualize how your services interact and identify critical paths
🛡️ Improve Reliability: Set up monitoring and alerts based on trace data to prevent outages
💡 Make Data-Driven Decisions: Use trace analytics to guide architecture and optimization decisions

You’re now equipped with enterprise-grade observability skills that are highly valued in DevOps and SRE roles! The ability to implement and manage distributed tracing makes you a more effective developer and system administrator. ⭐

Keep exploring, keep tracing, and remember - every request tells a story, and now you know how to read it! 🙌✨