📘 Search Engine: Elasticsearch Integration

🎯 Introduction

Welcome to the exciting world of Elasticsearch with TypeScript! 🎉 In this guide, we’ll explore how to build powerful search capabilities into your applications using Elasticsearch and TypeScript together.

You’ll discover how Elasticsearch can transform your application’s search experience. Whether you’re building an e-commerce platform 🛒, a content management system 📚, or a data analytics dashboard 📊, understanding Elasticsearch integration is essential for delivering lightning-fast, relevant search results.

By the end of this tutorial, you’ll feel confident implementing full-text search, faceted navigation, and real-time analytics in your TypeScript projects! Let’s dive in! 🏊‍♂️

📚 Understanding Elasticsearch

🤔 What is Elasticsearch?

Elasticsearch is like having a super-smart librarian 📚 who can instantly find any book in a massive library, understand what you meant even if you misspelled the title, and suggest related books you might enjoy!

In technical terms, Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene. With TypeScript, you get:

• ✨ Type-safe queries and responses
• 🚀 Autocomplete for query building
• 🛡️ Compile-time validation of search parameters

💡 Why Use Elasticsearch with TypeScript?

Here’s why developers love this combination:

1. Lightning-Fast Search ⚡: Millisecond response times even with millions of documents
2. Intelligent Querying 🧠: Fuzzy matching, synonyms, and relevance scoring
3. Real-Time Analytics 📊: Aggregate and analyze data on the fly
4. Type Safety 🔒: Catch query errors before runtime

Real-world example: Imagine building a recipe search engine 🍳. With Elasticsearch, users can search for “chiken parmisan” (misspelled!) and still find “Chicken Parmesan” recipes, plus get suggestions for similar Italian dishes!

🔧 Basic Syntax and Usage

📝 Setting Up the Connection

Let’s start with connecting to Elasticsearch:

// 👋 Hello, Elasticsearch!
import { Client } from '@elastic/elasticsearch';

// 🎨 Define our document type
interface Recipe {
  id: string;
  title: string;
  ingredients: string[];
  cookingTime: number;
  difficulty: 'easy' | 'medium' | 'hard';
  emoji: string; // Every recipe needs an emoji!
}

// 🔌 Create the client
const client = new Client({
  node: 'http://localhost:9200',
  auth: {
    username: 'elastic',
    password: 'your-password'
  }
});

// 🏗️ Create an index with mappings
const createRecipeIndex = async () => {
  await client.indices.create({
    index: 'recipes',
    body: {
      mappings: {
        properties: {
          title: { type: 'text' },
          ingredients: { type: 'text' },
          cookingTime: { type: 'integer' },
          difficulty: { type: 'keyword' },
          emoji: { type: 'keyword' }
        }
      }
    }
  });
  console.log('✅ Recipe index created!');
};

💡 Explanation: We define TypeScript interfaces that match our Elasticsearch mappings, ensuring type safety throughout our application!

🎯 Common Operations

Here are the essential operations you’ll use daily:

// 🏗️ Indexing documents
const indexRecipe = async (recipe: Recipe) => {
  const response = await client.index({
    index: 'recipes',
    id: recipe.id,
    body: recipe
  });
  console.log(`✅ Indexed recipe: ${recipe.emoji} ${recipe.title}`);
  return response;
};

// 🔍 Basic search
const searchRecipes = async (query: string) => {
  const response = await client.search<Recipe>({
    index: 'recipes',
    body: {
      query: {
        multi_match: {
          query,
          fields: ['title^2', 'ingredients'], // Title is 2x more important!
          fuzziness: 'AUTO'
        }
      }
    }
  });
  
  return response.body.hits.hits.map(hit => hit._source!);
};

// 📊 Aggregations for analytics
const getRecipeStats = async () => {
  const response = await client.search({
    index: 'recipes',
    body: {
      size: 0, // We only want aggregations
      aggs: {
        by_difficulty: {
          terms: { field: 'difficulty' }
        },
        avg_cooking_time: {
          avg: { field: 'cookingTime' }
        }
      }
    }
  });
  
  return response.body.aggregations;
};

💡 Practical Examples

🛒 Example 1: E-Commerce Product Search

Let’s build a real product search system:

// 🛍️ Define our product type
interface Product {
  id: string;
  name: string;
  description: string;
  price: number;
  category: string;
  tags: string[];
  inStock: boolean;
  rating: number;
  emoji: string;
}

// 🔍 Advanced search with filters
class ProductSearchEngine {
  private client: Client;
  
  constructor(client: Client) {
    this.client = client;
  }
  
  // 🎯 Search with multiple criteria
  async searchProducts(params: {
    query?: string;
    category?: string;
    minPrice?: number;
    maxPrice?: number;
    inStock?: boolean;
    minRating?: number;
  }) {
    const must: any[] = [];
    const filter: any[] = [];
    
    // 📝 Full-text search
    if (params.query) {
      must.push({
        multi_match: {
          query: params.query,
          fields: ['name^3', 'description', 'tags'],
          fuzziness: 'AUTO'
        }
      });
    }
    
    // 🏷️ Category filter
    if (params.category) {
      filter.push({ term: { category: params.category } });
    }
    
    // 💰 Price range
    if (params.minPrice || params.maxPrice) {
      filter.push({
        range: {
          price: {
            ...(params.minPrice && { gte: params.minPrice }),
            ...(params.maxPrice && { lte: params.maxPrice })
          }
        }
      });
    }
    
    // 📦 Stock status
    if (params.inStock !== undefined) {
      filter.push({ term: { inStock: params.inStock } });
    }
    
    // ⭐ Rating filter
    if (params.minRating) {
      filter.push({ range: { rating: { gte: params.minRating } } });
    }
    
    const response = await this.client.search<Product>({
      index: 'products',
      body: {
        query: {
          bool: {
            must: must.length > 0 ? must : [{ match_all: {} }],
            filter
          }
        },
        sort: [
          { _score: 'desc' },
          { rating: 'desc' }
        ]
      }
    });
    
    return response.body.hits.hits.map(hit => ({
      ...hit._source!,
      score: hit._score
    }));
  }
  
  // 🎨 Get facets for filtering UI
  async getFacets(query?: string) {
    const baseQuery = query ? {
      multi_match: {
        query,
        fields: ['name^3', 'description', 'tags']
      }
    } : { match_all: {} };
    
    const response = await this.client.search({
      index: 'products',
      body: {
        size: 0,
        query: baseQuery,
        aggs: {
          categories: {
            terms: { field: 'category', size: 20 }
          },
          price_ranges: {
            range: {
              field: 'price',
              ranges: [
                { key: '🏷️ Under $25', to: 25 },
                { key: '💵 $25-$50', from: 25, to: 50 },
                { key: '💰 $50-$100', from: 50, to: 100 },
                { key: '💎 Over $100', from: 100 }
              ]
            }
          },
          avg_rating: {
            avg: { field: 'rating' }
          }
        }
      }
    });
    
    return response.body.aggregations;
  }
}

// 🎮 Let's use it!
const searchEngine = new ProductSearchEngine(client);

// Search for gaming keyboards under $100
const results = await searchEngine.searchProducts({
  query: 'gaming keyboard RGB',
  maxPrice: 100,
  inStock: true,
  minRating: 4
});

console.log(`🎮 Found ${results.length} gaming keyboards!`);
results.forEach(product => {
  console.log(`  ${product.emoji} ${product.name} - $${product.price} ⭐${product.rating}`);
});

🎯 Try it yourself: Add autocomplete functionality and search suggestions!

🎮 Example 2: Real-Time Log Analysis

Let’s build a log analysis system:

// 📊 Log entry type
interface LogEntry {
  timestamp: Date;
  level: 'debug' | 'info' | 'warn' | 'error';
  service: string;
  message: string;
  userId?: string;
  requestId?: string;
  duration?: number;
  emoji: string;
}

class LogAnalyzer {
  private client: Client;
  
  constructor(client: Client) {
    this.client = client;
  }
  
  // 📝 Index a log entry
  async log(entry: Omit<LogEntry, 'timestamp' | 'emoji'> & { level: LogEntry['level'] }) {
    const emojiMap = {
      debug: '🐛',
      info: '📘',
      warn: '⚠️',
      error: '🚨'
    };
    
    const logEntry: LogEntry = {
      ...entry,
      timestamp: new Date(),
      emoji: emojiMap[entry.level]
    };
    
    await this.client.index({
      index: `logs-${new Date().toISOString().slice(0, 10)}`, // Daily indices
      body: logEntry
    });
    
    console.log(`${logEntry.emoji} Logged: ${logEntry.message}`);
  }
  
  // 🔍 Search logs with complex queries
  async searchLogs(params: {
    startTime?: Date;
    endTime?: Date;
    levels?: LogEntry['level'][];
    services?: string[];
    searchText?: string;
    userId?: string;
  }) {
    const must: any[] = [];
    
    // ⏰ Time range
    if (params.startTime || params.endTime) {
      must.push({
        range: {
          timestamp: {
            ...(params.startTime && { gte: params.startTime }),
            ...(params.endTime && { lte: params.endTime })
          }
        }
      });
    }
    
    // 🎯 Level filter
    if (params.levels && params.levels.length > 0) {
      must.push({ terms: { level: params.levels } });
    }
    
    // 🏢 Service filter
    if (params.services && params.services.length > 0) {
      must.push({ terms: { service: params.services } });
    }
    
    // 👤 User filter
    if (params.userId) {
      must.push({ term: { userId: params.userId } });
    }
    
    // 📝 Text search
    if (params.searchText) {
      must.push({
        match: {
          message: {
            query: params.searchText,
            operator: 'and'
          }
        }
      });
    }
    
    const response = await this.client.search<LogEntry>({
      index: 'logs-*',
      body: {
        query: { bool: { must } },
        sort: [{ timestamp: 'desc' }],
        size: 100
      }
    });
    
    return response.body.hits.hits.map(hit => hit._source!);
  }
  
  // 📊 Get error statistics
  async getErrorStats(hours: number = 24) {
    const response = await this.client.search({
      index: 'logs-*',
      body: {
        size: 0,
        query: {
          bool: {
            must: [
              { term: { level: 'error' } },
              {
                range: {
                  timestamp: {
                    gte: new Date(Date.now() - hours * 60 * 60 * 1000)
                  }
                }
              }
            ]
          }
        },
        aggs: {
          errors_over_time: {
            date_histogram: {
              field: 'timestamp',
              interval: 'hour'
            }
          },
          by_service: {
            terms: { field: 'service' }
          },
          avg_duration: {
            avg: { field: 'duration' }
          }
        }
      }
    });
    
    return response.body.aggregations;
  }
}

// 🎮 Example usage
const logger = new LogAnalyzer(client);

// Log some events
await logger.log({
  level: 'info',
  service: 'auth-service',
  message: 'User login successful',
  userId: 'user123'
});

await logger.log({
  level: 'error',
  service: 'payment-service',
  message: 'Payment processing failed',
  userId: 'user456',
  duration: 1500
});

// Search for errors
const errors = await logger.searchLogs({
  levels: ['error'],
  startTime: new Date(Date.now() - 60 * 60 * 1000) // Last hour
});

console.log(`🚨 Found ${errors.length} errors in the last hour!`);

🚀 Advanced Concepts

🧙‍♂️ Advanced Topic 1: Custom Analyzers and Tokenizers

When you’re ready to level up, create custom text analysis:

// 🎯 Custom analyzer for product names
const createCustomAnalyzer = async () => {
  await client.indices.create({
    index: 'products_advanced',
    body: {
      settings: {
        analysis: {
          analyzer: {
            product_analyzer: {
              type: 'custom',
              tokenizer: 'standard',
              filter: ['lowercase', 'synonym_filter', 'stop']
            }
          },
          filter: {
            synonym_filter: {
              type: 'synonym',
              synonyms: [
                'laptop,notebook,computer',
                'phone,mobile,smartphone',
                'tv,television,telly'
              ]
            }
          }
        }
      },
      mappings: {
        properties: {
          name: {
            type: 'text',
            analyzer: 'product_analyzer'
          }
        }
      }
    }
  });
  console.log('✨ Custom analyzer created!');
};

// 🪄 Using the custom analyzer
const searchWithSynonyms = async (query: string) => {
  const response = await client.search({
    index: 'products_advanced',
    body: {
      query: {
        match: {
          name: {
            query,
            analyzer: 'product_analyzer'
          }
        }
      }
    }
  });
  
  // Searching for "laptop" will also find "notebook" and "computer"!
  return response.body.hits.hits;
};

🏗️ Advanced Topic 2: Percolator Queries for Real-Time Alerts

For the brave developers, implement reverse search:

// 🚀 Alert system using percolator
interface Alert {
  id: string;
  name: string;
  userId: string;
  query: any;
  emoji: string;
}

class AlertSystem {
  private client: Client;
  
  constructor(client: Client) {
    this.client = client;
  }
  
  // 📝 Register an alert
  async createAlert(alert: Alert) {
    await this.client.index({
      index: 'alerts',
      id: alert.id,
      body: {
        name: alert.name,
        userId: alert.userId,
        emoji: alert.emoji,
        query: alert.query // This is the percolator query
      }
    });
    console.log(`🔔 Alert created: ${alert.emoji} ${alert.name}`);
  }
  
  // 🎯 Check which alerts match a document
  async checkAlerts(doc: any) {
    const response = await this.client.search({
      index: 'alerts',
      body: {
        query: {
          percolate: {
            field: 'query',
            document: doc
          }
        }
      }
    });
    
    const matchedAlerts = response.body.hits.hits.map(hit => hit._source);
    matchedAlerts.forEach(alert => {
      console.log(`🚨 Alert triggered: ${alert.emoji} ${alert.name}`);
      // Send notification to user
    });
    
    return matchedAlerts;
  }
}

// Example: Price drop alerts
const alertSystem = new AlertSystem(client);

// User wants alert when gaming laptops under $1000 appear
await alertSystem.createAlert({
  id: 'alert-1',
  name: 'Gaming Laptop Deal',
  userId: 'user123',
  emoji: '💸',
  query: {
    bool: {
      must: [
        { match: { category: 'laptops' } },
        { match: { tags: 'gaming' } }
      ],
      filter: [
        { range: { price: { lte: 1000 } } }
      ]
    }
  }
});

⚠️ Common Pitfalls and Solutions

😱 Pitfall 1: Mapping Conflicts

// ❌ Wrong way - changing field types causes conflicts!
await client.index({
  index: 'products',
  body: {
    price: "29.99" // String instead of number! 💥
  }
});

// ✅ Correct way - consistent types!
interface StrictProduct {
  price: number; // Always a number
}

const indexProduct = async (product: StrictProduct) => {
  // TypeScript ensures price is always a number
  await client.index({
    index: 'products',
    body: product
  });
};

🤯 Pitfall 2: Deep Pagination Performance

// ❌ Dangerous - deep pagination kills performance!
const getAllResults = async () => {
  const results = [];
  let from = 0;
  
  while (true) {
    const response = await client.search({
      index: 'products',
      from: from, // Gets slower as from increases! 💥
      size: 100
    });
    
    results.push(...response.body.hits.hits);
    if (results.length >= response.body.hits.total.value) break;
    from += 100;
  }
  return results;
};

// ✅ Safe - use search_after for efficient pagination!
const getAllResultsEfficiently = async () => {
  const results = [];
  let searchAfter: any[] | undefined;
  
  while (true) {
    const response = await client.search({
      index: 'products',
      body: {
        size: 100,
        sort: [{ _id: 'asc' }],
        ...(searchAfter && { search_after: searchAfter })
      }
    });
    
    const hits = response.body.hits.hits;
    if (hits.length === 0) break;
    
    results.push(...hits);
    searchAfter = hits[hits.length - 1].sort;
  }
  
  return results;
};

🛠️ Best Practices

1. 🎯 Use TypeScript Interfaces: Define interfaces for all document types
2. 📝 Version Your Mappings: Use index aliases for zero-downtime updates
3. 🛡️ Handle Errors Gracefully: Elasticsearch operations can fail
4. 🎨 Design for Scale: Use time-based indices for logs
5. ✨ Optimize Queries: Use filters instead of queries when possible

🧪 Hands-On Exercise

🎯 Challenge: Build a Movie Search Engine

Create a type-safe movie search system:

📋 Requirements:

• ✅ Index movies with title, genre, year, rating, and actors
• 🔍 Full-text search across title and actor names
• 🎭 Filter by genre and year range
• ⭐ Sort by rating or relevance
• 📊 Show genre distribution and average ratings
• 🎨 Each movie needs an emoji based on genre!

🚀 Bonus Points:

• Add “More Like This” recommendations
• Implement autocomplete suggestions
• Create saved searches for users

💡 Solution

// 🎬 Our type-safe movie search system!
interface Movie {
  id: string;
  title: string;
  genre: string[];
  year: number;
  rating: number;
  actors: string[];
  director: string;
  plot: string;
  emoji: string;
}

class MovieSearchEngine {
  private client: Client;
  private genreEmojis = {
    action: '💥',
    comedy: '😂',
    drama: '🎭',
    horror: '👻',
    romance: '💕',
    scifi: '🚀',
    thriller: '🔪',
    animation: '🎨'
  };
  
  constructor(client: Client) {
    this.client = client;
  }
  
  // 🏗️ Create index with proper mappings
  async createIndex() {
    await this.client.indices.create({
      index: 'movies',
      body: {
        mappings: {
          properties: {
            title: { 
              type: 'text',
              fields: {
                suggest: { 
                  type: 'search_as_you_type' 
                }
              }
            },
            genre: { type: 'keyword' },
            year: { type: 'integer' },
            rating: { type: 'float' },
            actors: { type: 'text' },
            director: { type: 'text' },
            plot: { type: 'text' },
            emoji: { type: 'keyword' }
          }
        }
      }
    });
    console.log('🎬 Movie index created!');
  }
  
  // ➕ Index a movie
  async indexMovie(movie: Omit<Movie, 'emoji'>) {
    const emoji = this.genreEmojis[movie.genre[0]] || '🎬';
    const fullMovie: Movie = { ...movie, emoji };
    
    await this.client.index({
      index: 'movies',
      id: movie.id,
      body: fullMovie
    });
    console.log(`✅ Indexed: ${emoji} ${movie.title}`);
  }
  
  // 🔍 Search movies with filters
  async searchMovies(params: {
    query?: string;
    genres?: string[];
    yearFrom?: number;
    yearTo?: number;
    minRating?: number;
    sortBy?: 'rating' | 'year' | 'relevance';
  }) {
    const must: any[] = [];
    const filter: any[] = [];
    
    if (params.query) {
      must.push({
        multi_match: {
          query: params.query,
          fields: ['title^3', 'actors^2', 'director', 'plot'],
          type: 'best_fields',
          fuzziness: 'AUTO'
        }
      });
    }
    
    if (params.genres && params.genres.length > 0) {
      filter.push({ terms: { genre: params.genres } });
    }
    
    if (params.yearFrom || params.yearTo) {
      filter.push({
        range: {
          year: {
            ...(params.yearFrom && { gte: params.yearFrom }),
            ...(params.yearTo && { lte: params.yearTo })
          }
        }
      });
    }
    
    if (params.minRating) {
      filter.push({ range: { rating: { gte: params.minRating } } });
    }
    
    const sort: any[] = [];
    if (params.sortBy === 'rating') {
      sort.push({ rating: 'desc' });
    } else if (params.sortBy === 'year') {
      sort.push({ year: 'desc' });
    }
    sort.push('_score'); // Always include relevance
    
    const response = await this.client.search<Movie>({
      index: 'movies',
      body: {
        query: {
          bool: {
            must: must.length > 0 ? must : [{ match_all: {} }],
            filter
          }
        },
        sort
      }
    });
    
    return response.body.hits.hits.map(hit => hit._source!);
  }
  
  // 🎯 Autocomplete suggestions
  async getSuggestions(prefix: string) {
    const response = await this.client.search({
      index: 'movies',
      body: {
        size: 5,
        query: {
          multi_match: {
            query: prefix,
            type: 'bool_prefix',
            fields: ['title.suggest', 'title.suggest._2gram', 'title.suggest._3gram']
          }
        },
        _source: ['title', 'year', 'emoji']
      }
    });
    
    return response.body.hits.hits.map(hit => hit._source);
  }
  
  // 📊 Get statistics
  async getStats() {
    const response = await this.client.search({
      index: 'movies',
      body: {
        size: 0,
        aggs: {
          genre_distribution: {
            terms: { field: 'genre', size: 20 }
          },
          rating_stats: {
            stats: { field: 'rating' }
          },
          movies_by_decade: {
            histogram: {
              field: 'year',
              interval: 10
            }
          }
        }
      }
    });
    
    return response.body.aggregations;
  }
  
  // 🎬 More like this
  async getRecommendations(movieId: string) {
    const response = await this.client.search<Movie>({
      index: 'movies',
      body: {
        query: {
          more_like_this: {
            fields: ['genre', 'actors', 'director'],
            like: [{
              _index: 'movies',
              _id: movieId
            }],
            min_term_freq: 1,
            min_doc_freq: 1
          }
        }
      }
    });
    
    return response.body.hits.hits.map(hit => hit._source!);
  }
}

// 🎮 Test it out!
const movieEngine = new MovieSearchEngine(client);

// Search for sci-fi movies
const scifiMovies = await movieEngine.searchMovies({
  query: 'space',
  genres: ['scifi'],
  minRating: 7,
  sortBy: 'rating'
});

console.log('🚀 Top sci-fi movies:');
scifiMovies.forEach(movie => {
  console.log(`  ${movie.emoji} ${movie.title} (${movie.year}) - ⭐${movie.rating}`);
});

🎓 Key Takeaways

You’ve learned so much! Here’s what you can now do:

• ✅ Build powerful search engines with Elasticsearch and TypeScript 💪
• ✅ Create type-safe queries that catch errors at compile time 🛡️
• ✅ Implement advanced features like faceted search and analytics 🎯
• ✅ Handle real-time data with efficient indexing strategies 🐛
• ✅ Scale your search to millions of documents! 🚀

Remember: Elasticsearch + TypeScript = Search superpowers! The combination gives you speed, relevance, and type safety. 🤝

🤝 Next Steps

Congratulations! 🎉 You’ve mastered Elasticsearch integration with TypeScript!

Here’s what to do next:

1. 💻 Build the movie search engine from the exercise
2. 🏗️ Add Elasticsearch to an existing project
3. 📚 Explore advanced features like machine learning
4. 🌟 Learn about Elasticsearch cluster management

Remember: Every search giant started with a single query. Keep building, keep searching, and most importantly, have fun! 🚀

Happy searching! 🎉🔍✨