AI 原生开放教材

外观

生产级 Agentic RAG 课程系统课程·英文原文 ↗仓库 ↗

Week 3: Keyword Search First - The Critical Foundation ​

🚨 The 90% Problem: Most RAG systems jump straight to vector search and miss the foundation that powers the best retrieval systems. We're doing it right!

This folder contains the materials for Week 3 of the arXiv Paper Curator project, where we implement the keyword search foundation that professional RAG systems rely on using OpenSearch and BM25 scoring.

🎯 Why Keyword Search First? ​

The Professional Path: Unlike tutorials that jump straight to vector embeddings, we build the foundation that successful companies use:

  1. 🔍 Exact Match Power: Keywords excel at finding specific technical terms, paper IDs, and precise phrases
  2. 📊 Interpretable Results: You can understand exactly why a paper was retrieved
  3. ⚡ Speed & Efficiency: BM25 is computationally fast and doesn't require expensive embedding models
  4. 📈 Production Reality: Companies like Elasticsearch, Algolia, and enterprise search use keyword search as their foundation

The Learning Path:

Week 3: Master BM25 keyword search    ← YOU ARE HERE
Week 4: Add intelligent chunking
Week 5: Introduce vector embeddings for hybrid retrieval  
Week 6: Optimize the complete system

🚀 Before You Start ​

Essential Environment Setup:

bash
# 1. Ensure you have the correct environment configuration
cp .env.example .env

# 2. Verify OpenSearch settings are properly configured
# Your .env should contain these critical Week 3 settings:
# OPENSEARCH__HOST=http://opensearch:9200
# OPENSEARCH__INDEX_NAME=arxiv-papers

Important: Week 3 requires the .env file to be properly configured for OpenSearch connectivity and indexing. The defaults in .env.example work out of the box.

Contents ​

week3_opensearch.ipynb ​

A comprehensive Jupyter notebook that guides you through building the keyword search foundation:

  1. Infrastructure Validation

    • Verify all Week 1-2 services are running correctly
    • OpenSearch cluster health and connectivity testing
    • Environment setup for Week 3 search components

  2. OpenSearch Service Integration

    • Building a production-grade search client with factory patterns
    • Implementing dependency injection with FastAPI lifespan management
    • Creating JSON-based index configurations with proper analyzers

🔍 OpenSearch Dashboard in Action ​

You'll add your OpenSearch dashboard screenshot here showing real paper searches with BM25 scoring and result rankings.

OpenSearch Dashboard Retrieving Papers
OpenSearch Dashboards showing keyword search results with BM25 relevance scoring

Search Architecture Overview:

  • OpenSearch Client: Factory-pattern service with health monitoring and CRUD operations
  • Query Builder: Advanced search query construction with field boosting and filtering
  • Index Management: JSON-based schema with English analyzers and strict mapping
  • Search API: RESTful endpoints (/search) with GET/POST methods for different query types
  • BM25 Algorithm: Industry-standard text relevance scoring with multi-field search

  1. Index Management & Schema Design

    • Creating indexes with proper field mappings for academic papers
    • Implementing English language analyzers for improved search relevance
    • Testing index health, statistics, and document management operations

  2. Document Indexing Pipeline

    • Migrating paper data from PostgreSQL to OpenSearch
    • Bulk indexing operations with error handling and validation
    • Real-time indexing verification and performance monitoring

  3. BM25 Search Implementation

    • Multi-field search across title, abstract, and content with field boosting
    • Advanced query features: highlighting, pagination, category filtering
    • Testing two-letter queries (AI, ML, NN, CV) as specifically requested
    • Fuzzy matching and typo tolerance configuration

  4. Search API Development

    • FastAPI integration with OpenSearch dependency injection
    • RESTful endpoints: GET /search for simple queries, POST /search for advanced search
    • Response formatting with pagination, metadata, and highlighted snippets
    • Error handling and service health monitoring endpoints

  5. Airflow Pipeline Updates

    • Sequential task execution: setup → fetch → opensearch → report → cleanup
    • Real OpenSearch indexing replacing placeholder operations
    • Updated daily reporting with search statistics and health metrics
    • Graceful handling of PDF processing limits (>20MB or >30 pages)

  6. End-to-End Pipeline Verification

    • Complete flow testing: arXiv API → PostgreSQL → OpenSearch → Search API
    • Performance benchmarking and response time measurement
    • Production readiness assessment with monitoring and alerting

Week 3 Architecture:

Week 3 OpenSearch Flow Architecture
Complete Week 3 architecture showing the OpenSearch integration flow

Key Features Implemented ​

🔍 Production-Grade Search System ​

  • BM25 Scoring: Industry-standard relevance ranking algorithm
  • Multi-field Search: Query across title (3x boost), abstract (2x boost), and content (1x boost)
  • Advanced Filtering: Category filtering, date ranges, and custom field queries
  • Real-time Highlighting: Search term highlighting in results with HTML markup
  • Pagination Support: Efficient large result set handling with size/from parameters

🏗️ Clean Architecture Implementation ​

  • Factory Pattern: Consistent service creation across all components
  • Dependency Injection: Proper service lifecycle management with FastAPI lifespan
  • Query Builder Pattern: Flexible search query construction for complex operations
  • Error Handling: Comprehensive exception management with graceful degradation

📊 Performance & Monitoring ​

  • Sub-100ms Search: Optimized query performance for real-time applications
  • Health Monitoring: Cluster status, index statistics, and service availability
  • Resource Limits: PDF processing constraints to prevent memory issues
  • Scalability: Design patterns supporting horizontal scaling

🔧 Development Best Practices ​

  • Type Safety: Full type hints throughout the codebase
  • Testing: End-to-end verification with realistic data scenarios
  • Documentation: Comprehensive docstrings and architectural explanations
  • Code Quality: Following established patterns from Week 1-2 implementations

Expected Outcomes ​

By completing Week 3, students will have:

  1. Working Search System: Fully functional OpenSearch BM25 search with 28+ indexed papers
  2. RESTful API: Complete search API with both simple and advanced query capabilities
  3. Production Architecture: Clean, scalable code following industry best practices
  4. Performance Insights: Understanding of search optimization and monitoring strategies
  5. End-to-End Pipeline: Complete RAG system foundation ready for Week 4 development

Success Criteria ​

  • ✅ OpenSearch Integration: Factory pattern service with health checks operational
  • ✅ BM25 Search: Relevance scoring with multi-field queries returning relevant results
  • ✅ Advanced Features: Filtering, highlighting, pagination, and fuzzy search working
  • ✅ API Endpoints: RESTful search with both GET and POST methods functional
  • ✅ Two-Letter Queries: AI, ML, NN, CV queries working as specifically requested
  • ✅ Pipeline Integration: Airflow DAGs updated for OpenSearch indexing
  • ✅ Performance: Sub-100ms search response times with proper error handling

What's Next? ​

Week 4 Preview: Chunking Strategies and Hybrid Retrieval

  • Intelligent document chunking strategies for better context preservation
  • Combining keyword search (BM25) with vector embeddings for hybrid retrieval
  • Building the complete foundation for modern RAG systems
  • Evaluation metrics to measure retrieval quality

Ready to build production-grade search? Let's get started with Week 3! 🔍✨