Week 4: Document Chunking and Hybrid Search

Overview

Week 4 implements a production-grade hybrid search system that combines the precision of BM25 keyword search with the semantic understanding of vector embeddings. This system provides the foundation for retrieval-augmented generation (RAG) by intelligently breaking documents into searchable chunks and enabling multiple search modes.

What We Built

🧩 Section-Based Document Chunking

Intelligent Segmentation: Leverages document structure (parsed sections) for natural chunk boundaries
Context Preservation: 100-word overlaps between chunks maintain semantic continuity
Adaptive Processing: Handles both structured (with sections) and unstructured (paragraph-based) documents
Optimal Sizing: Targets 600-word chunks with minimum 100-word threshold

🔍 Unified Hybrid Search System

Single Index Architecture: One OpenSearch index (arxiv-papers-chunks) supports all search modes
Multiple Search Types:
- BM25 Keyword Search: Fast (~50ms) traditional text matching
- Vector Similarity Search: Semantic search using 1024-dimensional embeddings
- Hybrid Search: RRF (Reciprocal Rank Fusion) combining both approaches
Production API: RESTful endpoint /api/v1/hybrid-search/ with comprehensive validation

🤖 Real Embedding Integration

Jina AI Embeddings: Production-grade 1024-dimensional vectors optimized for retrieval
Automatic Generation: FastAPI endpoints automatically generate query embeddings
Fallback Strategy: Graceful degradation to BM25 when embeddings unavailable
Performance Optimized: Efficient embedding generation and storage

Architecture

System Overview

Complete Week 4 architecture showing hybrid search with chunking, embeddings, and RRF fusion

Data Flow

Raw Papers → PDF Parsing → Section Extraction → Chunking → Embedding → Indexing → Search

The diagram above illustrates the complete Week 4 implementation:

Data Processing Pipeline: arXiv papers flow through chunking and embedding generation
Unified OpenSearch Index: Single index supporting BM25, vector, and hybrid search modes
Hybrid Retrieval Pipeline: RRF fusion combining keyword precision with semantic understanding
Production API Layer: FastAPI endpoints with automatic embedding generation

System Components

1. Document Processing Pipeline

python

# Located in: src/services/indexing/text_chunker.py
TextChunker.chunk_paper(
    title="Paper Title",
    abstract="Abstract text",
    full_text="Complete paper content",
    sections=parsed_sections_dict,
    target_words=600,
    overlap_words=100
)

2. Embedding Service

python

# Located in: src/services/embeddings/factory.py
embeddings_service = make_embeddings_service()
vectors = await embeddings_service.embed_query(["query text"])

3. Unified Search Client

python

# Located in: src/services/opensearch/client.py
results = opensearch_client.search_unified(
    query="machine learning",
    query_embedding=vector,
    use_hybrid=True,
    size=10
)

4. Production API

python

# Located in: src/routers/hybrid_search.py
POST /api/v1/hybrid-search/
{
  "query": "neural networks",
  "use_hybrid": true,
  "size": 5
}

Key Features

Hybrid Search Modes

Mode	Speed	Recall	Precision	Use Case
BM25 Only	~50ms	High	Medium	Exact keyword matching
Vector Only	~100ms	Medium	High	Semantic similarity
Hybrid (RRF)	~2-4s	High	High	Best overall relevance

RRF (Reciprocal Rank Fusion)

Algorithm: Combines BM25 and vector search rankings using reciprocal rank fusion
Implementation: Manual fusion algorithm (OpenSearch 2.19 compatibility)
Weighting: Configurable balance between keyword and semantic relevance
Fallback: Automatic fallback to BM25 if vector search fails

Section-Based Chunking Strategy

python

# Chunking Parameters (optimized through testing)
CHUNK_SIZE = 600        # Target words per chunk
OVERLAP_SIZE = 100      # Words overlapping between chunks  
MIN_CHUNK_SIZE = 100    # Minimum viable chunk size
SECTION_BASED = True    # Use document structure when available

Benefits:

Semantic Coherence: Chunks respect natural document boundaries
Context Preservation: Overlaps prevent information loss at boundaries
Retrieval Accuracy: Better matching of user queries to relevant content
Scalability: Handles documents from 1,000 to 100,000+ words

Implementation Details

OpenSearch Index Configuration

Index Name: arxiv-papers-chunks

Key Fields:

json

{
  "arxiv_id": "2508.18563v1",
  "title": "Paper title",
  "chunk_text": "Chunk content...",
  "chunk_id": "unique_chunk_identifier", 
  "section_name": "Introduction",
  "embedding": [0.123, 0.456, ...],  // 1024 dimensions
  "paper_categories": ["cs.AI", "cs.LG"],
  "published_date": "2025-08-25T23:43:33"
}

Search Query Structure

BM25 Query (keyword matching):

json

{
  "query": {
    "bool": {
      "should": [
        {"match": {"chunk_text": {"query": "machine learning", "fuzziness": "AUTO"}}},
        {"match": {"title": {"query": "machine learning", "boost": 2.0}}},
        {"match": {"abstract": {"query": "machine learning", "boost": 1.5}}}
      ]
    }
  }
}

Hybrid Query (RRF manual fusion):

Execute BM25 query → get ranked results
Execute vector query → get ranked results
Apply RRF fusion algorithm → combine rankings
Return merged results with hybrid scores

Environment Configuration

Required Variables:

bash

# Core Services
POSTGRES_DATABASE_URL=postgresql+psycopg2://rag_user:rag_password@postgres:5432/rag_db
OPENSEARCH__HOST=http://opensearch:9200

# Embeddings (Required for Hybrid Search)
JINA_API_KEY=jina_your_api_key_here

# Chunking Configuration
CHUNKING__CHUNK_SIZE=600
CHUNKING__OVERLAP_SIZE=100
CHUNKING__MIN_CHUNK_SIZE=100
CHUNKING__SECTION_BASED=true

# OpenSearch Configuration
OPENSEARCH__INDEX_NAME=arxiv-papers
OPENSEARCH__CHUNK_INDEX_SUFFIX=chunks
OPENSEARCH__VECTOR_DIMENSION=1024

API Reference

Hybrid Search Endpoint

Endpoint: POST /api/v1/hybrid-search/

Request Body:

json

{
  "query": "transformer neural networks",
  "use_hybrid": true,
  "size": 10,
  "from": 0,
  "categories": ["cs.AI", "cs.LG"],
  "latest_papers": false,
  "min_score": 0.0
}

Response:

json

{
  "query": "transformer neural networks",
  "total": 15,
  "hits": [
    {
      "arxiv_id": "2508.18563v1",
      "title": "Paper Title",
      "authors": "Author Names",
      "abstract": "Paper abstract...",
      "score": 0.8542,
      "chunk_text": "Relevant chunk content...",
      "chunk_id": "chunk_uuid",
      "section_name": "Related Work"
    }
  ],
  "size": 10,
  "from": 0,
  "search_mode": "hybrid"
}

Performance Benchmarks

Test Environment: 3 papers, 81 chunks, single-node OpenSearch

Search Type	Avg Response Time	Throughput	Recall@10	Precision@10
BM25 Only	52ms	~200 req/s	0.78	0.65
Vector Only	105ms	~95 req/s	0.82	0.71
Hybrid (RRF)	2.4s	~25 req/s	0.89	0.84

Key Insights:

Hybrid search provides best relevance at cost of response time
BM25 excellent for high-throughput keyword matching
Vector search good semantic understanding with moderate speed
Embedding generation accounts for ~2s of hybrid search time

Production Deployment

Scaling Considerations

OpenSearch Cluster:

yaml

# Recommended minimum for production
opensearch:
  image: opensearchproject/opensearch:2.19.0
  environment:
    - cluster.name=rag-cluster
    - node.name=rag-node-1
    - discovery.type=single-node
    - OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g
  deploy:
    resources:
      limits:
        memory: 2G
      reservations:
        memory: 1G

Embedding Service Optimization:

Batch Processing: Process multiple embeddings per API call
Caching: Cache frequent query embeddings
Rate Limiting: Respect Jina AI API limits (1000 requests/minute)
Fallback Strategy: BM25-only mode when embeddings unavailable

Monitoring and Observability

Key Metrics:

Search request latency (p50, p95, p99)
Embedding generation success rate
Index document count and size
Search mode usage distribution (BM25 vs Hybrid)

Health Checks:

OpenSearch cluster health
Embedding service availability
Index document count validation
Sample query execution

Troubleshooting

Common Issues

1. Hybrid Search Returns BM25 Mode

bash

# Check embedding service
curl -X POST "http://localhost:8000/api/v1/hybrid-search/" \
  -H "Content-Type: application/json" \
  -d '{"query": "test", "use_hybrid": true}'

# Check logs for embedding errors
docker compose logs api | grep -i embedding

2. Empty Search Results

bash

# Verify index exists and has documents
curl "http://localhost:9200/arxiv-papers-chunks/_count"

# Check index mapping
curl "http://localhost:9200/arxiv-papers-chunks/_mapping"

3. Slow Embedding Generation

bash

# Check Jina API key configuration
docker compose exec api env | grep JINA

# Test direct embedding service
curl -X POST "https://api.jina.ai/v1/embeddings" \
  -H "Authorization: Bearer $JINA_API_KEY" \
  -d '{"model": "jina-embeddings-v3", "input": ["test"]}'

Testing

Running the Week 4 Notebook

Start Services:

bash

docker compose up --build -d

Open Notebook:

bash

cd notebooks/week4
jupyter notebook week4_hybrid_search.ipynb

Execute All Cells: The notebook includes:
- Environment setup and health checks
- Section-based chunking demonstration
- Real embedding generation with Jina AI
- All search modes (BM25, Vector, Hybrid)
- Production API endpoint testing
- Performance comparison

Manual Testing

Test BM25 Search:

bash

curl -X POST "http://localhost:8000/api/v1/hybrid-search/" \
  -H "Content-Type: application/json" \
  -d '{"query": "machine learning", "use_hybrid": false, "size": 3}'

Test Hybrid Search:

bash

curl -X POST "http://localhost:8000/api/v1/hybrid-search/" \
  -H "Content-Type: application/json" \
  -d '{"query": "neural networks", "use_hybrid": true, "size": 3}'

Next Steps (Week 5)

Week 4 provides the search foundation for Week 5's LLM integration:

LLM Integration: Connect Ollama for answer generation
RAG Pipeline: Query → Search → Context → Generate → Response
Context Management: Optimize retrieved chunks for LLM input
Answer Quality: Implement citation and source attribution
Conversation Memory: Support multi-turn conversations

The hybrid search system is production-ready and provides the retrieval accuracy needed for high-quality RAG applications.

File Structure

src/
├── routers/
│   └── hybrid_search.py          # FastAPI endpoints
├── services/
│   ├── opensearch/
│   │   ├── client.py              # Unified search client
│   │   ├── factory.py             # Client factory
│   │   └── index_config_hybrid.py # Index configuration
│   ├── indexing/
│   │   ├── text_chunker.py        # Section-based chunking
│   │   ├── hybrid_indexer.py      # Document indexing
│   │   └── factory.py             # Indexing service factory
│   └── embeddings/
│       ├── jina_client.py         # Jina AI client
│       └── factory.py             # Embedding service factory
├── schemas/
│   └── api/
│       └── search.py              # Request/response models
└── config.py                      # Configuration management

notebooks/week4/
├── README.md                      # This document
├── week4_hybrid_search.ipynb      # Interactive tutorial
└── data/                          # Sample data directory

Resources

OpenSearch Documentation: https://opensearch.org/docs/
Jina AI Embeddings: https://jina.ai/embeddings/
FastAPI Documentation: https://fastapi.tiangolo.com/
Reciprocal Rank Fusion Paper: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf
Week 4 Notebook: week4_hybrid_search.ipynb

Week 4: Document Chunking and Hybrid Search ​

Overview ​

What We Built ​

🧩 Section-Based Document Chunking ​

🔍 Unified Hybrid Search System ​

🤖 Real Embedding Integration ​

Architecture ​

System Overview ​

Data Flow ​

System Components ​

1. Document Processing Pipeline ​

2. Embedding Service ​

3. Unified Search Client ​

4. Production API ​

Key Features ​

Hybrid Search Modes ​

RRF (Reciprocal Rank Fusion) ​

Section-Based Chunking Strategy ​

Implementation Details ​

OpenSearch Index Configuration ​

Search Query Structure ​

Environment Configuration ​

API Reference ​

Hybrid Search Endpoint ​

Performance Benchmarks ​

Production Deployment ​

Scaling Considerations ​

Monitoring and Observability ​

Troubleshooting ​

Common Issues ​

Testing ​

Running the Week 4 Notebook ​

Manual Testing ​

Next Steps (Week 5) ​

File Structure ​

Resources ​

Week 4: Document Chunking and Hybrid Search

Overview

What We Built

🧩 Section-Based Document Chunking

🔍 Unified Hybrid Search System

🤖 Real Embedding Integration

Architecture

System Overview

Data Flow

System Components

1. Document Processing Pipeline

2. Embedding Service

3. Unified Search Client

4. Production API

Key Features

Hybrid Search Modes

RRF (Reciprocal Rank Fusion)

Section-Based Chunking Strategy

Implementation Details

OpenSearch Index Configuration

Search Query Structure

Environment Configuration

API Reference

Hybrid Search Endpoint

Performance Benchmarks

Production Deployment

Scaling Considerations

Monitoring and Observability

Troubleshooting

Common Issues

Testing

Running the Week 4 Notebook

Manual Testing

Next Steps (Week 5)

File Structure

Resources