Retrieval Plan
Overview
This document outlines the strategy for implementing text retrieval in the knowledge base application. The goal is to enable grounded question answering over imported documents without requiring an external LLM API.
Chunking Approach
Documents are split into chunks using a paragraph-aware algorithm:
- Split on double newlines (paragraph boundaries)
- Merge short paragraphs until chunk reaches ~500 characters
- Each chunk gets a unique ID, document reference, and metadata
Keyword Matching
The retrieval system uses keyword-based matching:
- Tokenize the question into individual words
- Filter out stop words (words shorter than 3 characters)
- For each chunk, count how many question keywords appear in the content
- Rank chunks by keyword overlap score
- Return top 2 most relevant chunks as citations