---
name: rag-pipeline-checkup
description: "Verify a RAG pipeline end-to-end, chunking, embeddings, retrieval quality, reranking, grounded answers with citations. Use when building or debugging retrieval-augmented generation."
version: 1.0.0
author: Zorost Intelligence
license: MIT
metadata:
  zorost:
    tags: [rag, retrieval, embeddings, citations, build]
    related_skills: [eval-first-development, error-analysis-50]
    program_weeks: [7]
---

# RAG Pipeline Checkup

## 1 · Purpose

Debug a RAG system as five separable stages, so "the answer was wrong" becomes
"retrieval missed the policy section" or "the model ignored the citation", each with
its own fix.

## 2 · When to use

- Building any RAG feature, or when a RAG feature's answers are wrong, ungrounded,
  or stale.
- Before adding complexity (HyDE, hybrid search, graphs) to a pipeline whose basics
  are unmeasured.

## 3 · Inputs

- The document corpus and the chunking code/config.
- A question set: ≥ 20 real questions with the passage that *should* answer each
  (the retrieval golden set, build it with `eval-first-development`).
- The ability to log intermediate stage outputs.

## 4 · Procedure

1. **Chunking.** Print 10 random chunks. Check: each chunk is one topic, carries its
   source and section, and respects the size budget (~300 to 500 tokens with 10 to 15%
   overlap is the default start). Fix splits that cut tables or definitions in half.
2. **Embedding.** Embed three paraphrases of one question. Confirm they retrieve
   each other's chunks, if paraphrases diverge wildly, the embedding model does not
   match the domain. Try a domain-appropriate model before tuning anything else.
3. **Retrieval.** Score the golden set: for each question, is the right passage in
   the top-k? Record recall@5. Below 0.8, fix retrieval before touching generation,
   a model cannot cite what it never saw.
4. **Reranking.** Add a reranker over the top-20 and re-measure recall@5 of the
   reranked top-5. Keep it only if recall rises; record the latency cost either way.
5. **Generation.** Require citations: every factual sentence must reference a
   retrieved chunk ID, and the answer must say "not in the documents" when retrieval
   came back empty. A schema constrains shape, never truth, spot-check that cited
   chunks actually contain the claim.
6. **End-to-end eval.** Score the full pipeline on the question set with the answer
   grader. Record: retrieval recall, answer score, citation precision (sampled).
7. Attribute every failure to exactly one stage. Fix the stage with the most
   failures first. Re-run the full eval after each fix.

## 5 · Anti-rationalization

| Excuse | Answer |
|---|---|
| "The model is bad; let's upgrade the LLM." | Most RAG failures are retrieval failures wearing a generation costume. Measure recall first. |
| "Chunk size doesn't matter much." | It is the highest-leverage knob you own. Measure at two sizes before dismissing it. |
| "Citations slow the model down." | Ungrounded answers cost more than tokens, they cost trust. |
| "It worked in the demo." | Demos use the questions the builder remembers. The golden set uses the ones users ask. |

## 6 · Red flags

- No recall number exists for retrieval alone.
- Chunks lack source metadata, so citations are impossible by construction.
- The answer never says "not in the documents", the model always finds *something*.
- A fix was applied to generation while retrieval recall sat below 0.5.

## 7 · Verify

- recall@5 on the retrieval golden set is recorded, before and after each change.
- The end-to-end answer score is recorded against the same question set.
- A sample of 10 answers has citation precision checked by hand (cited chunk
  actually supports the claim).
- Every known failure is attributed to one stage in the log.

## 8 · ZoroLogistics example

Week 7's policy bot: 20 ops questions ("who approves a refund over $500?") with the
handbook section that answers each. Baseline: recall@5 = 0.65, the chunker split
the approval table across two chunks. Fix: table-aware chunking → recall 0.90,
answer score 0.61 → 0.88 without touching the model. The lesson the pipeline teaches:
retrieval was the ceiling all along.

---
© 2026 Zorost Intelligence LLC · zorost.com