Week 07: Quiz (10 questions, 8/10 to pass)

Answer from the concepts and the notebook code. Each question notes where to find it.

1. (MCQ) RAG exists primarily to provide, in production order: (a) speed, cost, scale; (b) grounding, freshness, permissions; (c) fewer tokens, longer context, lower temperature; (d) fine-tuning, quantization, caching. (see Concepts §"The seven-stage pipeline")

2. (Short answer) Name the seven stages of the RAG pipeline in order. (see Concepts diagram / KB §2)

3. (MCQ) In 01-rag-policy-bot.ipynb, section_aware_chunks splits on: (a) every 256 tokens, (b) ## headings, (c) sentence boundaries, (d) embedding-similarity drops. (see notebook cell 4)

4. (Short answer) Given a 10-question golden set, vector-only retrieval hits 7 questions and hybrid+rerank hits 9. Write both recall@5 values and explain what the delta means. (see Concepts Worked example 1)

5. (MCQ) RRF fuses ranked lists by: (a) averaging the raw scores, (b) summing 1/(k+rank) across lists, (c) taking the max score, (d) multiplying ranks. (see Concepts Worked example 2)

6. (Short answer) In 02-knowledge-graph-lab.ipynb, what does the SERVES edge's cost attribute equal, and how is the graph kept connected despite planted NaN distances? (see notebook cell 4)

7. (MCQ) The single highest-leverage retrieval upgrade for answer quality is: (a) a bigger embedder, (b) reranking with a cross-encoder, (c) a longer chunk, (d) more questions in the eval set. (see Concepts §"Hybrid search and reranking")

8. (Short answer) Why must retrieval (recall@k) and generation (groundedness) be measured separately? What does each failure point to? (see Concepts §"Retrieval evals")

9. (MCQ) A question best answered by a knowledge graph rather than vector RAG is: (a) "what is the late-delivery refund?", (b) "which carriers reach Long Beach, and at what cost?", (c) "define dangerous goods", (d) "summarize the customs policy". (see Concepts §"Knowledge graphs")

10. (Short answer) In notebook 1 cell 8, why does recall_at_k check whether any of the top-k chunk doc_ids matches a relevant doc_id, rather than requiring the exact chunk? (see notebook cells 7 to 8)

Answer key

1. (b): grounding (citable text), freshness (re-index not retrain), permissions (filter the index before retrieval).
2. ingest → chunk → embed → index → retrieve → rerank → generate (with citations).
3. (b): it splits on ## section headings so each chunk is one coherent policy section carrying its doc_id and section name as citation metadata.
4. vector-only recall@5 = 0.70; hybrid+rerank recall@5 = 0.90. The delta (2 recovered questions) shows the keyword arm + reranker fetched the right document for queries vector-only missed; recall says nothing about answer quality.
5. (b): 1/(60+rank_vec) + 1/(60+rank_kw); agreement across signals outranks one strong signal, and no score normalization is needed.
6. cost = carrier base_rate_usd_per_km_ton × lane distance_km; NaN distances (and transit days) are filled with defaults (900 km / 3 days) so every lane remains connected.
7. (b): a cross-encoder reranker re-scores query and passage together; it is cheap relative to generation and moves the best chunk to the top.
8. Low recall@k is an indexing/chunking problem (fix the data path); low groundedness with high recall is a prompt/generation problem (fix the instruction/citations). A single end-to-end score hides which layer is broken.
9. (b): it is a constrained traversal over an explicit SERVES/ARRIVES_AT relationship (shortest/cheapest path), which chunk retrieval only approximates. The others are lookup/definition questions vector RAG handles.
10. Recall@k is about did we surface the right document, not the exact chunk, a relevant document may be split across several chunks, and k would have to be huge to guarantee the exact passage. The doc_id check is the standard recall definition.