# Week 11: Quiz (10 questions, 8/10 to pass)

Answer on your own first, then check the answer key. Each question notes where the
answer lives, so you can re-read the exact section or cell you missed.

## Questions

1. **(MCQ)** Ng's claim about evals + error analysis is a claim about *what*, exactly? (see Concepts intro + the KB's "Why this file matters")
   - A) Being more ethical
   - B) Velocity, teams that can measure failure ship faster
   - C) Reducing GPU cost
   - D) Replacing code review

2. **(MCQ)** Why does the knowledge base insist golden sets be drawn from *real* inputs rather than invented ones? (see Concepts §"Golden sets")
   - A) Real inputs are easier to generate
   - B) Invented cases encode your beliefs; real cases encode the problem
   - C) Real inputs need no labels
   - D) Invented cases are always too short

3. **(MCQ)** Which metric family fits "generation with a correct-but-variable answer"? (see Concepts §task-metric table)
   - A) Accuracy
   - B) Recall@k
   - C) LLM-as-judge with a rubric
   - D) Schema conformance only

4. **(Short answer)** In `notebooks/01-zoroeval-harness.ipynb`, what does the groundedness rubric's level "3" catch, and why is the rubric anchored this way? (see notebook cell 12 + Concepts §"LLM-as-judge")

5. **(MCQ)** A judge returns the same score on 9 of 10 doubled samples, but on one grounded answer (human label 5) it returns 1. What do these two facts tell you? (see Concepts §Worked example 1)
   - A) The judge is perfect
   - B) Self-agreement is high but the judge still misses that dimension, recalibrate or fall back to human review
   - C) The rubric is too vague to use
   - D) Calibration is irrelevant

6. **(MCQ)** In the workshop notebook, what does HLP classify as "upstream" rather than "fixable"? (see notebook cell 10)
   - A) Any failure a human would also make, given the same information
   - B) Any failure on ticket triage
   - C) Failures that happen more than 5 times
   - D) Failures with a `trace_id`

7. **(Short answer)** In the seed-42 run log, clustering the 34 wrong traces gives `field_date_format` (8) and `hallucinated_rate` (8) tied at the top. What is the top fixable class's coverage, and why do we fix the *largest* class first? (see Concepts §Worked example 2)

8. **(MCQ)** In `notebooks/02-error-analysis-workshop.ipynb`, `ci_gate(score, threshold)` returns `False` when the extraction score is 0.88 and the threshold is 0.90. What is the point of a gate that can say "no"? (see notebook cell 15 + Concepts §"evals in CI")
   - A) It makes the pipeline run faster
   - B) A number nobody is willing to block on is a number nobody trusts
   - C) It guarantees the model is never deployed
   - D) It replaces the eval set

9. **(MCQ)** Which is a **code metric** (not a model metric)? (see Concepts §code vs model table)
   - A) "Is the answer grounded in the passage?"
   - B) "Was the ticket routed to the right queue?"
   - C) "Did the endpoint return valid JSON in under 2 seconds?"
   - D) "Does the summary capture every claim?"

10. **(Short answer)** List the five steps of the error-analysis loop, and state *why* the fifth step (add the failure class back to the eval set) is the one that makes the loop a ratchet. (see Concepts §"The error-analysis loop")

## Answer key

1. **B)** Ng's claim is about *velocity*: measuring failure lets a team ship faster, because every change either moves a number or it does not. It is not a soft claim about being "data-driven."

2. **B)** Invented cases encode your *beliefs* about the problem; real cases encode the problem itself. One real customer ticket can be worth ten invented ones.

3. **C)** When the answer can be phrased many ways, string matching fails, so you grade with an LLM-as-judge and a rubric (often paired with a similarity score).

4. **Level 3 catches "the core answer is supported, but it includes at least one claim (a number, date, or policy detail) not present in the passage."** The rubric is anchored, each level names a concrete failure, so a "3 vs 1" is defensible and worth investigating instead of shrugging off.

5. **B)** Self-agreement 0.90 means the rubric is stable, but stability is not correctness: a systematic miss (human 5 → judge 1) means the judge is unreliable on that dimension, so you recalibrate or fall back to human review. A judge you have not calibrated is a second unverified model.

6. **A)** Upstream = even a competent human, given the same information, would fail, so the fix is the data/intake, not the model (e.g., a schema contract both model and human need).

7. **Coverage ≈ 8 / 34 ≈ 24%** of the wrong traces. We fix the largest class first because **frequency matters more than cleverness**, the biggest class is the one change worth making this week, and it is the class you add back to the eval set.

8. **B)** The gate's authority comes from being willing to block: a number nobody is willing to block on is a number nobody will trust. Here the extraction score failing its bar blocks the release.

9. **C)** A code metric measures the deterministic plumbing (schema/parse/latency), runs fast and cheap on every commit. The other three measure the model's probabilistic output.

10. **Read traces → cluster failures → prioritize by frequency → fix → add the failure back to the eval set.** Step five is the ratchet: putting the failing cases (and the fix's behavior) into the eval set ensures that class can never silently regress, turning error analysis from a one-off into a permanent improvement loop.