Week 11: Exercises & Checklist

Graded exercises

1. Easy: Run notebooks/01-zoroeval-harness.ipynb end-to-end. Record the judge-agreement rate (on the doubled sample) and the calibration agreement (judge vs your human labels) it prints in the Week 11 sheet of the Excel tracker. Without an API key, run the code-metric cells (extraction + triage accuracy) and record those, noting the judge cells were skipped.

2. Standard: Add a fourth golden set to the ZoroEval class. Pick a new task (shipment-note summarization or ticket priority classification), write a 10-item golden set with a metric that fits the task (judge-with-rubric or exact-match), and run it through the same judge/agreement pipeline. Document the metric you chose and why.

3. Stretch: Turn the CI gate into a real script zoroeval/gate.py (or scripts/zoroeval_gate.py): it reads a JSON file of eval scores, applies the thresholds, prints a pass/fail table, and exits with a non-zero code when any metric falls below threshold. Demonstrate it (a) passing a good artifact and (b) blocking a deliberately broken one.

4. Portfolio: Commit ZoroEval v1 as the eval harness + CI gate milestone (tracked in curriculum/projects/README.md): the harness class, the three golden sets, the calibrated judge and its agreement numbers, the error-analysis note (top failure cluster + the fix), and the gate script. This is the artifact every later week adds failure classes to, make it inspectable.

Hints

1. Easy: The code-metric cells (extraction + triage) run with no key; the judge cells skip when OPENAI_API_KEY is unset. Record whichever numbers actually ran, and note the skipped cells.

2. Standard: Pick a metric that fits the task: exact-match for priority labels, judge-with-rubric for summarization. Write 10 real items sourced from the domain, not copied from the existing three sets.

3. Stretch: ci_gate(score, threshold) already returns a boolean; the script just reads a JSON of scores, applies each metric's threshold, prints PASS/FAIL, and sys.exit(1) when any metric fails. Demonstrate both the passing and blocking case.

4. Portfolio: The gate on your harness is self-referential: a harness with no threshold, or a judge nobody calibrated, fails its own gate. Ship the class, the three sets, the agreement numbers, and the top-fix note together.

Checklist (mirrors manifest.json + Excel tracker)

• [ ] Mon: Study evals and error analysis (knowledge-base 07 + Ng's letters).
• [ ] Tue: Define golden eval sets for extraction, RAG, and triage; implement LLM-as-judge with a rubric.
• [ ] Wed: Run the judges; measure judge agreement; calibrate the rubric.
• [ ] Thu: Error analysis: cluster failures from Weeks 6 to 10 artifacts; pick the top fix.
• [ ] Fri: Use case: ZoroEval v1 gates one artifact in a CI-style script; document the loop.
• [ ] Sat: Take the Week 11 quiz (quiz.md), pass with 8/10; record the score in Notes.
• [ ] Milestone: Update the Excel tracker; commit ZoroEval.