Week 13: Exercises & Checklist

Graded exercises

Easy: Run notebooks/01-loop-log-and-eval-gates.ipynb end-to-end. It prints a verifier pass rate at the end (a number from 0 to 1). Record that number in the Week 13 sheet of the Excel tracker. If any unit check fails, fix the classifier and re-run, the goal is a passing verifier, not a screenshot of a red cell.
Standard: Add one more unit check (for the billing category) and one more edge-case eval row (e.g. a ticket with two shipment ids). Re-run the verifier and record the before/after pass rate in the loop log. Write one sentence explaining which layer, the binary unit check or the statistical eval, caught more defects, and why.
Stretch: Run a real headless agent loop. If claude is installed, use claude -p '…' against the notebook as shown in the agentic-loop cell; otherwise use your harness's equivalent headless mode. Capture the output, then log the actual tokens and any defect it produced. Note the difference between the seeded example rows and real data.
Portfolio: Commit the Support Bot MVP, the loop log (all three loops), and the spec-v1-vs-v2 diff to your fork. Add a short retrospective: what you believed at the start, what the developer and external loops corrected, and how you knew. This is the artifact that proves you can run a measured loop, not just steer one harness.

Easy: The pass rate is the number the final cell prints; if any unit check is red, fix the keyword list first, the goal is a passing verifier, not a screenshot of a red cell.
Standard: The billing category keywords are in CATEGORY_KEYWORDS; write an assertion whose expected category matches a ticket that contains a billing keyword, and note which layer (unit vs eval) actually catches your new edge case.
Stretch: claude -p (or your harness's headless mode) is the same agent, scripted; capture stdout, then compare the real token count and defects against the seeded example rows in §4.
Portfolio: The retrospective needs three parts: the belief you started with, what each loop corrected, and the evidence (the pass-rate before/after) that told you it changed.

[ ] Mon: Study the three loops and verifier patterns (knowledge-base 01).
[ ] Tue: Spec the support bot MVP; define the verifier (eval suite + unit tests).
[ ] Wed: Agentic loop session: build → test → fix until the verifier passes; log the loop.
[ ] Thu: Developer loop: review with fresh eyes, update the spec, add missed edge cases.
[ ] Fri: Use case: external loop, have someone real use the bot; file issues; update spec + evals.
[ ] Sat: Take the Week 13 quiz (quiz.md), pass with 8/10; record the score in Notes.
[ ] Milestone: Update the Excel tracker; commit the MVP and loop log.