Week 13: Quiz (10 questions, 8/10 to pass)

Answer from memory first, then check the key. Each question names the section or cell it tests, so a miss tells you exactly what to re-read.

Questions

1. (MCQ) Which loop runs on a minutes-scale cadence and closes against the verifier? (see Concepts, "The three loops")

   • A) External feedback
   • B) Developer feedback
   • C) Agentic coding
   • D) The configuration-management loop

2. (MCQ) The unit checks and the mini eval are two different verifier layers because… (see Concepts, "The verifier has layers")

   • A) unit tests are statistical and evals are binary
   • B) the eval measures accuracy as a rate over many examples and catches whole classes the hand-written checks missed
   • C) evals are cheaper to run than unit tests
   • D) unit tests replace the need for an eval

3. (Short answer) Write the exact formula the notebook's final cell uses to compute the verifier pass rate. (see notebook cell §7)

4. (MCQ) In spec-driven development, when feedback finds a gap, the fix goes… (see Concepts, "SPEC.md patterns")

   • A) into the code first, then the spec if there's time
   • B) into the spec first, then the code, then the test/eval case
   • C) into a chat message to the agent, then lost
   • D) only into the eval set

5. (Short answer) Name all six categories the Support Bot classifier distinguishes. (see notebook cell §1)

6. (MCQ) A blast-radius rule calibrates autonomy per action. Which action needs the tightest leash? (see Concepts, "Blast-radius rules")

   • A) reading a scratch file
   • B) editing the repo on a branch
   • C) writing production data
   • D) running the unit tests

7. (Short answer) What does classify_ticket('hello') return, and why? (see notebook cell §1 and the _check_unknown unit check)

8. (MCQ) A PreToolUse hook exists to… (see Concepts, "Harness primitives")

   • A) summarize the diff after editing
   • B) enforce or block a policy before a tool runs (e.g. refuse data/raw/ writes)
   • C) load a skill on demand
   • D) spawn a subagent

9. (Short answer, with numbers) Three agentic cycles total 12,000 input tokens and 3,000 output tokens at $3.00/1M input and $15.00/1M output. Show the arithmetic and give the total cost in dollars. (see Concepts, "Worked example 1")

10. (MCQ) The right way to convert an external-loop bug report into lasting change is… (see Concepts, "Worked example 2")

    • A) fix the code and move on
    • B) file it, then add a spec line and a new eval case so it cannot silently regress
    • C) ask the user to stop using that input
    • D) log it in the loop log and do nothing else

Answer key

1. C. Agentic coding runs in minutes: write → test → read the failure → fix, until the verifier passes. Developer feedback is tens of minutes to hours; external feedback is hours to weeks.

2. B. The 6 unit checks are binary and hand-written; the 60-ticket eval reports accuracy as a rate and can surface a whole class (e.g. customs) the hand-written checks never named.

3. verifier_pass_rate = (unit_passed + eval_correct) / (unit_total + eval_total), printed as round(verifier_pass_rate, 3), i.e. (unit checks passed + eval correct) / (unit checks + eval cases).

4. B. Spec first, then code, then the test/eval case, that ordering keeps the loops from drifting; the spec is the source of truth, the code its current implementation.

5. tracking, damage, refund, documents, customs, billing (plus a fallback 'unknown' when no keyword matches).

6. C. Writing production data causes real customer harm and is hard to undo, it needs manual approval. Reading a scratch file is nearly free to get wrong.

7. 'unknown'. 'hello' matches no keyword, so every category scores 0; classify_ticket returns the best category only when its score is > 0, otherwise 'unknown'.

8. B. A PreToolUse hook runs before a tool call and can block it, turning "please remember not to touch X" into something the harness enforces.

9. $0.081. Input: 12,000 × $3.00 / 1,000,000 = $0.036. Output: 3,000 × $15.00 / 1,000,000 = $0.045. Total = $0.036 + $0.045 = $0.081.

10. B. File it, then decide spec-vs-eval and add the failing input to the eval set, that is the triage rule that makes feedback permanent rather than a one-off fix.