Week 17: Quiz (10 questions, 8/10 to pass)

Each question names its source section or notebook cell. Answer, then check the key.

1. (MCQ) OpenClaw's memory model is "the model only remembers what gets saved to disk." Which file holds stable preferences, and which holds durable facts/decisions? (see Concepts §"OpenClaw: a personal assistant runtime" and reference/agents/openclaw.md §2.6)

   • (a) MEMORY.md for preferences; USER.md for facts.
   • (b) USER.md for preferences; MEMORY.md for facts.
   • (c) DREAMS.md for both.
   • (d) There is no file-based memory.

2. (MCQ) The OpenClaw permission tier, ordered least → most permissive, is: (see Concepts §"OpenClaw: a personal assistant runtime")

   • (a) full → auto → ask → allowlist → deny
   • (b) deny → allowlist → ask → auto → full
   • (c) ask → deny → auto → full → allowlist
   • (d) allowlist → deny → full → auto → ask

3. (MCQ) "Hermes" names two distinct things from Nous Research. Which statement is correct? (see Concepts §"Hermes: one name, two artifacts")

   • (a) Hermes is only a model; the framework is unrelated.
   • (b) Hermes the model is the brain you run; Hermes Agent is the framework that runs a model with a learning loop, gateway, and subagents.
   • (c) Hermes Agent is a model; Hermes is a benchmark.
   • (d) They are two names for the same artifact.

4. (MCQ) Why does pointing OpenClaw at Ollama's /v1 endpoint break tool calling? (see Concepts §"Hermes" and reference/agents/openclaw.md §6)

   • (a) The /v1 endpoint is slower.
   • (b) OpenClaw talks to Ollama's native /api/chat; /v1 is the OpenAI-compatible endpoint and breaks tool calls.
   • (c) /v1 requires a cloud subscription.
   • (d) It doesn't; /v1 is required.

5. (MCQ) In the observability notebook, a span is: (see notebook cell [4] and Concepts §"Agent operations")

   • (a) one instrumented step with inputs, outputs, latency, and tokens, linked into a trace.
   • (b) the entire run's final answer.
   • (c) a model's parameter count.
   • (d) a message in a chat channel.

6. (Short answer) The cost dashboard prices hermes-4-14b at $0.00 per token. What do you pay for a local model instead, and why does the runbook still track it in the dashboard? (see notebook cell [9] and Concepts §"Worked example 1")

7. (MCQ) The production-eval sampling cell decides SHIP vs HOLD against a threshold of 0.85. If the sampled pass rate p is below 0.85, the decision printed is: (see notebook cell [11])

   • (a) SHIP
   • (b) HOLD, investigate before shipping
   • (c) RETRY with a larger sample
   • (d) No decision is printed

8. (Short answer) The final cell prints COVERAGE as instrumented / total_ops over 100 operations, 85% of them traced. Explain what the other 15% represents and why the runbook targets 100% coverage rather than accepting 0.85. (see notebook cell [14] and Concepts §"Worked example 2")

9. (MCQ) Which is the single most common first-run OpenClaw failure, per reference/agents/openclaw.md §9.1? (see Concepts §"How it breaks")

   • (a) The Gateway port is always free.
   • (b) Tool calling breaks when pointed at Ollama's /v1 endpoint.
   • (c) Skills always fire.
   • (d) The model remembers everything automatically.

10. (Short answer) Name the four things the ops runbook states, and explain in one sentence why a runbook is "a standing discipline" rather than documentation. (see notebook cell [13] and Concepts §"Agent operations")

---

Answer key

1. (b): USER.md holds stable preferences (who you are, how you like answers); MEMORY.md holds durable facts/decisions to persist.

2. (b): deny → allowlist → ask → auto → full, from least to most permissive.

3. (b): The model (Hermes 2/3/4) is the brain; Hermes Agent is the separate framework. They're independent artifacts you can mix and match.

4. (b): OpenClaw uses Ollama's native /api/chat; the /v1 endpoint is the OpenAI-compatible surface and breaks tool calling.

5. (a): A span is one instrumented step (model call or tool call) with inputs, outputs, latency, and tokens; the trace is the sequence of spans for a run.

6. You pay in hardware and latency (the model runs on your machine), not per-token API spend. The dashboard still tracks it because an always-on assistant's resource cost and latency matter for the runbook's latency/cost alerts even at $0 marginal token cost.

7. (b): HOLD (below 0.85): investigate before shipping.

8. The other 15% are operations that bypass the tracer (untraced_step). The runbook targets 100% because a failure in uninstrumented traffic is invisible, the 15% you don't trace is exactly where a silent regression will hide, so the metric is the honest "how much do we actually see?"

9. (b): Tool calling breaking when pointed at Ollama's /v1 endpoint is the most common first-run failure.

10. The runbook states: thresholds (pass rate, cost/day, latency, coverage), dashboards to watch, and what-to-do-when steps (for pass-rate drops, cost spikes, latency spikes, and low coverage). It is a standing discipline because it encodes the recurring decisions and thresholds you act on every day, not a one-time description you write and forget.