Week 09: Quiz (10 questions, 8/10 to pass)

Answer on your own first, then check the answer key. Each question notes where the answer lives, so you can re-read the exact section or cell you missed.

Questions

1. (MCQ) A model has 13B parameters. How much weight-only VRAM does it need in INT4, and which rule gives you the answer? (see Concepts §"One number per format")

   • A) ~26 GB, 13B × 2 bytes
   • B) ~13 GB, 13B × 1 byte
   • C) ~6.5 GB, 13B × 0.5 bytes
   • D) ~3.25 GB, 13B × 0.25 bytes

2. (MCQ) Under integer quantization with scale ≈ 0.0039 and zero_point = 0, the float 0.5 maps to which integer? (see Concepts §Worked example 2)

   • A) 128
   • B) 255
   • C) 0
   • D) 64

3. (Short answer) Why does the knowledge base say perplexity is "a screen, not a verdict"? Give the specific ZoroLogistics failure a near-flat perplexity ladder can hide. (see Concepts §"Perplexity is a screen, not a verdict")

4. (MCQ) On the 7B triage model, Q2_K loses ~8 F1 points vs FP16 while its perplexity looks only slightly worse. What is the correct production decision, and why? (see Concepts §Worked example 3)

   • A) Ship Q2_K, it is the smallest and cheapest
   • B) Ship Q4_K_M, it keeps most F1 at a fraction of VRAM; avoid Q2_K
   • C) Ship FP16, any quantization is too risky
   • D) Ship Q8_0, it is the only near-lossless option

5. (MCQ) In notebooks/01-quantization-lab.ipynb, what does the perplexity() helper (cell 12) actually compute? (see notebook cell 12)

   • A) exp(mean cross-entropy loss) over the note's tokens
   • B) The triage accuracy over 30 tickets
   • C) The bytes-per-parameter of each format
   • D) The fraction of tokens that decode correctly

6. (Short answer) In the lab notebook, what is the difference between LIVE_FP16 and LIVE_QUANT, and what hardware condition sets each to True? (see notebook cell 4)

7. (MCQ) Which mechanism explains why vLLM sustains higher throughput than a naive batcher under mixed-length load? (see Concepts §"Serving")

   • A) It quantizes every model to INT4 automatically
   • B) PagedAttention + continuous batching back-fill idle GPU time with new tokens
   • C) It always uses a batch size of 1
   • D) It runs the model in FP32 for accuracy

8. (MCQ) In notebooks/02-vllm-serving-and-benchmarks.ipynb, why does the same OpenAI SDK client work against both vLLM and Ollama? (see notebook cell 14)

   • A) Both engines were trained on the same data
   • B) Both expose an OpenAI-compatible /v1 surface; only base_url changes
   • C) The notebook patches the SDK for each engine
   • D) Ollama imports vLLM under the hood

9. (Short answer) You measure: FP16 triage accuracy 0.92, INT8 0.92, INT4 0.89, with INT4 at 25% of FP16 VRAM. Write the one-sentence production recommendation the Friday report requires, with the numbers. (see Concepts §Worked example 3 + Friday gate)

10. (MCQ) Which failure mode does the "How it breaks" section attribute to weight-only quantization on GPU? (see Concepts §"How it breaks")

    • A) It always makes the model larger
    • B) It reliably shrinks memory but is not automatically faster without fused kernels
    • C) It removes the KV cache
    • D) It requires Apple MPS to run

Answer key

1. C) INT4 is 4 bits = 0.5 bytes/param, so 13B × 0.5 = 6.5 GB. The rule is "roughly 1 GB per billion parameters at 4-bit," and bytes = params × (bits / 8).

2. A) q = round(0.5 / 0.0039) = round(128.2) = 128. The scale is (max − min) / 255 = 1.0 / 255 ≈ 0.0039.

3. Perplexity is a fast proxy for fluency, not for the specific job. A quantized model can have near-identical perplexity yet miss a named entity, emit broken JSON, or route a billing ticket to claims. You must run the task eval (e.g., 30-ticket triage accuracy) to make the shipping decision.

4. B) Q4_K_M keeps triage F1 at ~0.897 (a ~1.5-point loss) for ~3.5× less VRAM and ~3.4× lower cost; Q2_K's 8-point F1 drop (0.912 → 0.831) would silently misroute a meaningful share of tickets, so it is the avoid zone.

5. A) It computes math.exp(total_loss / total_tokens), the exponentiated mean cross-entropy over each note's tokens, averaging loss across notes so short and long notes contribute equally.

6. LIVE_FP16 = (DEVICE in ("cuda", "mps")): FP16 runs on NVIDIA and Apple Silicon. LIVE_QUANT = (DEVICE == "cuda"), bitsandbytes 8/4-bit needs CUDA only. On MPS only FP16 runs live; on CPU both are False and the fallback path runs.

7. B) Continuous batching admits new requests token-by-token and evicts finished ones immediately, and PagedAttention's fixed-size KV pages eliminate fragmentation, so the GPU back-fills a straggler's idle time with new work.

8. B) Both expose an OpenAI-compatible /v1/chat/completions endpoint; the notebook only changes base_url (vLLM on :8000, Ollama on :11434) and the model name, which is the interoperability superpower.

9. A passing answer names the trade and the ship/no-ship line. E.g., "Ship INT4: it keeps 96.7% of FP16 triage accuracy (0.89 vs 0.92) at 25% of the VRAM, and here are the tickets it misfiles." (Your numbers must come from your run, not the estimate.)

10. B) Weight-only quants shrink memory reliably, but the speed win depends on fused kernels, without them you get smaller but not quicker.