K

Knowledge Distillation

• Category: Math & training
• What it actually means: Training a student model to reproduce selected behavior or output distributions from a more capable teacher, often alongside ordinary target labels.
• Why it matters: It can transfer useful behavior into a smaller or cheaper model when serving the teacher directly is impractical.
• In practice: Define teacher outputs, temperature, student loss, and a held-out eval set, then compare the student with both the teacher and a label-only baseline.
• Common confusion: Distillation transfers behavior on the training distribution; it does not copy every capability, fact, or safety property of the teacher.
• Related terms: Fine-tuning, Loss Function, Logits, Quantization
• Sources: Distilling the Knowledge in a Neural Network

KV Cache

• Category: Models & inference
• What people say: A cache that makes token generation faster.
• What it actually means: Stored key and value tensors from earlier positions in autoregressive generation. Reusing them avoids recomputing attention projections for the unchanged prefix at every decoding step.
• Why it matters: It reduces repeated computation but consumes memory that grows with sequence length, layers, batch, and model configuration.
• Common confusion: A KV cache is runtime attention state for a sequence. Prefix caching reuses eligible KV state across requests, while prompt caching is a broader provider or application reuse contract.
• Learn it: KV Cache and Flash Attention
• Related terms: Attention, Autoregressive, Prefix Caching, Prompt Cache