06, Model Engineering: Quantization, Fine-Tuning & Serving

Purpose. How to make models cheaper, faster, and genuinely yours, the selection order that tells you when to touch the weights, the quantization formats that shrink them, the fine-tuning techniques that reshape them, the serving engines that put them in production, and the cost/latency discipline that keeps the whole thing affordable.

Part of AI Engineering Lab · Developed by Zorost Intelligence AI Lab · zorost.com

1. The selection order: cheapest lever first

Model engineering is not "fine-tune everything." It is a decision order that Andrew Ng popularized and that most production AI teams converge on: start with the cheapest, fastest, most reversible lever and only escalate when a measured gap forces you to. The order is:

Managed model API + a prototype. Call a frontier or mid-size model through an API and build the end-to-end product skeleton first. You learn what the task actually is before you optimize how it is done.
Prompting and context engineering, with an eval. System prompts, few-shot examples, structured-output schemas, and context budgeting. This is where most "the model can't do it" problems are actually solved, for pennies and minutes, not GPU-days.
RAG. When the answer must come from your data, policies, tickets, product records, retrieve it and put it in the context instead of memorizing it in the weights.
Agentic workflow. When the job is multi-step, decompose, call tools, verify, retry, give the model a loop rather than a bigger model.
Fine-tune (or train) last. Only when the eval shows the base model plus all four levers above still misses a specific, high-volume target.

Why this order? Each step trades money and risk for capability, and each step is harder to undo than the last. Prompt changes deploy in seconds and revert instantly; a fine-tune costs GPU-hours and a training run, can regress other skills (catastrophic forgetting), and has to be re-run when the base model improves. The rule of thumb is blunt but true: fine-tuning is justified by a measured gap, not by instinct. If you cannot point to an eval score that prompting + RAG + tooling fail to reach, you are not ready to train.

Worked example: ZoroLogistics bill-of-lading extraction. Week 6 tried prompts and structured output on a hosted model and reached 91% field accuracy. The naive instinct is "fine-tune to close the last 9%." The disciplined read: error analysis showed the misses were mostly handwriting variants in the source document, a data problem, not a weights problem. The right next lever was better OCR preprocessing (context/input engineering), then RAG over a glossary of carrier abbreviations, then a fine-tune only if a stable, high-volume style/format gap remained. Each escalation was justified by the eval, not by enthusiasm.

The exception that proves the rule: fine-tuning is the right tool for style, tone, strict output format, and niche domain vocabulary, things that are expensive to express in a prompt and impossible to express in retrieved text, and for compressing a large model's behavior into a small, cheap, on-device specialist (see §3.8 distillation).

2. Quantization: shrink the numbers, keep the skill

Quantization lowers the numerical precision of weights (and sometimes activations) to shrink memory and speed up inference. An LLM's billions of parameters are stored as floating-point numbers; representing each one in fewer bits cuts file size, VRAM, and memory bandwidth almost linearly, at a small accuracy cost.

2.1 Numeric formats

Format	Bits/param	What it is and when to use it
FP32	32	Full single precision. The reference format for training checkpoints; ~4× the memory of INT8 with negligible inference benefit. Almost never used for serving.
FP16	16	Standard half precision. ~2× smaller than FP32 with negligible quality loss for inference; the baseline for "full-precision" serving.
BF16	16	Same exponent range as FP32 but fewer mantissa bits. More stable for training (no overflow), so it is now the training default; inference quality ≈ FP16.
FP8	8	E4M3/E5M2 formats used for inference (and some training) on Hopper-class GPUs via the Transformer Engine. Needs hardware support to pay off.
INT8	8	~2× smaller than FP16; needs per-tensor or per-channel scale factors to map floats to integers. Near-lossless in practice.
INT4	4	~4× smaller than FP16; the aggressive but widely accepted chat/inference default (GGUF Q4, bitsandbytes NF4).

The two families matter differently: FP8/BF16/FP16 are floating-point, they spread a limited number of distinct values over a wide range, which suits weights whose magnitudes vary a lot. INT8/INT4 are integer, they need a scale factor (per-tensor or per-channel) to map a float range onto a small set of integers; the quality of that mapping is most of what separates a good quant from a bad one.

2.2 Quantization methods

Method	What it does	Bits	Needs calibration data?	Best fit
GGUF k-quants (`llama.cpp`)	A self-contained file format plus a family of practical quant tiers baked into llama.cpp and everything downstream of it (Ollama, LM Studio). `Q2_K` → `Q8_0` from lossy to near-lossless.	2 to 8	No	Local CPU/GPU inference; the standard "drop-in" artifact
bitsandbytes	Runtime quantization library: 8-bit (`LLM.int8()`) and 4-bit NF4 (normal-float-4, a data-aware format). The backbone of QLoRA. `load_in_4bit=True`.	8 / 4	No	Loading a big model in a notebook; QLoRA fine-tuning
AWQ	Activation-aware weight quantization: protects the small fraction of weights that matter most (by activation magnitude) and scales the rest.	4 (also 3/8)	No (activation stats only)	High-quality 4-bit serving on vLLM/TGI
GPTQ	Layer-wise quantization using second-order (Hessian) information from a calibration dataset. High quality but slower to produce.	4 / 8 / 3	Yes (calibration set)	Batch-quantizing a model once, then serving it anywhere

Weight-only vs activation quantization. Most local quants (GGUF, AWQ, GPTQ) are weight-only: they shrink the weights but compute activations in higher precision at runtime. The memory win is real (weights dominate storage), but the speed win depends on the kernel: on CPU, smaller weights reduce bandwidth pressure; on GPU, weight-only quants do not always speed up compute unless the engine has fused kernels for them. Do not assume 4-bit = 4× faster; it is reliably ~4× smaller and often faster.

GGUF k-quant ladder (lower number = smaller/faster but lossier): Q2_K (aggressive) → Q3_K_* → Q4_K_M (the community sweet spot for quality/VRAM) → Q5_K_M → Q6_K → Q8_0 (near-FP16). Start at Q4_K_M; step up to Q5_K_M/Q6_K when quality matters and VRAM allows; avoid Q2_K unless memory is desperate.

Calibration in one line. Methods that need a dataset (GPTQ) use it to observe which values the weights take and how sensitive each layer is, then assign scale factors that minimize the reconstruction error. More representative calibration → better quant; garbage calibration → a model that looks small and behaves worse than its size.

2.3 Acceptable vs avoid

Acceptable at 4-bit (and 8-bit is often essentially free): conversation, summarization, classification, RAG, and harness-level coding assistance, tasks where a tiny word-choice drift does not break correctness.
Be cautious / avoid heavy quantization (2 to 3-bit, or 4-bit on a model you have not tested): precise arithmetic or math, long-horizon multi-step reasoning, low-resource languages, and exact code generation where a single wrong token breaks compilation. Small models degrade faster under quantization than large ones, so a 1B model at 4-bit is riskier than a 70B at 4-bit.

Heuristic. Quantize as aggressively as you like for the easy path, but keep a higher-precision model (or a hosted API) as the fallback for the hard cases. Routing "easy → Q4 local, hard → FP16/API" is cheaper than either extreme alone.

2.4 Measuring quality loss

Perplexity is a fast proxy: lower is better, and it captures how well the model predicts held-out text. The general ladder is FP16/BF16 ≈ FP32 for inference; 8-bit ≈ negligible loss; 4-bit k-quants ≈ small, usually imperceptible loss for chat/RAG/summarization; 2 to 3-bit = noticeable degradation.
But perplexity is not task accuracy. A quantized model can have near-identical perplexity and still flub your specific eval (the wrong output format, a missed named entity, a broken JSON). Always eval on your task, your golden set with your metric, before shipping a quant. Perplexity screens candidates; the task eval decides.

How to run the sweep. Pick one task eval, run it against FP16 (baseline) → Q8 → Q4_K_M → Q3/Q2, and record (metric, VRAM, latency, cost). Plot metric vs bits. The knee of that curve, where the next quant tier costs more than it saves, is your production choice. This is exactly the Week 9 quantization-sweep exercise.

2.5 How a quantizer maps floats to integers (worked example)

Under the hood, integer quantization maps each float weight w to an integer q with:

q = round(w / scale) + zero_point          # quantize
ŵ = scale × (q − zero_point)                # dequantize

scale is (max − min) / 255 for an 8-bit range, and zero_point centers the mapping. The model stores the small integers plus the per-tensor (or per-channel) scales; at inference the engine multiplies back. The entire craft is choosing scale so the mapping discards the least signal.

Concrete numbers. A layer whose weights span [−0.5, 0.5] gets scale = 1.0 / 255 ≈ 0.0039 and zero_point = 0. So 0.5 → round(0.5 / 0.0039) = 128 and −0.5 → −128. Reconstructing gives ±0.5 with a maximum error of ~0.002, imperceptible when summed over millions of weights. Per-channel quantization (one scale per output channel) beats per-tensor because it does not let a single outlier weight inflate the scale for the whole layer; that is why good 4-bit quants are per-channel or use a nonlinear (NF4) grid.

FP8/BF16/FP16 skip all of this, they are still floats, just with fewer exponent/mantissa bits, which is why they are simpler and why INT formats exist mainly to go smaller than a float can.

3. Fine-tuning: change the weights, not just the prompt

3.1 When it pays vs when it doesn't

Fine-tuning pays when the target is stable, high-volume, and hard to express in a prompt or retrieve:

Style and tone: "write like our support team, formal but warm."
Strict output format: a fixed JSON schema or template the base model keeps drifting from.
Domain vocabulary and behavior: freight-specific fields, medical phrasing, a house taxonomy.
Compression/distillation: teach a small specialist to mimic a large teacher so you can run it cheaply or on-device.

Fine-tuning does not pay when you want new knowledge (that is RAG's job, weights are the wrong place to store facts that change), when the behavior is a one-off or prompt-fixable, or when you have too little high-quality data (see §3.3). New knowledge should be retrieved, not memorized, and a fine-tuned model that "knows" last quarter's prices is already wrong this quarter.

3.2 The adapters: LoRA, QLoRA, DoRA

Parameter-efficient fine-tuning (PEFT) freezes the base weights and trains only a small set of new parameters, so you get most of the benefit of fine-tuning for a fraction of the compute and storage.

LoRA (Low-Rank Adaptation). Freezes the base weights and injects trainable low-rank matrices into attention/MLP layers. The weight update is factored as ΔW = B·A, where A and B are small matrices of rank r. Trains ~0.1 to 1% of parameters; the adapter is tiny, swappable, and can be merged back into the base weights.
- Rank r intuition: the rank is the number of "degrees of freedom" you give the update. r=4-16 is plenty for style/format; r=32-64 for harder task shifts. More rank = more capacity but more risk of overfitting and more memory.
- Alpha intuition: alpha scales the update's magnitude (effective strength ≈ alpha/r). Raise alpha for a stronger update; lower it to be conservative. In practice alpha = 2·r is a common default.
QLoRA. LoRA on top of a 4-bit (NF4) base model (bitsandbytes), with double quantization and paged optimizers. This is what lets you fine-tune a 70B on a single 24 GB consumer GPU.
DoRA (Weight-Decomposed LoRA). Splits each weight into magnitude + direction and adapts only the direction with LoRA while the magnitude is learned separately. It often matches full fine-tuning better than plain LoRA at the same rank, at a modest extra cost.

Rank/alpha quick table:

Task type	Rank `r`	Alpha	Notes
Style / tone / format only	8	16	Minimal capacity; least overfit risk
Domain behavior (extraction, classification)	16 to 32	32 to 64	The most common starting point
Hard task shift (reasoning, tool use)	64	128	Only with plenty of data; watch overfitting

When the adapter overfits (train score up, eval score down), reduce r or alpha, do not add epochs.

3.3 SFT dataset design: quality beats quantity

Supervised fine-tuning (SFT) is imitation learning on (instruction → answer) pairs. The single most important rule: a few hundred clean, diverse, on-task examples beat tens of thousands of noisy ones. A small model internalizes whatever pattern dominates the data, if 80% of your examples are off-target or contradictory, that is what you teach.

Diversity over volume. Cover the edge cases and the failure modes from your error analysis, not just the happy path. The goal is coverage of behavior, not count of rows.
Match the chat template. Use the model's native chat format, a messages array of role/content (the most portable form today), or a legacy instruction/input/output triple if your tooling requires it. A template mismatch silently degrades training.
De-duplicate and clean. Remove near-duplicates, fix malformed examples, and strip anything you would not want the model to imitate.
Hold out an eval set. Never train and evaluate on the same examples; the eval set is the gate that decides whether the adapter ships.

Sizing. For a narrow task, 200 to 1,000 high-quality examples is often enough to move the needle; 5,000 to 20,000 is a substantial dataset; beyond that you are usually buying diminishing returns unless the task is genuinely broad. How many you need is a function of how much variety the task has, not how impressive the number looks.

Example (chat-template JSONL):

json

{"messages": [{"role": "system", "content": "You triage freight support tickets."},
              {"role": "user", "content": "Shipment #8841 arrived 3 days late and the box is crushed."},
              {"role": "assistant", "content": "{\"category\": \"damage\", \"priority\": \"high\"}"}]}

3.4 SFT vs DPO vs RLVR

Objective	What it optimizes	Data needed	Use it for
SFT	Imitation of good answers	`(instruction, answer)` pairs	Style, format, task behavior, the standard first step
DPO	Preference: chosen > rejected, directly (no reward model)	`(prompt, chosen, rejected)` pairs	Alignment and preference shaping; simpler/more stable than RLHF/PPO
RLVR	A checkable reward signal (tests pass, answer correct, code compiles)	Verifiable tasks + a reward function	Frontier reasoning/coding gains; the current RL frontier method

Why DPO over RLHF/PPO? Classic RLHF needs a separately trained reward model and a policy-optimization loop (PPO) that is finicky to tune. DPO folds the preference signal directly into a supervised-style objective, simpler, more stable, and often as good. RLVR goes a step further: instead of a learned reward model (which can be gamed or miscalibrated), it uses a verifiable reward, the unit test passes or it doesn't, so the signal cannot be fooled.

The practical sequence: SFT first to teach the shape of the task, then DPO to sharpen preference, then RLVR only if you have a verifiable reward and a reason to push further.

3.5 Catastrophic forgetting checks

Fine-tuning on one task can erode the model's general skills. Always run a before/after eval on the base model's original capabilities, not just the new task:

Keep a general benchmark slice (or your existing Week 6/7 eval sets) and confirm it does not regress more than a small threshold.
Compare base vs fine-tuned on the same eval and publish both columns, the deploy decision is "does the gain on the new task justify any loss elsewhere?", not "did the new task improve?"
If forgetting is real, lower the learning rate, add a small fraction of general data back into the mix, or reduce rank/alpha.

The before/after table is non-negotiable. Two columns (base, fine-tuned) × two rows (new-task eval, general eval) is the minimum artifact that lets anyone, including future you, see the actual trade.

3.6 Frameworks (quick reference)

Hugging Face TRL + PEFT: the default Python stack: SFTTrainer, DPOTrainer, GRPOTrainer (RLVR), and LoraConfig/get_peft_model.
Unsloth: a drop-in speed/memory layer for fine-tuning (2 to 5× faster, ~50 to 70% less VRAM), native TRL integration; popular for LoRA/QLoRA on Llama/Mistral/Qwen.
Axolotl: YAML-config-driven fine-tuning over TRL/PEFT; declare model, dataset, and LoRA/QLoRA/DoRA config, and it wires the run. Ideal for reproducible, shareable recipes.

Budget note. A 7 to 8B QLoRA run on a free Colab tier or a 16 GB MacBook is entirely feasible for SFT; DPO/RLVR are heavier but still doable on small models. Fine-tuning does not require a datacenter, it requires a clean dataset.

3.7 Distillation

Distillation trains a small student model to mimic a large teacher (by matching logits or hidden states). Its goal is different from LoRA: you shrink the model, not just the adapter. It is the right tool when a large teacher has proven a task and you want a compact specialist to deploy cheaply or on-device, the student inherits behavior, not size. LoRA and distillation compose: distill a 70B teacher's task into a 7B student, then LoRA the student for your specific format.

3.8 The fine-tuning loop is an eval loop

Fine-tuning is not a one-shot: it is a cycle of train → eval → error-analyze → improve the dataset → retrain. The eval, not the loss curve, decides when you stop and whether you ship:

Train a first adapter on a small, clean dataset.
Run the eval; compute before/after on the new task and the general slice.
Cluster the remaining failures (Week 11's error-analysis habit).
Add examples that target the biggest failure class, this is how the dataset improves, one failure class at a time.
Re-train and re-check; stop when the eval gain per added example flattens.

The trap is treating the dataset as fixed and the model as the only variable. In practice the dataset is the higher-leverage variable, and the fastest way to improve a fine-tune is almost always better data, not more epochs or a bigger rank.

4. Serving: from a model file to an API

4.1 Engine comparison

Engine	Key mechanism	Strengths	OpenAI-compatible	Best fit
vLLM	PagedAttention + continuous batching	Highest throughput, huge ecosystem	Yes (`vllm serve <model>`)	General production serving, the default
SGLang	RadixAttention (prefix caching) + fast structured decoding	Fast shared-prefix and JSON/tool decoding	Yes	Agent loops, repeated system prompts
TGI	Rust/Python, polished defaults	Ops-friendly, broad quant support, easy Docker	Yes	HF-native, containerized deploys

Choose vLLM for general high-throughput serving, SGLang for prefix-heavy workloads (many requests sharing a long system prompt) and structured outputs, and TGI when you want a batteries-included, HF-native container.

4.2 Continuous batching

Traditional serving waits for a whole batch of requests to finish before starting the next; one slow response stalls everyone. Continuous batching admits new requests token-by-token and evicts finished ones immediately, so the GPU is never idle waiting for a straggler. This, combined with vLLM's PagedAttention, which stores the KV cache in fixed-size pages to nearly eliminate fragmentation, is why modern engines sustain far higher throughput than naive batchers.

Concrete effect. A naive server handling 8 requests of varying length idles whenever the longest one is still generating; a continuous-batching server back-fills that idle time with the next token of a new request. Under mixed-length load the throughput difference can be several×.

4.3 KV cache memory

Every token generated must attend to all prior tokens, so the server caches each layer's key/value vectors per token. The KV cache grows with context length × layers × hidden size, and at long context it can dwarf the weights.

Rough worked example. A 7B model with ~32 layers and a 4096 KV dimension holds 2 × 32 × 4096 × 2 bytes ≈ 0.5 MB of KV cache per token at FP16. A 4,096-token context is then ~2 GB of KV cache, on top of the ~4 GB of Q4 weights. Double the context, and KV doubles while weights stay fixed. That is why context length is a first-class memory cost, not an afterthought.

Practical consequences:

Cap max_model_len to what your workload actually needs, every unused context slot is wasted VRAM.
Quantize the KV cache (FP8) on supported engines to reclaim memory.
Use SGLang's prefix cache (or vLLM's) so requests sharing a prefix do not recompute it.

4.4 Throughput vs latency knobs

Knob	Turn it up for throughput	Turn it down for latency
`max_num_batched_tokens` / `max_num_seqs`	Larger batch = higher throughput	Smaller batch = lower per-request latency
`gpu_memory_utilization` (~0.9)	Higher = more KV cache, more concurrency	Leave headroom to avoid OOM
`tensor_parallel_size`	Split across GPUs for bigger models	Single GPU for small models (less comm overhead)
`max_model_len`	n/a	Cap context to save KV memory
`quantization` (awq/gptq/fp8)	Smaller weights = more fits	Same model, fewer bytes = faster decode

The tension is real: you buy throughput with latency. Decide which one the workload cares about (batch jobs → throughput; interactive UX → latency) and tune accordingly. Speculative decoding is an advanced latency lever that pays only when it does not cost accuracy.

4.5 OpenAI-compatible APIs

All three engines expose an OpenAI-compatible /v1/chat/completions (and /v1/completions) surface, so any OpenAI SDK client, and any coding-agent harness, can point at your local server by swapping base_url. This is the single most useful interoperability fact in serving: you can develop against a local model and, later, switch to a hosted endpoint (or vice versa) with a one-line change.

python

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")  # vllm serve
client.chat.completions.create(model="my-7b", messages=[...])

4.6 Serving a quantized model with vLLM (worked)

Putting §2 and §4 together, serve the Week 9 Q4_K_M (or AWQ/GPTQ) 7B behind an OpenAI-compatible endpoint:

# AWQ/GPTQ 4-bit (serving quants built for the engine)
vllm serve TheBloke/My-7B-AWQ --quantization awq --gpu-memory-utilization 0.9 \
       --max-model-len 4096 --max-num-seqs 32

# Or a GGUF model (llama.cpp-style quant) via vLLM's GGUF loader
vllm serve ./my-7b-q4_k_m.gguf --max-model-len 4096

Then point any OpenAI SDK at http://localhost:8000/v1 as above. The flags map directly to §4.4's knobs: --gpu-memory-utilization 0.9 (VRAM share), --max-model-len 4096 (cap context → cap KV cache), --max-num-seqs 32 (concurrency → throughput vs latency). A higher --max-num-seqs raises throughput and per-request latency; a lower one does the opposite. The same shape of command exists for SGLang (python -m sglang.launch_server) and TGI (Docker ghcr.io/huggingface/text-generation-inference), which is what makes the engine choice a config decision rather than a rewrite.

5. Cost & latency engineering

Cost and latency are architecture constraints, not afterthoughts. The sharp metrics are not cost-per-token and average latency but:

Cost per resolved task: how much a correct, accepted answer costs, retries included. A cheap model that needs three attempts is not cheap.
p95 latency: the latency that 95% of requests beat. Averages hide the tail, and the tail is what users feel.

The standing exercise: measure cost per resolved task and p95 latency, then halve both without losing eval score (Zorost Signal, "Cost and latency engineering for LLM systems"). Every meaningful reduction forces a real tradeoff, and that tradeoff is where the engineering lives:

Cache: prompt/prefix caching, plus semantic caching of common answers so the model is not even called.
Route: a smaller model on the easy path, a bigger one only when the small one is unsure (an eval-gated cascade).
Quantize: same model, half the VRAM, faster decode, lower per-token cost.
Cut passes: fewer retrieval calls, one reasoning pass instead of three, a verifier that runs on a sample instead of everything.
Batch: amortize compute across requests when latency allows.

Each of these must be re-checked against the eval; the whole point is that you only keep the reductions that do not degrade the score. Cost per resolved task, not per token, is the metric that catches the "cheap but wrong, so it retries" trap; p95, not mean, is the metric that catches the tail.

6. ZoroLogistics example: quantizing the 7B triage model

The Week 8 support-ticket triage model is a 7B open model that classifies inbound freight tickets into \{billing, delay, damage, documentation, other\} and extracts a short structured summary. Running it in FP16 needs ~14 GB of VRAM and, on a rented A10, costs real money per ticket. The task: quantize it and find the point where cost and latency drop without the triage eval falling off a cliff.

Sweep (measured on the Week 11 golden triage eval; p95 latency = end-to-end per ticket at 1 concurrent request; $/task on an A10-class GPU):

Variant	Format	VRAM	Triage F1	p95 latency (per ticket)	$/task
FP16 (baseline)	FP16	~14 GB	0.912	820 ms	$0.031
8-bit (GGUF `Q8_0` / bitsandbytes 8-bit)	INT8	~7 GB	0.909	540 ms	$0.017
4-bit (GGUF `Q4_K_M`)	INT4	~4 GB	0.897	410 ms	$0.009
2-bit (GGUF `Q2_K`)	INT2	~2.4 GB	0.831	320 ms	$0.006

Decision. Q4_K_M is the production pick: 3.5× less VRAM, ~2× lower latency, and ~3.4× cheaper per task for a ~1.5-point F1 loss, a clearly acceptable trade for triage, where the cost of a mis-route is a quick human re-route, not a safety event. Q8_0 is the conservative alternative when the eval budget is tight. Q2_K is the avoid zone: the F1 drop is real and would silently misroute a meaningful share of tickets. The report ships with the perplexity ladder and the task F1, because perplexity alone would have made Q2_K look fine.

What this teaches. Quantization is a decision, not a default. The right answer is the knee of the curve on your eval, and the artifact you keep is the sweep table, so when the next base model drops, you re-run the sweep instead of re-arguing the decision.

7. Bringing it together: the deployment gate

The three disciplines, quantization, fine-tuning, serving, are not separate labs; they are three knobs on one decision, and the eval is the only thing allowed to turn them. This section makes the through-line concrete.

A minimal QLoRA recipe (conceptual)

python

from transformers import BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True)
lora = LoraConfig(r=16, lora_alpha=32,
                  target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
                  lora_dropout=0.05, bias="none", task_type="CAUSAL_LM")
model = get_peft_model(model, lora)   # base loaded in 4-bit → this is QLoRA

Train for 1 to 3 epochs with a low learning rate (~2e-4), hold out an eval set, and log loss and eval score, not loss alone. Stop when the eval stops improving, not when the loss curve looks nice. This is the recipe that fits a 7 to 8B fine-tune on a free Colab tier or a 16 GB Mac.

A worked "halve both" pass

The ZoroLogistics support bot starts at $0.040 per resolved task, p95 900 ms, eval 0.90. Four levers, each re-checked against the eval:

Lever	Effect
Route ~60% of easy tickets to a Q4 7B local model	Easy path drops to ~$0.004 and ~400 ms
Prefix-cache the shared system prompt	~15% fewer billed tokens on the API path
Quantize the API model 8-bit (or switch to a cheaper tier)	Lower per-token cost, no eval change
Run the verifier on a 20% sample instead of 100%	Cuts verification cost without moving the score

Net: ~$0.019 per resolved task and p95 ~430 ms, eval still 0.90, both roughly halved, score held. The point is not these specific numbers but the method: every reduction is measured against cost-per-resolved and p95, and nothing is kept that drops the eval. (Zorost Signal.)

The five-question gate

Ask these before any model-engineering change ships:

Did I exhaust prompt → RAG → agentic before touching the weights?
Is this quant justified by a task eval (not just perplexity)?
Does this fine-tune ship a before/after table covering both the new task and general skills?
Am I optimizing cost per resolved task and p95 latency, not cost-per-token and the mean?
Is the serving engine and batch size matched to the workload's throughput-vs-latency need?

If the answer to any is "no," the change is not ready, regardless of how good the demo looked.

The selection order, on one diagram

mermaid

%%{init:{"theme":"base","fontFamily":"Helvetica Neue,Helvetica,Arial,sans-serif","flowchart":{"curve":"basis","padding":14,"nodeSpacing":45,"rankSpacing":55},"themeVariables":{"fontSize":"15px","background":"#FFFFFF","primaryColor":"#EEF2F7","primaryTextColor":"#14213D","primaryBorderColor":"#14213D","secondaryColor":"#FFF1E3","secondaryTextColor":"#14213D","tertiaryColor":"#E7F4F1","tertiaryTextColor":"#14213D","lineColor":"#64748B","textColor":"#14213D","edgeLabelBackground":"#FFFFFF","clusterBkg":"#F5F8FC","clusterBorder":"#94A3B8","nodeBorder":"#14213D","mainBkg":"#EEF2F7","titleColor":"#14213D"}}}%%
flowchart TD
    A["Cheapest, most reversible lever first"] --> P["1 · Prompt + context + eval"]
    P -->|"measured gap remains?"| R["2 · RAG"]
    R -->|"gap remains?"| G["3 · Agentic workflow"]
    G -->|"gap remains?"| F["4 · Fine-tune (LoRA/DPO)"]
    F --> Q["5 · Quantize + serve"]
    Q --> E["Eval gates every step"]
    E -.->|"no gap → stop"| A

Every edge is a measured gate: you move one step only when the eval shows the cheaper lever cannot close the gap. The return arrow is the discipline, the eval, not enthusiasm, decides whether the next step happens at all. Notice that quantization (5) is a serving decision that can apply to any model, dense or fine-tuned; it is on the diagram because it is part of "make it affordable," not because it is the last thing you learn.

Quantization math table: sizes across formats

§2.1 lists the formats; here is what they mean in gigabytes, which is the number that decides whether a model fits your hardware. The formula is bytes = params × bits ÷ 8.

Model	FP32	FP16 / BF16	INT8	INT4 (Q4)	One GPU that fits Q4
7B	28 GB	14 GB	7 GB	~4 GB	8 GB consumer GPU
13B	52 GB	26 GB	13 GB	~7 GB	16 GB
70B	280 GB	140 GB	70 GB	~38 GB	2× 24 GB (or 1× 48 GB)
671B MoE	~2.7 TB	~1.34 TB	~671 GB	~340 GB	A node of GPUs, or FP8 across many

Reading the table like an engineer:

Each halving of bits roughly halves bytes: FP16 → INT8 → INT4 is a clean ÷2 ladder (plus a little overhead at the low end for scale factors and the KV cache).
MoE is the outlier that teaches the tradeoff. A 671B MoE at INT4 is ~340 GB, far more memory than a 70B dense at FP16 (140 GB), even though the MoE's per-token compute is only ~37B-active. You pay for the reasoning-per-FLOP in VRAM, exactly as 03-llm-core-concepts.md's MoE section says.
The "which GPU fits Q4" column is the deployment question, and it is why the 7B Q4_K_M triage model runs on a single consumer GPU while the 671B MoE needs a node. Size, not just quality, is a selection criterion.

The arithmetic is also the honest check on "can I fine-tune this locally": QLoRA fine-tunes a 7B in ~4 GB of weights (plus gradients/optimizer state for the adapter), which is why it fits a 16 GB MacBook or a free Colab tier, while a 70B at FP16 (140 GB) does not.

LoRA rank/alpha, with worked numbers

§3.2 gives the intuition; here is the arithmetic that turns rank into a trainable-parameter count you can actually budget. Freezing the base and training only A (r × d) and B (d × r) on the attention projections of a 7B (32 layers, hidden 4096, targeting q/k/v/o):

Rank r	Trainable params (q/k/v/o)	Adapter size (FP16)	Share of 7B	Use
8	~8.4 M	~17 MB	~0.12%	Style / tone / format
16	~16.8 M	~34 MB	~0.24%	The default starting point
32	~33.6 M	~67 MB	~0.48%	Domain behavior
64	~67.1 M	~134 MB	~0.96%	Hard task shift; watch overfitting

The math for one cell: r = 16 → A is 16 × 4096 and B is 4096 × 16 = 131,072 params per module × 4 modules × 32 layers = 16.8 M trainable params. The reason the adapter is tiny is the low-rank factorization: instead of learning a full 4096 × 4096 update (16.8 M params per module), you learn two small matrices whose product is the update.

Two consequences follow:

The trainable share is what makes QLoRA feasible. The 4-bit base (~4 GB) is frozen; you only back-propagate through ~17 to 134 MB of adapters, so the optimizer state stays tiny and the whole thing fits on one modest GPU.
rank/alpha are the overfitting dial, not quality dials. More rank = more capacity = more risk of memorizing the training set. When the adapter overfits (train score up, eval score down), reduce r or alpha, do not add epochs. Alpha scales the update's magnitude (effective strength ≈ alpha / r), so alpha = 2·r is the common default and raising alpha past that is a "stronger update" lever to use cautiously.

The practical read of the table: start at r = 16, alpha = 32, and only move up for a genuinely hard task with plenty of clean data. Most "the LoRA didn't work" failures are data problems (see §3.3), not rank problems.

The cost-per-task engineering worksheet

§5 says the sharp metric is cost per resolved task, not cost per token. Here is the worksheet that computes it, with a worked ZoroLogistics fill-in. The structure matters more than the numbers: you are adding up every call in the path, then dividing by the success rate, because retries are real.

Line	Component	Tokens	$/Mtok	Cost
1	Input (prompt + evidence)	1,650	$1.00	$0.00165
2	Output	300	$3.00	$0.00090
3	Embedding (query + chunks)	1,650	$0.02	$0.00003
4	Rerank (20 candidates × 30 tokens)	600	$0.50	$0.00030
5	Verifier / judge (sample or full)	400	$1.00	$0.00040
n/a	Raw cost per attempt	n/a	n/a	$0.00328
6	Success rate	0.85	n/a	retry factor = 1/0.85 = 1.176
n/a	Cost per resolved task	n/a	n/a	$0.00328 × 1.176 ≈ $0.00386

The two lines that do the real work are line 5 and line 6. The verifier is a cost you add deliberately, and the §5 lever is to run it on a sample (20% of requests) rather than 100% once the failure rate is low, which cuts line 5 without moving the eval. The retry factor is the trap the whole metric exists to catch: a cheap model at $0.002/attempt that succeeds only 60% of the time costs $0.002 × 1/0.6 = $0.00333 per resolved task, more than the "expensive" model that gets it right the first time. Cost-per-token hides that; cost-per-resolved-task exposes it.

The halving exercise, in worksheet form. To halve the number, walk the lines: cache the shared prefix (line 1 → ~10%), cap the output (line 2), route easy tickets to a Q4 local model (lines 1 to 2 → ~$0.004), rerank fewer candidates (line 4), sample the verifier (line 5), and, the highest-leverage move, raise the success rate (line 6), because a retry factor of 1.0 beats any per-token discount. The before/after table (cost, p95 latency, eval score) is the deliverable; the worksheet is how you know which line to change.

Serving benchmark methodology

A serving benchmark that is not a benchmark is the fastest way to ship a "fast" model that falls over in production. The methodology, step by step:

Define the workload first. Real prompt-length and output-length distributions, not one synthetic prompt. The ZoroLogistics triage workload is short prompts (~1,650 tokens) and short outputs (~300); a summarization workload is long-prompt/long-output, they stress different things (prefill vs. decode).
Pick the metrics. Throughput (tokens/s), TTFT (time to first token, the prefill cost users feel first), TPOT (time per output token, the decode cost), and p95 end-to-end latency, never the mean, which hides the tail.
Fix hardware and engine version. A benchmark without the GPU model and the engine commit is unreproducible; vLLM/SGLang versions change behavior.
Sweep one knob at a time: max_num_seqs, quantization, tensor_parallel_size, max_model_len: and record the metric per setting (this is the §4.4 table in action).
Measure under steady-state concurrent load, not one warm request. A single-request latency number tells you nothing about the throughput/latency tradeoff you are about to ship.
Re-check the eval alongside every knob. Throughput is worthless if the quantization or the batch setting dropped the task F1. The §6 sweep table (variant, VRAM, F1, p95, $/task) is the correct shape: performance and quality in one table.
Report a before/after table with a decision, not just a leaderboard. "Q4_K_M at max_num_seqs=32: 3.5× less VRAM, ~2× lower p95, ~1.5-point F1 loss → ship" is a decision; a wall of tokens/s numbers is noise.

The methodology's through-line is the same as everywhere else in this file: measure the number you will actually be judged on (cost-per-resolved-task and p95 latency), and never let a performance win be recorded without the eval score next to it.

Shrink-the-model decision: quantization vs LoRA vs distillation

Three techniques all make a model "cheaper or more yours," but they change different things and compose rather than compete. The decision table:

Technique	What it changes	Result	Use it when
Quantization	The precision of existing weights	Same model, fewer bytes	You need it smaller/faster and the eval tolerates the loss
LoRA	A small trainable adapter on frozen weights	New behavior, tiny added size	You need style/format/domain behavior without retraining the base
Distillation	A smaller student model trained to mimic a large teacher	A genuinely smaller model	A large teacher has proven the task and you want a compact specialist

Decision rules:

Shrink without changing behavior → quantize. It is the cheapest and most reversible; the §6 sweep decides how far.
Change behavior on a frozen base → LoRA. The adapter is the smallest possible "customization" artifact, and it merges back into the base when you want a single file.
Shrink the model itself → distill. When the model size (not just the precision) is the problem, on-device, edge, or a fleet of cheap GPUs, a student inherits behavior, not size.
They compose. Distill a 70B teacher into a 7B student, LoRA the student for your exact format, then quantize the result to Q4. Each step is a separate, eval-gated lever, the same discipline as the selection order, applied to the model file.

The mistake to avoid is treating them as interchangeable. "Quantize it" does not add a behavior; "LoRA it" does not make it smaller (the base is still there); "distill it" does not fix a format drift. Name which of the three problems you actually have before reaching for a technique.

Fine-tuning data: the quality checklist

§3.3's rule is "quality beats quantity." Here is the checklist that operationalizes it, run it over the dataset before the first training run, because the dataset is the higher-leverage variable:

Check	What to look for	The fix if it fails
On-task and diverse	Every example teaches the target behavior; edge cases and failure modes are covered	Add examples for the biggest failure classes from error analysis
Chat-template correct	The `messages` array (or legacy triple) matches the model's native format	Re-encode with the right template, a mismatch silently degrades training
De-duplicated	Near-duplicate rows removed	Deduplicate; duplicates teach the model to over-weight one answer
Clean	No malformed examples, no off-target or contradictory answers	Strip anything you would not want imitated
Train/eval disjoint	The eval set shares no examples with training	Hold out before training; never grade on what you trained on
Sized to the variety	200 to 1,000 for a narrow task; more only if the task is genuinely broad	Add variety, not just volume, coverage of behavior, not count of rows

The last row is the one people get wrong in both directions: too little data for a broad task (the model underfits the behavior), or a mountain of near-identical rows that teach one narrow pattern (the model overfits the happy path). The question is never "how many rows" but "how much distinct behavior does this cover."

Latency levers, ranked by cost

Latency has many levers, and they are not equal. Ranked from cheapest to most expensive:

Lever	What it costs	What it buys	ZoroLogistics instance
Cap `max_model_len`	Nothing (a flag)	Less KV, faster prefill	Cap the triage context at 4,096 instead of the model max
Prompt/prefix caching	A stable prefix	Skip re-prefill of the shared head	Freeze the support-agent system prompt
Quantization	A small eval loss	Smaller weights, faster decode	Q4_K_M triage model
Smaller model on the easy path (routing)	A router + two models	Fast path for ~60% of requests	Q4 7B for easy tickets, API for hard
Continuous batching	Engineering complexity	Back-fill idle GPU time	vLLM `max_num_seqs=32`
Fewer retrieval/rerank passes	A small recall trade	Less per-request work	Retrieve 3 chunks instead of 8
Speculative decoding	A draft model + correctness risk	Lower TPOT on decode-bound work	Only if the draft matches your domain

The ranking is the same argument as the selection order, restated for latency: pull the cheap, reversible levers first (a flag, a cache, a quant), and reach for the expensive ones (a router, speculative decoding) only when the eval says the cheap ones are not enough. Every lever is re-checked against the eval, because a latency win that drops the score is a loss wearing a speedometer.

The deployment gate, as a checklist

§7's five-question gate is the release check; here it is as a pass/fail table you can run in a review meeting, with the evidence each question demands:

Gate question	Pass requires	ZoroLogistics evidence
Exhausted prompt → RAG → agentic first?	A written log of what each lever moved	"Prompt+RAG reached 0.91; fine-tune targeted the remaining 9%"
Quant justified by a task eval?	A sweep table (metric, VRAM, latency, cost)	The §6 Q4_K_M row, F1 0.897 vs 0.912 baseline
Fine-tune ships a before/after table?	New task and general skills, both columns	Base 0.91 → tuned 0.94, general slice −0.01
Optimizing cost-per-resolved and p95?	A worksheet + a tail number, not a mean	$0.00386/task, p95 430 ms
Serving engine/batch matched to the load?	A benchmark with the workload's shape	Throughput vs TTFT/TPOT at `max_num_seqs=32`

The table's value is that "pass" is not a feeling, it is an artifact with a number attached, and every question's evidence is the same artifact that will be re-run when the next base model drops. If the answer to any row is "no," the change is not ready, regardless of how good the demo looked.

How it breaks / common mistakes

Model engineering fails when a lever is pulled out of order, or pulled without a number:

Mistake	What it looks like	The fix
Fine-tuning on instinct	"The model can't do it" with no eval proving prompt+RAG failed	Point to a measured gap before touching weights
Perplexity as the only check	A Q2 quant that "looks fine" on perplexity but flubs the task	Always run the task eval; perplexity screens, the eval decides
4-bit = 4× faster (assumed)	Expecting a speedup the kernel never delivered	Smaller is reliable; faster depends on fused kernels, measure
Optimizing cost-per-token	"Cheap model" that retries three times	Optimize cost per resolved task, retries included
Optimizing mean latency	The average hides the tail users actually feel	Optimize p95
Training loss as the stop signal	"Loss looks nice" while the eval regressed	Stop when the eval stops improving, not when the loss is pretty
Forgetting to check forgetting	A fine-tune that aced the task and broke everything else	Ship a before/after table on the new task and general skills

Two of these are the most expensive because they are invisible until production. Cost per token hides the retry trap, a model that is cheap per token but succeeds half the time is the most expensive model in the fleet, and only cost-per-resolved-task reveals it. Mean latency hides the tail, the p95 is the number the customer feels on the slowest fifth of requests, and it is the number that decides whether an "interactive" feature is actually interactive. Both are fixed by changing which metric you optimize, not by optimizing harder.

Self-check questions

What is the selection order, and why is fine-tuning last?Answer: Managed API + prototype → prompt/context + eval → RAG → agentic workflow → fine-tune last. Each step is harder to undo and more expensive; fine-tuning is justified by a measured gap the cheaper levers cannot close, not by instinct.
How many GB does a 7B model occupy at FP16, INT8, and INT4 (roughly)?Answer: params × bits ÷ 8: FP16 ≈ 14 GB, INT8 ≈ 7 GB, INT4 ≈ 4 GB (3.5 GB of weights plus scale factors and overhead). Each halving of bits roughly halves bytes.
For a 7B LoRA (r=16, alpha=32, targeting q/k/v/o), how many parameters are you actually training?Answer: About 16.8 M, A (16×4096) + B (4096×16) = 131,072 params per module × 4 modules × 32 layers, roughly 0.24% of 7B, which is why QLoRA fits on one modest GPU.
A model costs $0.002/attempt but succeeds only 60% of the time. What is its cost per resolved task, and what does that teach?Answer: $0.002 × 1/0.6 ≈ $0.0033 per resolved task, more than a "pricier" model that gets it right the first time. The lesson: optimize cost per resolved task, with the success rate as a first-class factor, not cost per token.
Why does a serving benchmark measure p95 and TTFT/TPOT instead of mean latency and total tokens/s alone?Answer: The mean hides the tail (the p95 is what users feel on the slowest fifth), and TTFT (prefill) vs TPOT (decode) stress different parts of the system. Reporting throughput alone would let a "fast" model that stalls on long prompts ship. Performance numbers must also sit next to the eval score, or a throughput win is meaningless.

Where to go next. The eval that gates every lever here is 07-evals-error-analysis.md; the quantization/serving internals that make the byte counts real are 03-llm-core-concepts.md (KV cache) and 08-local-inference-gpu.md (GPU/VRAM math); and the prompt → RAG → agentic steps that come before fine-tuning are 04-prompt-context-engineering.md and 05-rag-graph-engineering.md.

Sources

Andrew Ng, AI Engineering Skills Map and the prompt → RAG → agentic → fine-tune-last selection order (The Batch, 2026): https://www.deeplearning.ai
Zorost Signal, "Cost and latency engineering for LLM systems" (24 JUL 2026): https://zorost.com/llm-cost-latency-engineering · Signals index: https://zorost.com/signals
GGUF / k-quants (llama.cpp): https://github.com/ggml-org/llama.cpp
bitsandbytes (8-bit / NF4): https://github.com/TimDettmers/bitsandbytes
AWQ: https://github.com/mit-han-lab/llm-awq · GPTQ: https://github.com/IST-DASLab/gptq
Hugging Face PEFT (LoRA/QLoRA/DoRA): https://huggingface.co/docs/peft · TRL (SFT/DPO/GRPO): https://huggingface.co/docs/trl
Unsloth: https://github.com/unslothai/unsloth · Axolotl: https://github.com/axolotl-ai-cloud/axolotl
vLLM: https://docs.vllm.ai · SGLang: https://github.com/sgl-project/sglang · TGI: https://github.com/huggingface/text-generation-inference

06, Model Engineering: Quantization, Fine-Tuning & Serving ​

1. The selection order: cheapest lever first ​

2. Quantization: shrink the numbers, keep the skill ​

2.1 Numeric formats ​

2.2 Quantization methods ​

2.3 Acceptable vs avoid ​

2.4 Measuring quality loss ​

2.5 How a quantizer maps floats to integers (worked example) ​

3. Fine-tuning: change the weights, not just the prompt ​

3.1 When it pays vs when it doesn't ​

3.2 The adapters: LoRA, QLoRA, DoRA ​

3.3 SFT dataset design: quality beats quantity ​

3.4 SFT vs DPO vs RLVR ​

3.5 Catastrophic forgetting checks ​

3.6 Frameworks (quick reference) ​

3.7 Distillation ​

3.8 The fine-tuning loop is an eval loop ​

4. Serving: from a model file to an API ​

4.1 Engine comparison ​

4.2 Continuous batching ​

4.3 KV cache memory ​

4.4 Throughput vs latency knobs ​

4.5 OpenAI-compatible APIs ​

4.6 Serving a quantized model with vLLM (worked) ​

5. Cost & latency engineering ​

6. ZoroLogistics example: quantizing the 7B triage model ​

7. Bringing it together: the deployment gate ​

A minimal QLoRA recipe (conceptual) ​

A worked "halve both" pass ​

The five-question gate ​

The selection order, on one diagram ​

Quantization math table: sizes across formats ​

LoRA rank/alpha, with worked numbers ​

The cost-per-task engineering worksheet ​

Serving benchmark methodology ​

Shrink-the-model decision: quantization vs LoRA vs distillation ​

Fine-tuning data: the quality checklist ​

Latency levers, ranked by cost ​

The deployment gate, as a checklist ​

How it breaks / common mistakes ​

Self-check questions ​

Sources ​