AI 原生开放教材

外观

AI Engineering Lab（24 周自学课程）系统课程·英文原文 ↗原件 ↓仓库 ↗

Week 10, Fine-Tuning: LoRA, SFT & DPO ​

Part of AI Engineering Lab · Week 10 of 24 · Section: Model Engineering · Category: Fine-tuning 🎯 Use case: Fine-tune a small open model on ZoroLogistics support tone and bill-of-lading entity extraction.

The problem ​

ZoroLogistics' base model does almost everything right. Prompted for bill-of-lading extraction, it drifts from the strict JSON schema; asked to triage a support ticket, it routes "the invoice has the wrong weight" to documents instead of billing often enough that ops notices. These are not knowledge gaps that RAG can fill, the answer is not in a retrieved passage; it is a behavior: a fixed output shape and a house routing style that are expensive to express in a prompt and impossible to express in retrieved text. When a measured eval shows prompting + RAG + tooling still miss a specific, high-volume target, that gap is the justification for touching the weights.

The problem is that the instinct is to reach for fine-tuning first, and full fine-tuning is expensive, slow, and destructive, updating all 1.5B (or 7B) parameters costs GPU-hours and can overwrite the general skills the model already has. Without this week's two tools, LoRA/QLoRA (train ~0.05% of the parameters) and a before/after eval (prove the specialist gain is worth the general cost), you either overpay for the fix or ship an adapter that secretly regressed everything else. This week you learn to close a measured gap on a free-Colab budget and to read the deploy/no-deploy decision as a table of numbers, not a feeling.

Objectives ​

  • [ ] By Friday you can state Ng's selection order (prompt → RAG → agentic → fine-tune last) and give one concrete reason fine-tuning is justified only after the cheaper levers fail an eval.
  • [ ] By Friday you can build a ~300-example SFT dataset in chat-template format from data.bol_samples() extraction and data.support_tickets() triage tasks.
  • [ ] By Friday you can train a LoRA adapter with TRL's SFTTrainer on a small base model and read the loss curve.
  • [ ] By Friday you can create ~100 preference pairs, run a small DPO step, and produce a before/after eval table that includes a catastrophic-forgetting check on a generic task.

Day-by-day plan ​

DayStudyRunShipTime
MonThe selection order and when fine-tuning pays (§1, §3.1 of reference/knowledge-base/06-model-engineering.md)State, in one sentence, the measured gap that justifies your fine-tuneA written justification2 h
TueSFT dataset design, chat templates, quality-vs-quantityBuild the ~300-example dataset from bol_samples() + support_tickets()sft_train.jsonl + stats2 h
WedLoRA/QLoRA mechanics, rank/alpha, the QLoRA recipeTrain the adapter with SFTTrainer; capture the loss curvelora_adapter/ + loss curve2 h
ThuDPO and preference data; catastrophic forgettingBuild ~100 pairs, run a DPO step, grade before/afterBefore/after eval table2 h
FriRead the trade: gain vs forgettingWrite the deploy/no-deploy decisionweek-10-before-after.md2 h

Concepts ​

The week's thesis, straight from reference/knowledge-base/06-model-engineering.md §1, is that fine-tuning is the last lever, justified by a measured gap, not the first instinct. The order is a managed-API prototype, then prompting/context engineering with an eval, then RAG, then an agentic workflow, and only then the weights. Each step is harder to undo than the last: a prompt change deploys in seconds; a fine-tune costs GPU-hours, can regress other skills, and must be re-run when the base model improves. The blunt rule: *if you cannot point to an eval score that prompting + RAG

  • tooling fail to reach, you are not ready to train.*

When fine-tuning pays (and when it doesn't) ​

Fine-tuning is the right tool for style, tone, strict output format, and niche domain vocabulary, behaviors that are expensive to express in a prompt and impossible to express in retrieved text, and for compressing a large model's behavior into a small specialist. It does not pay for new knowledge (that is RAG's job: weights are the wrong place to store facts that change, a model that "knows" last quarter's prices is already wrong this quarter), for one-off prompt-fixable behavior, or when you lack clean data.

Fine-tune paysFine-tune does not pay
Style/tone ("write like our support team")New or changing facts (use RAG)
Strict output format (fixed JSON schema)One-off, prompt-fixable behavior
Domain vocabulary / house taxonomyToo little high-quality data
Compression: teach a small specialist to mimic a large teacherA gap you have not measured first

LoRA, QLoRA, DoRA: training a fraction of the weights ​

Parameter-efficient fine-tuning (PEFT) freezes the base weights and trains only a small set of new parameters. LoRA (Low-Rank Adaptation) factors the weight update as ΔW = B·A where A and B are small matrices of rank r, injected into the attention/MLP projections. QLoRA stacks LoRA on a 4-bit NF4 base (your Week 9 tool) so a 7 to 8B fine-tune fits on a free Colab GPU or a 16 GB laptop. DoRA splits each weight into magnitude + direction and adapts only the direction, often matching full fine-tuning better at the same rank. See reference/knowledge-base/06-model-engineering.md §3.2.

Worked example 1: LoRA parameter-count math. The notebook trains Qwen/Qwen2.5-1.5B-Instruct (hidden size 1536, MLP intermediate 8960) with r=16, alpha=32, targeting q/k/v/o projections (1536→1536 each) and gate/up/down (1536→8960 each). For each module of shape (in, out), LoRA adds r × (in + out) trainable parameters. The four attention projections cost 4 × 16 × 3072 = 196,608; the three MLP projections cost 3 × 16 × 10,496 = 503,808. Total: 700,416 ≈ 0.70M trainable parameters, against ~1.54B frozen: about 0.05% of the model, squarely in the "0.1 to 1%" PEFT range. That is why the adapter saves to a few MB and why training needs a fraction of the memory. Rank is the number of "degrees of freedom" you give the update (r=4-16 for style/format, r=32-64 for harder shifts), and alpha ≈ 2·r is a common default for the update's strength.

Task typeRank rAlphaNotes
Style / tone / format only816Minimal capacity; least overfit risk
Domain behavior (extraction, classification)16 to 3232 to 64The most common starting point
Hard task shift (reasoning, tool use)64128Only with plenty of data; watch overfitting

SFT: imitation learning on clean data ​

Supervised fine-tuning (SFT) is imitation learning on (instruction → answer) pairs. The single rule to internalize: a few hundred clean, diverse, on-task examples beat tens of thousands of noisy ones, a small model internalizes whatever pattern dominates the data, so garbage in is literally what you teach. Match the model's native chat template (a messages array of role/content), hold out a separate eval set, and cover the failure modes from your error analysis rather than just the happy path. See reference/knowledge-base/06-model-engineering.md §3.3.

SFT → DPO → RLVR ​

SFT teaches the shape; DPO (Direct Preference Optimization) sharpens the preference. Classic RLHF needs a separately trained reward model and a finicky PPO loop; DPO folds the preference signal directly into a supervised-style objective on (prompt, chosen, rejected) pairs, simpler, more stable, and often as good. RLVR goes further, using a verifiable reward (a test passes or it does not) instead of a learned one. The practical sequence is SFT first, then DPO, then RLVR only with a verifiable reward and a reason to push further.

ObjectiveOptimizesDataUse
SFTImitation of good answers(instruction, answer)Style, format, task behavior, first step
DPOChosen > rejected, directly(prompt, chosen, rejected)Preference shaping; no reward model
RLVRA checkable rewardVerifiable tasks + reward fnFrontier reasoning/coding gains

Frameworks and budget ​

The default Python stack is TRL + PEFT (SFTTrainer, DPOTrainer, LoraConfig). Unsloth is a drop-in speed/memory layer (2 to 5× faster, ~50 to 70% less VRAM) over the same TRL API, and Axolotl turns a fine-tune into a YAML recipe for reproducible, shareable runs. The budget reality is the point: a 7 to 8B QLoRA SFT run fits a free Colab T4 or a 16 GB MacBook, so fine-tuning does not require a datacenter, it requires a clean dataset.

The loop is an eval loop: and forgetting is the cost ​

"Training happened" is not a result; the before/after number on the same golden set is, which is why reference/knowledge-base/07-evals-error-analysis.md is this week's co-reading. And because training weights can overwrite old skills, every fine-tune carries a mandatory catastrophic-forgetting check: hold out a generic task the model used to do well, grade it before and after, and refuse to ship an adapter that traded your specialist skill for general competence.

Worked example 2: the before/after decision. On the held-out set, the base model scores extraction field-F1 0.60, triage accuracy 0.72, and a generic five-question check 0.95. After the DPO adapter: extraction 0.88, triage 0.91, generic 0.93. The notebook's final number is net = (extraction delta) + (triage delta) − (forgetting cost) = (0.88−0.60) + (0.91−0.72) − (0.95−0.93) = 0.28 + 0.19 − 0.02 = +0.45. Positive net → the specialist gains earned the tiny generic regression → deploy. Had the forgetting cost exceeded the gains (e.g., generic 0.95 → 0.80), the honest answer is don't deploy, and stating that with numbers is a passing result. The two-column, two-row table (base vs fine-tuned × new-task vs general) is the minimum artifact that lets anyone, including future you, see the actual trade.

mermaid
%%{init:{"theme":"base","fontFamily":"Helvetica Neue,Helvetica,Arial,sans-serif","flowchart":{"curve":"basis","padding":14,"nodeSpacing":45,"rankSpacing":55},"themeVariables":{"fontSize":"15px","background":"#FFFFFF","primaryColor":"#EEF2F7","primaryTextColor":"#14213D","primaryBorderColor":"#14213D","secondaryColor":"#FFF1E3","secondaryTextColor":"#14213D","tertiaryColor":"#E7F4F1","tertiaryTextColor":"#14213D","lineColor":"#64748B","textColor":"#14213D","edgeLabelBackground":"#FFFFFF","clusterBkg":"#F5F8FC","clusterBorder":"#94A3B8","nodeBorder":"#14213D","mainBkg":"#EEF2F7","titleColor":"#14213D"}}}%%
flowchart TD
    A["Eval shows a gap on a high-volume task"] --> B{"Prompt + RAG + agentic already tried?"}
    B -- "no" --> C["Fix with the cheaper lever first"]
    B -- "yes" --> D["Build a clean SFT dataset (chat template, held-out eval)"]
    D --> E["Train LoRA/QLoRA (r=16, ~0.05% of params)"]
    E --> F["SFT then DPO to sharpen preference"]
    F --> G{"Before/after: specialist gain > forgetting cost?"}
    G -- "no" --> H["Don't deploy; improve the dataset (better data > bigger rank)"]
    H --> D
    G -- "yes" --> I["Deploy the adapter; add failures back to the eval set"]

How it breaks ​

Fine-tuning prevents "prompt it harder forever" thrash, and causes these failures if misused:

  • Forgetting the general skills. The adapter overfits the new task and regresses the generic check, the exact cost the before/after table exists to surface.
  • Memorizing the eval. Train and grade on the same examples and the number measures memorization, not capability; a held-out set is non-negotiable.
  • Template mismatch. Feeding the model a chat format it was not trained on silently degrades the run; match the tokenizer's native template.
  • Garbage dominance. A noisy dataset teaches the dominant pattern, clean, diverse, on-task beats many noisy rows.
  • Rank/alpha misuse. Rank too high (or too many epochs) overfits: train score up, eval score down. Reduce r or alpha, do not add epochs.
  • Storing facts in weights. A fine-tune that "knows" volatile policy data goes stale; retrieve facts, fine-tune behavior.

Notebook walkthrough ​

Two notebooks carry the week. notebooks/01-sft-dataset-and-lora-training.ipynb builds the dataset and trains the adapter. After the seeded setup (cell 2), cell 6 builds 150 extraction examples from data.bol_samples(), each a messages array whose assistant turn is the ground-truth JSON of bol["fields"], and cell 8 builds 150 triage examples from data.support_tickets(). Cell 10 combines them, writes sft_train.jsonl, and prints mean words/example. Cell 11 is the pre-train cost estimate (QLoRA r=16 → ~0.75 GB weights, 300 examples, ~38 optimizer steps, fits a T4). Cell 15 sets the LoRA config (r=16, alpha=32, targets q/k/v/o/gate/up/down); cell 17 loads the base, QLoRA on CUDA (4-bit NF4) or plain FP16 LoRA on MPS, and cell 18 wraps it, printing print_trainable_parameters() (expect ~0.70M trainable = ~0.05%). Cell 22 runs SFTTrainer (batch 2 × grad-accum 4, LR 2e-4, 1 epoch) and captures the loss history; cell 24 prints the step-wise curve and saves the adapter; cell 26 smoke-tests a fresh extraction. Cell 28 prints the final SFT_FINAL_LOSS and SFT_EXAMPLES. A clean run shows loss falling then flattening; a flatline from step 0 means LR too high or data not tokenizing.

notebooks/02-dpo-and-before-after-evals.ipynb measures the trade. Cell 6 holds out the eval set (20 BoLs, 30 tickets, 5 generic questions). Cell 8 builds ~100 preference pairs, chosen = correct, rejected = a corrupted weight or a plausible-but-wrong category. Cells 12 to 14 grade the before state (base model) with three deterministic helpers; cell 17 runs the DPO step (beta=0.1, r=8) wrapped in a try/except so a config issue degrades gracefully; cell 19 grades the after state on the same set. Cell 21 assembles the before/after table with a delta column, and cell 23 prints the final BEFORE_AFTER_NET_GAIN = extraction delta + triage delta − forgetting cost. The shape to look for: specialist deltas positive and materially larger than any generic regression, if the generic drop is bigger than the gains, the number is negative and the honest decision is "don't deploy."

The use case (Friday) ​

Deliverable: a before/after eval table plus a one-paragraph deploy/no-deploy decision for the fine-tuned model. The table grades the base model and the adapter on the same held-out set, BoL field extraction, ticket triage, and a generic task (the forgetting check), with a metric per row, so the decision is a reading of numbers rather than a feeling.

Acceptance gate (Zorost-style): a stranger can open the table and see the exact delta the fine-tune bought, "triage accuracy 0.72 → 0.91, extraction field-F1 0.60 → 0.88, generic task 0.95 → 0.93", and you can show them what changed: the specific tickets the adapter now routes correctly and the generic answers it started to fumble. If the adapter does not beat the base by enough to justify its forgetting cost, the honest answer is don't deploy, and saying so with the numbers is a passing result.

Stretch variant: swap plain LoRA for DoRA (use_dora=True in LoraConfig) or switch to a 4-bit QLoRA base, keep everything else fixed, and compare the before/after table against your plain-LoRA run. Write one sentence on whether the extra mechanism was worth it for your task, not in general.

Common pitfalls ​

PitfallFix
Fine-tuning before measuring a gapRun prompt + RAG + agentic with an eval first; fine-tune only on a measured miss
No held-out eval setSplit the dataset; train and grade on different examples
Ignoring catastrophic forgettingAlways add a generic-task before/after check; gate on net gain
Feeding the wrong chat templateUse the model's native messages format (or the tokenizer's template)
"More data is better" with noisy dataA few hundred clean, diverse examples beat thousands of noisy ones
Overfitting (train up, eval down)Reduce rank/alpha or epochs, never add epochs to fix overfit
Reading the loss curve as successLoss proves training happened; only the eval proves it helped
Storing volatile facts in weightsRetrieve facts (RAG); fine-tune behavior only

Glossary ​

  • Fine-tuning: continuing training on task data to change a model's behavior, format, or style.
  • PEFT: parameter-efficient fine-tuning: freeze the base, train a small set of new parameters.
  • LoRA: low-rank adaptation: ΔW = B·A of rank r injected into attention/MLP layers.
  • QLoRA: LoRA on top of a 4-bit NF4 base via bitsandbytes; fits big fine-tunes on one card.
  • DoRA: weight-decomposed LoRA that adapts direction separately from magnitude.
  • SFT: supervised fine-tuning: imitation learning on (instruction → answer) pairs.
  • DPO: direct preference optimization: trains on (prompt, chosen, rejected) with no reward model.
  • Chat template: the model's native message format, usually a messages array of role/content.
  • Catastrophic forgetting: the regression of general skills when fine-tuning on a narrow task.
  • Rank r / alpha: the capacity (r) and strength (alpha) knobs of a LoRA update.
  • Before/after eval: grading the base and the adapter on the same held-out set, then reading the delta.

Self-check (quiz) ​

Take quiz.md, 10 questions, pass with 8/10. Record the score in your tracker Notes.

Exercises ​

The four graded exercises live in exercises.md: Easy (run the SFT notebook, record loss + dataset size), Standard (rebuild the dataset with a new seed + 50 hand-written examples), Stretch (swap LoRA for DoRA or QLoRA), Portfolio (commit the adapter + before/after table + deploy decision). Hints are in the same file.

Sources ​