AI 原生开放教材

外观

AI Engineering Lab（24 周自学课程）系统课程·英文原文 ↗原件 ↓仓库 ↗

03: Coding-Agent Harnesses, LLM Routers & Local Model Stacks ​

AI Engineering Lab research note · synthesized from official docs and primary sources · Aug 2026 Audience: engineers standing up agentic coding workflows and local inference. Everything below is written in our own words; no third-party training text is reproduced. Sources are listed at the end.

This note covers four layers of the stack a working AI engineer touches every day: (A) the harnesses and IDEs that turn an LLM into a coding agent, (B) the local inference runtimes that let you run those models yourself, (C) quantization (how to make models fit), and (D) fine-tuning (when to change the weights instead of the prompt).

A) Harnesses & AI Coding Tools ​

A harness is the software between the model and the environment: it gives an LLM tools (shell, file editing, search), a loop (plan → act → verify → repeat), persistence (sessions, logs), and coordination primitives (subagents, skills, workflows). The model supplies reasoning; the harness supplies agency.

A.1 DeepSeek Harness (DSH) ​

  • What it is. An open-source agent harness (not an editor and not a model) from DeepSeek AI. Its central design claim is "everything is a plugin." Models, tools, skills, sessions, sandboxes, storage, loops, scheduling, and even the UI are plugin bundles composed by configuration on top of a plugin kernel called Cordis.
  • Who builds it. DeepSeek AI. Repository: github.com/deepseek-ai/deepseek-harness (npm package @deepseek-ai/dsh, MIT license).
  • Status. Developer preview, explicitly "iterating rapidly" with compatibility-breaking changes expected.
  • Key features.
    • Runtime modes (picked per session): Standard (full toolset, file editing, shell, file/web search, skills, planning, goals, subagents, workflows), Code (tools exposed through a code-mode SDK so the model can orchestrate multi-step operations in one TypeScript program), Minimal (a two-tool agent with persistent bash + str_replace_editor, useful for benchmarking models in a bare environment), and Creator (runtime inspection and preset authoring).
    • Skills: reusable, task-specific instruction packs (e.g., a "skill" for a domain workflow) loaded on demand.
    • Goals: persisted, same-session completion objectives that drive automatic continuation rounds.
    • Subagents & forks: delegate work to background children that either start fresh or inherit the conversation; children stay continuable.
    • Ralph loops: fresh-agent iterative execution where each round starts a new child with no inherited context and uses the shared workspace as durable memory.
    • Workflows: scripted fan-out across many subagents (phases, structured results).
    • Traceability: everything the model sees is recorded in an append-only session log (system prompts, reasoning, tool calls, subagent scheduling). A Trajectory view lets you inspect by source; resume, fork, search, and replay operate on the same event stream.
    • MCP client, job management (background jobs), todo tracking, plan mode, token metering, and session compaction.
  • Install / run. Requires Node.js. Quick start launches the Web UI:
    sh
    npx @deepseek-ai/dsh web     # Web UI at http://127.0.0.1:3080 by default
    From source:
    sh
    git clone https://github.com/deepseek-ai/deepseek-harness.git
cd deepseek-harness && pnpm install && pnpm run build
pnpm dsh web
    Other entry modes: dsh --profile <name> (boot a named profile), dsh --profile headless "job" (run one fresh persisted session, print the final answer, exit), and dsh plugin --profile <name> <pnpm args> (manage a profile's plugins).
  • Configuration. A profile is an ordered stack of plugin bundles plus the user's own patch layer (cordis.patch.yml). Compose profiles without editing the source; --dump-config / --dump-default-config inspect the composed tree. "Agent presets" (e.g., code, minimal, standard, cordis) bundle a base toolset.
  • Authentication. DSH itself is MIT/free; you supply your own model credentials (API keys / provider config) through the Web UI or config. There is no per-seat license.
  • Pricing. Free, open source (MIT). Cost = your model provider.
  • Practical tips.
    1. Start with npx @deepseek-ai/dsh web, zero install surface; go to source only when you want to write plugins.
    2. Use skills for repeatable domain procedures and goals for long-running, multi-round objectives rather than one giant prompt.
    3. Read the Trajectory view when a run goes wrong, the append-only log is the fastest way to see why the model did something.
    4. Treat it as moving-target software: pin expectations loosely because it is a preview that breaks compatibility.

A.2 OpenCode (SST) ​

  • What it is. An open-source, terminal-first AI coding agent with an interactive TUI, plus desktop, web, and IDE-extension surfaces. It is deliberately model-agnostic and provider-agnostic.
  • Who builds it. SST (the team behind the SST framework / OpenNext). Site: opencode.ai.
  • Key features. Full TUI (session list, slash commands, diff review), an AGENTS.md convention for project memory/instructions, per-project opencode.json config, LSP integration, MCP server support, a permission/policy system, themes/keybinds, "share" for collaboration, and a client/server split so the core runs headless and any surface attaches.
  • Install / run.
    sh
    curl -fsSL https://opencode.ai/install | bash
# or: npm i -g opencode-ai
opencode
  • Model / provider configuration. Defined in opencode.json (project root) or ~/.config/opencode/opencode.json. A provider map holds named providers (baseURL, apiKey, models), and model sets the default (e.g., "anthropic/claude-sonnet-4", "openrouter/... ", or a local "ollama/llama3.2"). AGENTS.md (project) and ~/.config/opencode/AGENTS.md (global) are auto-loaded as persistent instructions.
  • Authentication. opencode auth login stores credentials; API keys can also come from environment variables per provider. Since it targets any OpenAI-compatible endpoint, it works with OpenRouter, Ollama, LM Studio, and local servers.
  • Typical workflow. opencode → describe the change → agent edits files and shows a diff → review/accept → it iterates until tests or your approval pass.
  • Pricing. Free and open source (MIT); you pay only your model provider.
  • Practical tips.
    1. Put a project AGENTS.md in the repo (build commands, conventions, "don't touch these files"), it is the single highest-leverage configuration.
    2. Keep a ~/.config/opencode/AGENTS.md for cross-project defaults (style, testing policy).
    3. Point a provider at Ollama/LM Studio to test prompts for free before paying a hosted model.

A.3 Cursor (Anysphere) ​

  • What it is. An AI-native code editor, a VS Code fork, built around an agent that edits many files at once.
  • Who builds it. Anysphere.
  • Key features. Tab (inline multi-line completion), Chat, and Agent (formerly Composer) for autonomous multi-file edits; Rules (.cursor/rules, legacy .cursorrules) for project instructions; MCP support; a CLI and SDK; and Cursor Router with "Auto" modes, Cost / Balance / Intelligence, that route each request to a model balancing cost, quality, and reliability.
  • Install / authenticate. Download the app from cursor.com, sign in with a Cursor account; manage teams from the dashboard. Extensions (VS Code ecosystem) mostly carry over.
  • Typical workflow. Open a repo → ⌘K for targeted edits, Tab to accept completions → open the Agent panel with a task and let it plan and edit across files → review diffs → loop on tests.
  • Pricing (as of Aug 2026, per Cursor's pricing doc). Hobby (free, limited), Pro $20/mo, Pro+ $60/mo, Ultra $200/mo; Teams Standard $40/user/mo, Teams Premium $120/user/mo; Enterprise (pooled usage, SCIM, audit logs) via sales. Auto (Cost mode) has fixed rates, $1.25/1M input, $6.00/1M output, $0.25/1M cache read, while Balance/Intelligence bill at the routed model's rate. Plans also include a monthly "Other Models" usage pool.
  • Practical tips.
    1. Commit Rules to the repo so every teammate's agent follows the same conventions.
    2. Prefer Agent mode over single edits for cross-file refactors; keep the task narrow and review the full diff.
    3. Watch the usage pool, third-party/model routing consumes it fastest; switch the Auto mode (Cost vs Intelligence) to match the task.

A.4 Claude Code (Anthropic) ​

  • What it is. Anthropic's official CLI coding agent that runs in your terminal and can operate headless in CI.
  • Who builds it. Anthropic.
  • Key features. CLAUDE.md memory files (repo + user level), subagents defined under .claude/agents/, hooks (PreToolUse, PostToolUse, Notification, Stop, etc.) for policy and automation, MCP integration (claude mcp add), a permission/sandbox system, and headless mode (claude -p "…") with --output-format stream-json for scripting and pipelines. It runs directly in the repo, giving it real filesystem context.
  • Install / authenticate.
    sh
    npm install -g @anthropic-ai/claude-code   # then run `claude`
# or the native installer: curl -fsSL https://claude.ai/install.sh | bash
    Authenticate either with a Claude account (Pro/Max subscription) or with ANTHROPIC_API_KEY for pay-as-you-go API billing.
  • Typical workflow. claude in a repo → describe the task → it plans, runs shell commands (with permission prompts), edits files → you review diffs → it verifies with tests → claude -p for non-interactive runs.
  • Pricing. Included with Claude subscriptions (Pro ~$20/mo; Max tiers ~$100 and ~$200/mo for higher usage), Team seats per user (~$25 to 60 depending on tier), Enterprise custom, or billed per-token through the API. (Verify current numbers at Anthropic's pricing page; tiers shift.)
  • Practical tips.
    1. Maintain a repo CLAUDE.md (test commands, build steps, architecture notes), it is read every session and compounds.
    2. Use hooks to auto-run linters/tests after edits or to block dangerous commands rather than prompting for each one.
    3. Use headless mode + --output-format stream-json to embed the agent in scripts and CI; combine with --allowedTools to bound what it can touch.

A.5 Other notable tools (one paragraph each) ​

  • Windsurf. An AI-first editor (originating from the Codeium team) whose flagship is Cascade, an agent that maintains deep repo-wide context and previews changes; it pairs fast Supercomplete-style completion with agentic multi-file edits, Rules, and Previews. Freemium; paid tiers unlock more agent "flow action" credits. Good fit for developers who want agentic editing with an IDE feel lighter than a full fork.
  • GitHub Copilot (agent mode). GitHub/Microsoft's assistant, now with agent mode in VS Code that iterates autonomously on multi-file tasks, plus Copilot Chat, inline completion, a CLI, and Copilot coding agent for issues/PRs. Tightly integrated with GitHub (PRs, Actions). Free tier + Pro (~$10/mo), Business/Enterprise tiers. Default choice when you live inside GitHub/VS Code.
  • Aider. An open-source CLI pair-programmer (pip install aider-chat) that works directly on a git repo, proposes edits, and auto-commits; supports a huge model list (hosted or local via Ollama/LM Studio) and an architect/editor split. Free (you pay the API). Best for git-native, terminal-only, model-flexible workflows.
  • Warp. A modern Rust terminal with built-in AI ("Warp AI") and an Agentic Mode that reads your terminal context and executes multi-step commands; the terminal itself is open source, AI features are subscription. Best when you want agent help inside the shell rather than a separate coding agent.
  • Amazon Q Developer. AWS's AI coding assistant (IDE extension + CLI) with an agent mode; free tier and a Pro tier (~$19/mo, and included with some AWS offerings). It shines in AWS-heavy environments (IaC, CDK, AWS API knowledge), less so as a general cross-provider agent.

A.6 OpenRouter (unified LLM API): and the "9Router" question ​

  • What it is. A unified API and router over hundreds of models from many providers. You use one key and one OpenAI-compatible endpoint (https://openrouter.ai/api/v1/chat/completions); OpenRouter routes your request to a provider and adds automatic failover.

  • Who builds it. OpenRouter (openrouter.ai), an independent service.

  • How routing works. Models are addressed by slug (provider/model, e.g. anthropic/claude-sonnet-4, deepseek/deepseek-chat, meta-llama/llama-3.3-70b-instruct). Beyond plain slugs, OpenRouter adds:

    • Provider routing: choose providers, or let OpenRouter sort by price/latency/throughput.
    • Automatic fallbacks / model fallbacks: if a provider errors or times out, the request reroutes.
    • Routers: higher-level helpers like the Auto Router (pick a good model for your prompt), Pareto Router (pick a coding model by minimum coding score), Fusion Router (multi-model deliberation as a slug), and the Free Models Router.
    • Variants as suffixes, :free (rate-limited free access), :extended, :thinking, :online, :nitro, :exacto.
    • BYOK: bring your own provider API keys so you pay the provider directly.

  • :free models. Append :free to a model id (e.g. meta-llama/llama-3.2-3b-instruct:free) for zero-cost access. Free variants carry stricter rate limits and availability than paid versions (limits are per-model; check the FAQ/rate-limits doc, commonly on the order of tens of requests per minute or a daily cap).

  • Credits & pricing. Pay-as-you-go per token: top up a credit balance, or use BYOK for discounted/provider-direct billing. OpenRouter adds a small markup for its routing/fallback service; exact per-model prices are listed per model on openrouter.ai/models.

  • Authentication. Create an API key at openrouter.ai/keys; send Authorization: Bearer <KEY>.

  • Practical tips.

    1. For local/harness experiments, drop an :free model into OpenCode/DSH config to test plumbing at zero cost, then swap to a paid slug.
    2. Enable fallbacks so a provider outage doesn't kill an agent loop.
    3. Use BYOK if you already pay for Anthropic/OpenAI keys, it removes OpenRouter's token markup.

  • Is "9Router" real? We found no legitimate LLM-routing product named "9Router." Searching npm surfaces only several 9router*-suffixed packages flagged as suspicious (typosquat-style, no credible project behind them), not a real tool to adopt. The phrase is almost certainly a mishearing of "OpenRouter", which we cover in full above. Treat "9Router" = OpenRouter unless a specific, citable product is named.

A.7 Comparison: coding-agent harnesses & tools ​

ToolMakerSurfaceModel choiceConfig/memoryAuthPricing
DeepSeek Harness (DSH)DeepSeekWeb GUI + CLI + headlessAny (BYO key)Profiles, cordis.patch.yml, presetsBYO provider keyFree (MIT)
OpenCodeSSTTUI + desktop/web/IDEAny OpenAI-compat + nativeopencode.json, AGENTS.mdopencode auth / envFree (MIT)
CursorAnysphereIDE (VS Code fork)Curated + Cursor Router.cursor/rulesCursor accountFree → $20 to $200/mo; Teams $40 to 120
Claude CodeAnthropicTerminal CLIClaude familyCLAUDE.md, .claude/Claude account or API keySubscription or API
WindsurfWindsurf teamIDECuratedRulesAccountFreemium credits
GitHub CopilotGitHub/MicrosoftVS Code / CLI / webCurated (OpenAI-based)copilot-instructions.mdGitHub accountFree → Pro ~$10/mo
AiderOpen sourceTerminal CLIAny (huge list).aider.conf.yml, repo conventionsBYO keyFree (pay API)
WarpWarpTerminal + AICurated + BYOn/aAccountFreemium
Amazon Q DeveloperAWSIDE + CLICurated (AWS)n/aAWS Builder ID/IAMFree + Pro ~$19/mo
OpenRouterOpenRouterAPI (no UI)100s via slugs/variantsn/aAPI key, BYOKPer-token credits

B) Local Inference Stacks ​

Running models locally gives you privacy, zero marginal token cost, and full control, at the cost of VRAM and throughput. The ecosystem splits into runtime engines (Ollama, llama.cpp, MLX) and serving engines (vLLM, SGLang, TGI).

B.1 Ollama ​

  • What / who. A friendly wrapper around llama.cpp that packages models as portable artifacts and exposes a simple CLI, REST API, and OpenAI-compatible endpoint. Built by the Ollama team (ollama.com).
  • Install.
    sh
    curl -fsSL https://ollama.com/install.sh | sh     # macOS/Linux; Windows has an installer
ollama pull llama3.2            # or a sized/quantized tag: llama3.2:7b, llama3.2:7b-q4_K_M
ollama run llama3.2             # interactive chat
ollama list                     # installed models
  • Modelfiles. A Modelfile is a declarative recipe: FROM <base> (a model or GGUF), SYSTEM prompt, PARAMETER (temperature, num_ctx, top_p, stop, etc.), TEMPLATE (chat template), and ADAPTER (LoRA). Build with ollama create mymodel -f Modelfile.
  • REST API. Served at http://localhost:11434 (/api/generate, /api/chat, /api/pull, /api/tags, …). Set OLLAMA_HOST to bind elsewhere and OLLAMA_MODELS to relocate storage.
  • OpenAI compatibility. http://localhost:11434/v1 speaks the OpenAI /chat/completions protocol, so any OpenAI SDK can point at it.
  • Tips. ① Pin quantized tags (:7b-q4_K_M) for predictable VRAM. ② Use OLLAMA_NUM_PARALLEL/num_ctx to trade concurrency and context against memory. ③ Treat it as the simplest path to local models; move to llama.cpp/vLLM when you need CLI-level flags or high throughput.

B.2 llama.cpp ​

  • What / who. The original C/C++ inference engine (now under the ggml-org org) that runs GGUF-format models on CPU and GPU with no Python dependency. It is the substrate under Ollama and many others.
  • GGUF. A self-contained, single-file model format bundling weights + metadata (tokenizer, chat template, quantization info). Convert from Hugging Face with convert_hf_to_gguf.py, quantize with llama-quantize.
  • Key binaries.
    • llama-cli: one-shot completion/interactive.
    • llama-server: OpenAI-compatible HTTP server (what you'd put behind an agent harness).
    • llama-quantize: convert a model to a smaller k-quant.
    • llama-perplexity, llama-bench, evaluation and benchmarking.
  • Common options. -ngl N (layers offloaded to GPU, set high/all for full GPU), -c/--ctx-size (context), -n (max tokens to generate), -t (threads), --temp, -fa/--flash-attn, and --jinja for chat templates.
  • Backends. CUDA (NVIDIA), Metal (Apple), ROCm/HIP (AMD), Vulkan, and SYCL, selected at build time (cmake flags such as -DGGML_CUDA=ON, -DGGML_METAL=ON).
  • Tips. ① Build with your GPU backend enabled or you silently fall back to CPU. ② Use llama-bench to compare quant/backend choices on your hardware before committing. ③ GGUF quants are the standard "drop-in" artifact for local stacks, learn the Q4_K_M vs Q8_0 tradeoff (see §C).

B.3 LM Studio ​

  • What / who. A desktop GUI for discovering, downloading, and running local models, with a built-in chat UI and a local OpenAI-compatible server. Made by the LM Studio team (lmstudio.ai).
  • Key features. Browse/search Hugging Face and one-click download of GGUF models; choose a runtime (MLX on Apple Silicon, CUDA, or Vulkan); run an in-app chat; expose http://localhost:1234/v1 (OpenAI-compatible) for external tools.
  • Fit. Best for interactive local experimentation and for quickly serving a model to a coding agent (OpenCode/DSH/Aider) without touching a CLI.
  • Tips. ① On Apple Silicon prefer the MLX runtime for speed. ② Watch the "loaded into memory" indicator, the GUI makes VRAM pressure visible. ③ Use it as the discovery layer, then graduate to llama.cpp/vLLM for scripted serving.

B.4 Apple MLX ​

  • What / who. Apple's machine-learning framework for Apple Silicon, exposing a NumPy-like API with unified memory (model, KV cache, and app share one memory pool, no host↔device copies). Repo: ml-explore/mlx.
  • mlx-lm toolkit. mlx_lm.generate, mlx_lm.convert (import Hugging Face models), mlx_lm.server (OpenAI-compatible server), mlx_lm.lora (LoRA/QLoRA fine-tuning), and mlx_lm.fuse.
  • Fit. The fastest local path on M-series Macs; memory bandwidth of unified RAM often beats a comparable GPU for large quants.
  • Tips. ① Prefer 8-bit/4-bit (-q --q-bits 4) to fit big models in unified memory. ② Use mlx_lm.server to feed an OpenAI-compatible client. ③ MLX is research/single-node oriented; for multi-GPU serving use vLLM/SGLang.

B.5 GPU setup: NVIDIA CUDA vs AMD ROCm vs Apple MPS ​

  • NVIDIA (CUDA). The default path. You need: a driver, the CUDA toolkit, and cuDNN (for training frameworks); for PyTorch, install a CUDA-matched wheel:
    sh
    pip install torch --index-url https://download.pytorch.org/whl/cu124   # pick the CUDA build matching your driver
    Verify hardware with nvidia-smi (shows GPU, driver, CUDA version, VRAM) and the software stack with:
    python
    import torch
torch.cuda.is_available()      # True when the build finds the driver
torch.cuda.get_device_name(0)  # e.g. "NVIDIA RTX 4090"
    Rule of thumb: driver version ≥ CUDA toolkit version (the driver is backward-compatible).
  • AMD (ROCm). AMD's CUDA-equivalent stack for Instinct/Radeon GPUs; PyTorch ships ROCm wheels (pip install torch --index-url https://download.pytorch.org/whl/rocm6.x). Check with rocm-smi and torch.cuda.is_available() (ROCm presents as the CUDA device abstraction via HIP). Historically less smooth than CUDA on consumer cards, verify your exact GPU is supported before buying.
  • Apple Silicon (Metal/MPS). No driver install; PyTorch uses the Metal Performance Shaders backend:
    python
    torch.backends.mps.is_available()   # True on M-series
device = torch.device("mps")
    MPS is great for inference and modest fine-tuning; some ops fall back to CPU. MLX (§B.4) is often faster on Apple hardware.
  • CUDA vs ROCm vs MPS summary. CUDA = broadest support + best tooling; ROCm = AMD's answer, strongest on datacenter Instinct; MPS/MLX = zero-setup on Macs, unified memory is the differentiator.

B.6 VRAM planning (rule-of-thumb table) ​

Rough memory per model size (weights only; add ~10 to 20% for activations/KV cache and always leave headroom for the OS and other processes). Rule: FP16 ≈ 2 bytes/param · INT8 ≈ 1 byte/param · INT4 ≈ 0.5 bytes/param → at 4-bit you need roughly 1 GB per billion parameters.

Model sizeFP168-bit (INT8)4-bit (INT4/GGUF Q4)Typical GPU to fit
7B~14 GB~7 GB~4 GB8 GB card (4-bit) or 16 GB (FP16)
8B~16 GB~8 GB~4.5 GB8 GB (4-bit) / 16 GB (FP16)
13B~26 GB~13 GB~7 GB12 to 16 GB (4-bit) / 24 GB (FP16)
34B~68 GB~34 GB~17 GB24 GB (4-bit)
70B~140 GB~70 GB~35 GB2×24 GB (4-bit) or 48 to 80 GB
405B+~810 GB~405 GB~200 GBMulti-GPU datacenter only

Notes: 8-bit here ≈ GGUF Q8_0 / bitsandbytes 8-bit; 4-bit ≈ GGUF Q4_K_M / bitsandbytes NF4. Apple unified memory follows the same table (a 128 GB M-series Mac runs a 70B at 4-bit comfortably).

B.7 Serving engines (vLLM, SGLang, TGI) ​

  • vLLM. The throughput workhorse. PagedAttention manages the KV cache in fixed-size pages (near-zero fragmentation), and continuous batching admits new requests token-by-token instead of waiting for whole batches. Ships an OpenAI-compatible server (vllm serve <model>). Best default for high-QPS production serving of many concurrent requests.
  • SGLang. Optimized for radix/prefix caching (RadixAttention shares cached prefixes across requests, huge wins for shared system prompts and agent loops), plus fast structured/constrained decoding. python -m sglang.launch_server. Best when many requests share a long common prefix or you need fast structured outputs.
  • TGI (Text Generation Inference). Hugging Face's production server (Rust + Python). Polished defaults, strong quantization support (bitsandbytes/GPTQ/AWQ/FP8), tensor parallelism, and easy Docker deployment (ghcr.io/huggingface/text-generation-inference). Best when you want HF-native, ops-friendly serving with conservative, well-tested features.
  • When to use which. Default vLLM for general high-throughput serving; SGLang for prefix-heavy workloads and structured decoding; TGI for HF-ecosystem shops wanting a batteries-included, containerized server.
  • Throughput/latency knobs (all engines). gpu_memory_utilization (VRAM share, ~0.9), max_num_batched_tokens / max_num_seqs (batch size, bigger = higher throughput, higher latency), tensor_parallel_size (split across GPUs), max_model_len (cap context to save KV memory), and quantization (awq/gptq/fp8). Raise batch for throughput; lower it (or add speculative decoding) for latency.

B.8 Comparison: serving engines ​

EngineKey mechanismStrengthsOpenAI-compat serverBest fit
vLLMPagedAttention + continuous batchingHighest throughput, huge ecosystemYes (vllm serve)General production serving
SGLangRadixAttention (prefix cache) + structured decodingFast shared-prefix & JSON/tool decodingYesAgent loops, repeated system prompts
TGIRust/Python, polished defaultsOps-friendly, broad quant support, DockerYesHF-native, containerized deploys

C) Quantization ​

Quantization lowers numerical precision to shrink memory and speed inference at a small accuracy cost.

C.1 Precision formats ​

FormatBits/paramNotes
FP1616Standard half precision; ~2× less memory than FP32 with negligible quality loss for inference.
BF1616Same range as FP32, fewer mantissa bits; more stable for training (no overflow), now the training default.
FP88E4M3/E5M2 formats; used for inference and some training on Hopper+ (Transformer Engine).
INT88~2× smaller than FP16; needs scale factors (per-tensor/channel).
INT44~4× smaller than FP16; the aggressive but widely-acceptable chat/inference default.

C.2 GGUF k-quants ​

GGUF "k-quants" are llama.cpp's practical quant tiers (lower number = smaller/faster but lossier): Q2_K (aggressive), Q3_K_*, Q4_K_M (the community sweet spot, good quality/VRAM ratio), Q5_K_M, Q6_K, and Q8_0 (near-FP16 quality). Recommendation: start with Q4_K_M; step up to Q5_K_M/Q6_K when quality matters and VRAM allows; avoid Q2_K unless memory is desperate.

C.3 Weight-quantization methods ​

  • bitsandbytes. Runtime quantization library: 8-bit (LLM.int8()) and 4-bit NF4 (normal-float-4, a data-aware format); the backbone of QLoRA. Easy drop-in via load_in_4bit=True.
  • AWQ (Activation-aware Weight Quantization). Protects the small fraction of weights that matter most (measured by activation magnitude) and scales the rest. Often better quality than GPTQ at 4-bit with no calibration data needed.
  • GPTQ. Layer-wise quantization using second-order (Hessian) information from a calibration dataset. High quality but needs calibration data and is slower to produce.

C.4 Accuracy & when quantization is (not) acceptable ​

  • Perplexity ladder (general trend). FP16/BF16 ≈ FP32 for inference; 8-bit ≈ negligible loss; 4-bit k-quants ≈ small, usually imperceptible loss for chat/summarization/RAG; 2 to 3-bit = noticeable degradation.
  • Acceptable for: conversation, summarization, classification, RAG, coding assistance at the harness level, where a tiny word-choice drift doesn't break correctness.
  • Be cautious / avoid heavy quantization for: precise arithmetic or math, long-horizon reasoning, low-resource languages, and exact code generation where a single token error breaks compilation. Always eval on your task (perplexity ≠ task accuracy) before shipping a quant.

D) Fine-Tuning Stacks ​

D.1 Parameter-efficient methods (PEFT) ​

  • LoRA (Low-Rank Adaptation). Freezes the base weights and trains low-rank matrices (ΔW = BA, rank r) injected into attention/MLP layers. Trains ~0.1 to 1% of parameters; adapters are small, swappable, and mergeable.
  • QLoRA. LoRA on top of a 4-bit (NF4) base model (bitsandbytes), with double quantization and paged optimizers, fine-tune a 70B on a single 24 GB GPU.
  • DoRA (Weight-Decomposed LoRA). Splits each weight into magnitude + direction and adapts direction with LoRA; often matches full fine-tuning better than plain LoRA at the same rank.

D.2 Training objectives ​

  • SFT (Supervised Fine-Tuning). Imitation learning on (instruction → answer) pairs. Teaches style, format, and task behavior; the standard first step.
  • DPO (Direct Preference Optimization). Trains directly on (chosen, rejected) preference pairs without a reward model, simpler and more stable than RLHF/PPO. Good for alignment and preference shaping.
  • RLVR (RL with Verifiable Rewards). Reinforcement learning where the reward is a checkable signal (unit tests pass, math answer correct, code compiles) rather than a learned reward model. The current frontier method for reasoning/coding gains.

D.3 Frameworks ​

  • Hugging Face TRL + PEFT. The default Python stack: SFTTrainer, DPOTrainer, GRPOTrainer (RLVR), and PEFT's LoraConfig/get_peft_model. Works across Transformers models and accelerator backends.
  • Unsloth. A drop-in speed/memory layer for fine-tuning (2 to 5× faster, ~50 to 70% less VRAM) with native TRL integration; popular for LoRA/QLoRA on Llama/Mistral/Qwen families. pip install unsloth.
  • Axolotl. A YAML-config-driven fine-tuning orchestrator over TRL/PEFT, declare model, dataset, LoRA/QLoRA/DoRA config, and it wires the run; ideal for reproducible, shareable training recipes.
  • Distillation. Train a small student model to mimic a larger teacher (logit or hidden-state matching). Different goal from LoRA: you shrink the model, not just the adapter. Useful for deploying a compact specialist after a large teacher proves a task.

D.4 Dataset formats ​

  • Chat-template format. A messages array (role/content) matching the model's chat template, the most portable today:
    json
    {"messages": [{"role": "system", "content": "..."},
              {"role": "user", "content": "..."},
              {"role": "assistant", "content": "..."}]}
  • ShareGPT format. A conversations array with from/value (system/human/gpt), commonly used with Axolotl. Convert between the two before mixing tooling.
  • Instruction/Alpaca format. Flat instruction/input/output triples (legacy but still common in older datasets).
  • Preference format (DPO). prompt + chosen + rejected. RLVR adds a reward/verifier column (or a reward function/unit-test harness).

D.5 When to fine-tune vs prompt/RAG (Ng's selection order) ​

Andrew Ng's practical default for application work, try each cheaper lever before training weights:

  1. Managed model API + a prototype.
  2. Context and prompting, with an eval (few-shot examples, system prompt, structured output).
  3. RAG if the answer must come from your data/knowledge.
  4. Agentic workflow if the job is multi-step with tools.
  5. Fine-tune (or train) only when the eval shows the base model + context cannot get there.

In short: fine-tuning is the last lever, justified by a measured gap, not the first instinct. Revisit it only when prompting + RAG + tooling fail your eval on a specific, high-volume task (e.g., a strict output format, a niche domain style, or hard-to-prompt reasoning).

Sources ​

Harnesses & tools

Local inference & serving

Quantization & fine-tuning

Note on recency: pricing and feature tiers (Cursor, Claude Code, Windsurf, Copilot) change frequently and were captured Aug 2026 from the linked official pages; re-check the vendor's pricing page before quoting in public material.