Skill: Ollama & llama.cpp: Running Models Locally

Part of AI Engineering Lab · Developed by Zorost Intelligence AI Lab · https://zorost.com Skill for: Week 8 (Open Models & Local Inference) · also used in Weeks 9, 10, 17

What it is	The two workhorses of local inference: Ollama (the friendly model manager + API) and llama.cpp (the engine underneath, GGUF quantized models)
Best for	Running open models offline, on a laptop or a homelab box, with zero cloud bills
Requirements	macOS / Linux / Windows; no GPU required (CPU works, GPU is faster)
Cost	Free (open source)

---

What it is

Ollama wraps llama.cpp in a daemon that manages models, exposes an OpenAI-compatible REST API on http://localhost:11434, and makes "download a model" a one-liner. llama.cpp is the raw C/C++ inference engine, more control, more flags, more patience. LM Studio (GUI) and Apple MLX (Apple Silicon-native) sit alongside; see the knowledge base for comparisons (knowledge-base/08).

Verify against live docs. Model tags (qwen2.5:3b, llama3.2:3b, etc.) rotate as publishers ship new releases, and the quantization ladder's exact names shift. Treat every specific tag and quant name here as "correct at time of writing", ollama search <term> and the Hugging Face model card are the source of truth for what exists today.

Install & first run

Ollama

# macOS / Linux / Windows, install from https://ollama.com/download
ollama pull qwen2.5:3b        # ~2 GB download, runs on CPU
ollama run qwen2.5:3b         # interactive chat, /bye to quit
ollama list                   # what you have locally

Useful model tags: qwen2.5:3b, llama3.2:3b, phi4-mini, gemma3:4b, deepseek-r1:7b: small first, measure, then size up. (Model names change often; ollama search <term> shows what's current.)

llama.cpp

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build --config Release -j