08, Local Inference & GPUs: Running Models on Your Own Hardware
Purpose. Run open models on a laptop with confidence, pick them by license and VRAM, operate the local stack (Ollama, llama.cpp, LM Studio, MLX), verify your GPU actually works, and do it all offline when privacy demands it.
Part of AI Engineering Lab · Developed by Zorost Intelligence AI Lab · zorost.com
1. Open vs closed models: and what "open" actually means
- Closed models (e.g. the flagship GPT, Claude, Gemini tiers) are served through an API only. You never see the weights; you rent access. They are often the strongest out of the box, but you cannot self-host, inspect, modify, or guarantee where your data goes.
- Open-weight models publish the trained weights (and usually the architecture and tokenizer) so you can download, run, quantize, and fine-tune them yourself. The catch: open weights are not automatically "open source" in the OSI sense, the license, not the download button, decides what you may legally do.
The practical decision is three questions, in order:
- Do I need to self-host (privacy, air-gap, cost, latency, offline)? If yes, you need an open-weight model.
- Is the license compatible with my use (commercial product? redistribution? fine-tuning?)? Read it, see the table below.
- Does a small-enough model do the task on my hardware? If a 70B is required and I have an 8 GB GPU, the open model is moot.