Program

Optimize local llama.cpp inference for speed.

The benchmark target is a GGUF model served through llama-server. Use the local huggingface-local-models skill to find llama.cpp-compatible repos, choose a quant, confirm exact .gguf filenames, and build portable --hf-repo / --hf-file commands.

Start with:

uv run scripts/inspect_llama_toolchain.py
uv run scripts/resolve_hf_gguf.py --repo <owner/repo> --quant Q4_K_M

Then benchmark:

uv run scripts/benchmark_llama.py \
  --server-cmd "llama-server --hf-repo <owner/repo> --hf-file <file.gguf> -c 4096 --port 8080" \
  --append-tsv research/results.tsv

Improve one variable at a time: quant, context, backend acceleration, thread count, GPU offload, batch sizing, cache type, or server concurrency. Do not claim a speedup until the same benchmark has been run before and after the change.