llama.cpp Inference Autoresearch

This project is a small multi-agent setup for optimizing local llama.cpp inference speed. It is intentionally simpler than the pre-training Autolab control plane and separate from the post-training benchmark.

Shape

• .opencode/agent/inference-lab.md Primary coordinator.
• .opencode/agent/optimizer.md Read-only speed optimizer that proposes one narrow next change.
• .opencode/agent/benchmarker.md Local benchmark runner for llama-server endpoints.
• .agents/skills/huggingface-local-models/ Local copy of the Hugging Face local-models skill for GGUF discovery, quant choice, and llama.cpp command construction.
• scripts/resolve_hf_gguf.py Resolves exact GGUF filenames from the Hugging Face model tree API.
• scripts/benchmark_llama.py Benchmarks a running or launched OpenAI-compatible llama-server endpoint.
• research/results.tsv Append-only speed benchmark ledger.

Quick Start

From this directory:

uv sync
uv run scripts/inspect_llama_toolchain.py

Resolve a GGUF file:

uv run scripts/resolve_hf_gguf.py --repo <owner/repo> --quant Q4_K_M

Benchmark an already-running server:

uv run scripts/benchmark_llama.py \
  --base-url http://127.0.0.1:8080/v1 \
  --runs 3 \
  --append-tsv research/results.tsv

Or let the benchmark helper launch and stop the server for one run:

uv run scripts/benchmark_llama.py \
  --server-cmd "llama-server --hf-repo <owner/repo> --hf-file <file.gguf> -c 4096 --port 8080" \
  --base-url http://127.0.0.1:8080/v1 \
  --runs 3 \
  --append-tsv research/results.tsv

OpenCode

Start OpenCode from inference/:

opencode

Use the inference-lab primary agent. The normal loop is:

1. benchmarker measures a baseline.
2. optimizer proposes one narrow speed change.
3. benchmarker repeats the same benchmark.
4. Keep the fastest reproducible command in the result ledger.

For a longer kickoff prompt:

uv run scripts/print_opencode_kickoff.py

Secondary Agents

Native examples are also checked in for Claude Code, Codex, and Pi:

• Claude Code: claude --agent inference-lab
• Codex: codex
• Pi: pi, then /inference-lab "<owner/repo> or local endpoint on :8080" 3

Guide files:

• docs/claude-subagents-guide.md
• docs/codex-subagents-guide.md
• docs/pi-subagents-guide.md

Metrics

The default benchmark records wall-clock completion tokens per second from the OpenAI-compatible chat endpoint. Treat the numbers as comparable only when the prompt, max_tokens, model, quant, context, and server command are controlled.

For raw kernel-level performance, run llama.cpp's own llama-bench separately and record the command and result in the notes column.