Sizing Local Hardware with llmfit

Before buying anything, check what a given memory budget can actually run. llmfit (MIT, brew install llmfit) detects your hardware and scores its model database for fit, speed, and quality.

llmfit system                                    # detect current hardware
llmfit --memory 96G --ram 128G fit --json         # simulate a different memory budget
llmfit --memory 96G --ram 128G info "meta-llama/Llama-3.3-70B-Instruct"

Two limits to know before trusting its output:

--memory overrides GPU VRAM, --ram overrides system RAM. They are not interchangeable. On a discrete Nvidia GPU, model weights normally reside in VRAM. CPU offload can keep part of the model or MoE experts in system RAM, but every transfer or CPU-computed expert adds a performance cost. The result depends on PCIe bandwidth, host-memory bandwidth, model architecture, and the offload strategy; there is no universal 5 tokens/sec ceiling. On Apple Silicon and AMD's Ryzen AI unified-memory chips, the two numbers should be set equal, because there is only one physical memory pool.

The tool cannot switch inference backend. It detects the backend of the machine it runs on (Metal, CUDA, ROCm) and keeps using that backend's speed model even when you override memory size to simulate different hardware. Run it on an Nvidia machine for Nvidia numbers, an Apple Silicon machine for Apple numbers. Simulating an RTX PRO 6000 from a MacBook gives you a correct memory-capacity answer (what fits) and a wrong throughput answer (tokens/sec), because the tool is still computing speed from the Mac's Metal roofline model, not from CUDA.