5. AI Gateways

AI gateways sit between configured applications and their LLM providers. They can handle routing, rate limiting, cost management, and active context transformation for requests sent through them. Traffic that bypasses the configured endpoint remains outside their telemetry and policy controls.

Edgee

Edgee is a Rust-based AI gateway. Its compression features shipped as Compressor V2 (announced July 2, 2026): three independent layers, each targeting a different part of the request.

Layer	What it compresses	Measured effect (Edgee's own benchmark)
Brevity	Model output (strips narrated planning, "I'll first read the file, then...")	~30% median cost reduction, 6 SWE-bench Lite coding tasks
Tool Surface Reduction (TSR)	MCP tools[] catalog, collapsed to one virtual search tool	~33% token-volume reduction, ~10% cost reduction, 8 MCP tasks
Tool result trimming	Conversation history / tool outputs	~10% cost reduction, 6 coding tasks (carried over and refined from V1)

Source: Edgee Blog, Introducing Compressor V2 | github.com/edgee-ai/edgee

Relationship to RTK: Edgee's own post names RTK as the direct inspiration for the tool-result-trimming layer. Structurally, RTK can only ever cover that one layer: it runs as a local shell hook and has no access to the MCP tool catalog or the model's own output. For a Claude Code user already running RTK, Edgee's brevity layer is the genuinely new capability; TSR overlaps with what Claude Code's native MCP Tool Search already does for free (§ MCP Tool Search), and trimming overlaps with what RTK already does locally.

Reading the "50% combined" claim critically: the three numbers above come from three separate experiments on two different workloads (coding vs. MCP), never measured together on the same sessions. The "50%" in the post's title tracks closest to brevity's raw aggregate (+51.1%, pulled up by one outlier task), relabeled as the combined figure. It is not an end-to-end measurement of all three layers running at once, and the effects are not mechanically additive (less narration also means less history left to trim). The statistical design is genuinely careful: paired per-task comparison, a sign test chosen over a paired t-test because cost differences are heavy-tailed, and a nonce injected into each replicate to defeat prompt-cache contamination between runs. But the sample sizes are small enough to matter: at n=6, the best achievable two-sided sign-test p-value is 0.031, meaning a perfect 6-of-6 result was the only outcome that could clear the conventional 0.05 threshold; one task flipping drops it to 5/6, p=0.22, not significant. The post also never reports SWE-bench Lite's actual metric, resolution rate (the share of issues whose patch still passes tests), only token cost. A cheaper agent that solves fewer tickets is not a net win, and Edgee's own benchmark repository (edgee-ai/compression-lab) confirms it tracks token consumption "rather than task completion rates."

A second, separate set of numbers exists, and it tells a different story. Edgee's documentation (distinct from the blog post) reports production averages across real customer traffic: brevity ~6.5%, tool result trimming ~19%, tool surface reduction ~25% (labeled "in development," i.e. not yet fully shipped), and an aggregate ~20% token-bill reduction across active customers over a rolling 30-day window. These production figures diverge sharply from the controlled benchmark above: brevity's real-world effect (6.5%) sits far below its benchmark median (~30%) or headline aggregate (+51.1%), while trimming's production effect (19%) nearly doubles its benchmark result (~10%). The same page answers the resolution-rate gap with one line, "zero measurable drift on SWE-Bench Verified samples," but gives no sample size, no definition of "measurable," and no confidence interval. It reads as a direct response to the resolution-rate critique, without the statistical rigor the benchmark post itself otherwise demonstrates. Source: Edgee docs, Why Edgee?

Portkey

Portkey is a managed AI gateway centered on unified routing across multiple LLM providers.

Attribute	Details
Source	portkey.ai
Model support	250+ LLMs via unified API
Features	Routing, fallbacks, load balancing, caching, guardrails
Observability	Built-in tracing and cost tracking

Portkey's semantic caching layer is particularly relevant for context optimization: identical or near-identical requests are cached and returned without an LLM call. In applications with repetitive query patterns (helpdesks, code review bots, internal search), cache hit rates can reduce total LLM calls by 30–60%.

Gateway vs. Output Compression: these categories complement each other. RTK/Headroom compress what goes into the context from tool outputs. Gateways compress or route the assembled prompt before it hits the model. Both reduce total token spend, but they intercept at different points in the pipeline.

LiteLLM

LiteLLM is an MIT-licensed Python proxy with Redis-backed caching, virtual keys, and configurable per-team or per-user budget caps. Unlike Edgee or Portkey, it ships no active compression layer of its own: its cost levers are caching and routing, not token-level compression of what gets sent. It can be paired with RTK or lean-ctx, which handle compression, when the goal also includes centralized budget enforcement for routed team traffic. See api-gateway.md for the budget-enforcement boundary.

Semantic Caching as a Library: GPTCache

Portkey's semantic caching (above) and Anthropic's prompt caching (§8) both require an exact or near-exact prefix match. GPTCache (Zilliz, Apache 2.0) targets a different case: it caches the full response, keyed by the embedding of the query, and serves it for any new query above a cosine-similarity threshold, skipping the LLM call entirely rather than reducing what gets sent to it. It's a library to integrate into your own code (not a drop-in gateway), with pluggable vector backends (Milvus, FAISS) and storage backends (Redis, MongoDB); embedding lookup adds roughly 3–8ms. Hit-rate figures of 30 to 70% circulate in vendor and blog material but are not independently verified. Treat them as an order of magnitude, not a measured number, until you benchmark your own traffic.