3.6 Prefix Caching Strategy

Earlier we said context engineering feels like "dancing in chains" — the chains are prefix caching. Now let's examine the shape of those chains in detail, and how Claude Code dances gracefully within their constraints.

Why Does Prefix Caching Matter?

Every API request requires the server to compute KV Cache for the input (a fundamental operation in the transformer attention mechanism). A complete request input can be 100K-200K tokens — computing that from scratch every time would be unacceptably slow and expensive.

Prefix caching works by having the server remember the KV Cache results from the previous request. On the next request, if the prefix is exactly identical, it reuses the previous computation and only processes the new additions. But there's a hard constraint at the transformer architecture level: the prefix must be byte-for-byte identical to reuse the KV Cache. Not "close enough" — any single byte of change invalidates all KV Cache from that position onward.

The Cache Chain: From System Prompt to Conversation Content

Claude Code doesn't just cache the system prompt — it explicitly sets cache breakpoints at two levels, system and messages, chaining them into a complete cache chain; the tools array, because it renders before system, is covered together by the system breakpoint:

┌──────────────────────────────────────────────────────────────────────┐
│  Request N:                                                          │
│                                                                      │
│  [system ← cache_control] [tools (covered by system)] [history...    │
│   ~~~~~~cache hit~~~~~~   ~~~~~~also hit~~~~~~      msg1 msg2 msg3   │
│                                                     ← cache_control  │
│                                                                      │
│  Request N+1:                                                        │
│                                                                      │
│  [system ← cache_control] [tools (covered by system)] [history...    │
│   ~~~~~~cache hit~~~~~~   ~~~~~~also hit~~~~~~      msg1 msg2 msg3   │
│                                                  ~~~~~~cache hit~~~~~~│
│                                                       msg4 msg5      │
│                                                       ← cache_control│
│                                                       ↑ only this    │
│                                                         needs compute│
└──────────────────────────────────────────────────────────────────────┘

First breakpoint: system prompt. splitSysPromptPrefix() marks cache_control at the static/dynamic boundary of the system prompt, allowing the core instruction portion to be shared across users (see "Splitting Strategy" below). The API's render order is tools → system → messages, so the breakpoint marked on the system block pulls the entire tools array in front of it into the cached prefix — which is why the tool definitions get cached without needing their own breakpoint.

Second breakpoint: message array. addCacheBreakpoints() marks cache_control on the last message. This means all historical messages (from the previous turn and before) are within the cached prefix, and each turn only needs to process newly added messages.

On the tools-array breakpoint: toolToAPISchema retains the ability to mark cache_control on a tool schema (options.cacheControl), but in the snapshot we analyzed, no call on the main query path passes it (claude.ts there only passes deferLoading), and the source comment saying "the tool schema carries the cache marker" describes intent that was never realized. So the breakpoints actually in effect are the two on system and messages, with tools covered by the former. Optional server-side tools (like advisor) sit at the end of the tools array — the design intent was to place a tools breakpoint before them so toggling wouldn't bust the cache; with the tools breakpoint unwired, that protection never took effect in the snapshot.

The two breakpoints chained together achieve full-pipeline caching: in a conversation that's been going for 20 turns, turn 21 only needs to process the latest user message and tool results — all the accumulated system + tools + 20 turns of history hit the cache. This is one of the core reasons Claude Code maintains low-latency responses.

Detail: Cache protection for fire-and-forget requests. Certain secondary requests (like background auxiliary queries) place the cache_control marker on the second-to-last message instead of the last. This way, temporary requests don't write their content into the main cache chain, avoiding pollution of subsequent normal conversation's cache prefix.

Detail: Special handling for assistant messages. cache_control is only marked on the last content block of a message, but skips thinking and redacted_thinking blocks — these blocks have unstable content, and marking them would reduce cache hit rates.

Cache Stability: Four Layers of Defense

The benefits of full-pipeline caching are enormous, but it also means the cost of cache invalidation is extremely high — a single unexpected cache break could force 100K+ tokens of prefix to be fully recomputed. Claude Code has built four layers of defense to maintain cache stability.

Layer 1: System Prompt Splitting — Maximizing Cache Sharing

The core problem: the system prompt contains both universally shared content (core instructions, safety rules, tool descriptions) and user-specific content (CLAUDE.md references, MCP tool instructions, output style preferences). If the entire system prompt can only be cached as a single unit, every user needs their own cache — millions of users means millions of cache entries, with most of the content being completely duplicated.

The solution: splitSysPromptPrefix() (src/utils/api.ts) uses the SYSTEM_PROMPT_DYNAMIC_BOUNDARY marker to split the system prompt at the shared/specific boundary, applying different cache strategies to each part.

But there's a decision tree here, because not all scenarios can use the optimal strategy:

1. Does the user have MCP tools installed? If yes, MCP tool schemas vary by user (different users install different MCP servers). Even if the system prompt text is identical, the tool arrays differ — global caching won't work. All blocks fall back to scope: 'org' (shared within organization).
2. Is global caching available and does the boundary marker exist? If yes, this is the optimal path: static content before the boundary uses scope: 'global' (shared across all users worldwide), dynamic content after the boundary is not cached.
3. Neither condition met? Everything falls back to scope: 'org'.

Scenario	Static Content Cache	Dynamic Content Cache	Sharing Scope
Has MCP tools	org	org	Within organization
No MCP + global cache supported	global	not cached	Worldwide
Fallback	org	org	Within organization

Layer 2: Session-Level Latching — Preventing Mid-Session Flips

Even with splitting done correctly, the cache can still break due to mid-session metadata changes. Two particularly insidious risks exist:

Risk 1: Cache expiration time changes

Server-side caches have an expiration time (TTL): 5 minutes for regular users, 1 hour for internal users and paid subscribers. But the TTL itself is part of the cache key — if a user exceeds their usage quota at turn 10, and the system downgrades them from "1 hour" to "5 minutes," the cache key changes, and all previously accumulated cache is invalidated.

Claude Code's solution: lock cache eligibility at session start, never change it mid-session. setPromptCache1hEligible() evaluates whether the user qualifies for 1-hour TTL on the first API call and caches the result for the rest of the session. Even if the user exceeds their quota mid-session, the TTL won't downgrade.

Risk 2: API request header changes

The Claude API supports several beta features (such as fast-mode, cache-editing, thinking-clear, etc.) enabled via HTTP request headers. These headers also affect the server's cache key — if turn 3's request includes the fast-mode header but turn 4 doesn't, the cache keys differ.

These features may be controlled by feature flags that can toggle server-side at any time. Imagine: a user is coding, a feature flag suddenly disables fast-mode, the next request is missing a header, and 50-70K tokens of cache are instantly invalidated.

Claude Code's solution: once a beta header is first sent, it continues to be sent in all subsequent requests for that session — even if the feature flag that triggered it has been turned off. The source code calls this a "sticky-on latch," applying to four headers: AFK mode, fast mode, cache editing, and thinking clear.

Both types of latching share a common reset point: /clear and /compact commands reset all latch states simultaneously. This makes sense — these commands inherently rebuild the conversation context, so the cache must be rebuilt anyway, and there's nothing left for the latches to protect.

Design trade-off: Latching sacrifices flexibility (can't disable a beta feature mid-session, can't downgrade TTL mid-session) in exchange for cache stability. Given the high cost of invalidating a 100K+ token cache, this is a worthwhile trade-off.

Layer 3: Tool Array and Message Array Arrangement Strategy

Tool arrangement: The design intent is to place optional server-side tools (like advisor) after a tools breakpoint, so enabling or disabling /advisor only changes the small section after it; but as 3.6 explains, the tools breakpoint is unwired in the snapshot, so this protection never actually took effect. MCP tools use the deferred loading (defer_loading) mechanism, not appearing in the tool array until Tool Search is invoked — complementing Layer 1's splitting strategy by minimizing cache differences caused by user-specific tools.

Cached Microcompact's cache awareness: As mentioned in section 3.4, Microcompact has two paths. When the cache is "warm" (not expired), it does not directly modify message content (which would break the entire message prefix cache). Instead, it sends cache_edits instructions telling the server to "delete certain tool_result blocks from the cache." This cleans up old content and frees space without requiring the client to re-upload or the server to reprocess the entire prefix — a sophisticated coordination between message-level caching and the compression mechanism.

Layer 4: Cache Break Detection — The Diagnostic Safety Net

With the first three layers of defense, the cache should be stable. But "should" and "actually is" always have a gap — Claude Code has built a diagnostic system to verify these defenses are truly working.

promptCacheBreakDetection.ts implements a two-phase snapshot comparison:

1. Before API call: Record a snapshot of all state that affects the cache key — system prompt hash, tool schema hash (per-tool granularity), beta header list, TTL settings, etc.
2. After API call: Check cache_read_input_tokens in the response. If it dropped by more than 5% and 2000 tokens compared to last time, it's classified as a cache break

When a break is detected, the system automatically attributes it to one of three causes:

• TTL expiration: Time since the last request exceeded the cache time window (user was away too long)
• Client-side change: By comparing hash diffs, it pinpoints exactly which field changed (it can even identify which specific tool's schema changed)
• Server-side eviction: Everything on the client side is unchanged but the cache still missed — indicating the server proactively evicted the cache

This detection system forms an improvement feedback loop: source code comments reveal that Layer 2's latching mechanisms were developed specifically after the detection system discovered cache breaks caused by header toggling. Detection first, discover the problem, then build the defense — a positive engineering cycle.

Summary: Claude Code's prefix caching is a full-pipeline solution — system and messages carry explicit cache breakpoints, with the tools array cached along with the system breakpoint, chained together into a complete cache pipeline. Four layers of defense (splitting, latching, arrangement strategy, break detection) collectively ensure this cache chain's stability. The end result: no matter how many turns the conversation has reached, each request only needs to process the latest incremental content — everything accumulated before is provided for free by the KV Cache.