7. Use Prompt Caching
Prompt caching improves latency and cost when requests share an exact static prefix.
7.1 Compare Two Cache-Keyed Requests
This cell places stable BrightCart policy text at the beginning of the input, sends the same request twice with a prompt_cache_key, and compares token metadata. Inspect cached_input_tokens on the second response when the endpoint returns cache details.
Note: PROMPT_CACHE_RETENTION is selected from the active MODEL_ID. It uses 24h for openai.gpt-5.5 and later models, and in_memory for openai.gpt-5.4.
python
from __future__ import annotations
base_support_policy = [
"BrightCart support policy:",
"1. Be empathetic, concise, and specific about the customer's order.",
"2. Do not promise refunds, credits, or delivery dates unless the policy context supports it.",
"3. For damaged-item replacements, check replacement status before offering concessions.",
"4. If a replacement delay exceeds 48 hours, offer expedited replacement or a 15% concession subject to agent approval.",
]
policy_reference_paragraph = (
"Expanded cacheable policy context: BrightCart agents should identify the customer, order ID, replacement status, "
"carrier scan age, promised delivery window, item category, prior concessions, and supervisor approval needs before "
"drafting a customer-facing answer. The assistant should preserve a calm tone, avoid unsupported promises, separate "
"confirmed facts from assumptions, recommend one clear next action, and document why any escalation, expedited "
"replacement, or concession is appropriate. Repeated policy context like this is intentionally stable across many "
"requests so prompt caching can reuse the prefix when the same cache key is supplied."
)
expanded_policy_context = "\n".join(
f"Policy reference paragraph {idx + 1}: {policy_reference_paragraph}"
for idx in range(32)
)
stable_support_policy = "\n".join(base_support_policy + [expanded_policy_context])
cache_input = [
{"role": "system", "content": stable_support_policy},
{"role": "user", "content": "Draft a two-sentence agent reply for Maya Chen about delayed replacement order ORDER-8831."},
]
estimated_cache_input_words = len(json.dumps(cache_input).split())
require(estimated_cache_input_words > 2048, f"Prompt-cache input should be over 2048 words; found {estimated_cache_input_words}.")
cache_payload = {
"model": MODEL_ID,
"input": cache_input,
"prompt_cache_key": "brightcart-support-policy-v1",
"prompt_cache_retention": PROMPT_CACHE_RETENTION,
"max_output_tokens": 1024,
"store": False,
}
print_request_shape(cache_payload)
print_labeled_json("Prompt-cache input size", {"estimated_input_words": estimated_cache_input_words, "target_minimum_tokens": 2048})
try:
cache_response_1 = create_response(**cache_payload)
cache_response_2 = create_response(**cache_payload)
cache_summary_1 = summarize_response(cache_response_1)
cache_summary_2 = summarize_response(cache_response_2)
cache_comparison = pd.DataFrame([
{
"request": "first",
"input_tokens": cache_summary_1.get("input_tokens"),
"cached_input_tokens": cache_summary_1.get("cached_input_tokens"),
"output_tokens": cache_summary_1.get("output_tokens"),
"total_tokens": cache_summary_1.get("total_tokens"),
},
{
"request": "second",
"input_tokens": cache_summary_2.get("input_tokens"),
"cached_input_tokens": cache_summary_2.get("cached_input_tokens"),
"output_tokens": cache_summary_2.get("output_tokens"),
"total_tokens": cache_summary_2.get("total_tokens"),
},
])
record_check("Prompt caching", "pass" if cache_summary_2.get("cached_input_tokens") is not None else "warn", {"first": cache_summary_1, "second": cache_summary_2})
cache_reply = output_text(cache_response_2).strip()
record_response("Prompt-cache token comparison", "table", cache_comparison)
record_response("Cached support-policy reply", "text", cache_reply)
print_labeled_text("Result", cache_reply)
print_labeled_json("First request summary", cache_summary_1)
print_labeled_json("Second request summary", cache_summary_2)
print_label("Response summary")
display_wrapped_table(cache_comparison, max_col_width_px=260)
print_key_takeaway('cached_input_tokens is the metadata field to inspect for prompt-cache reuse.')
except Exception as exc:
handle_example_error("Prompt caching", exc)Request shape
| field | value |
|---|---|
| model | openai.gpt-5.4 |
| max_output_tokens | 1024 |
| store | False |
| prompt_cache_key | brightcart-support-policy-v1 |
| prompt_cache_retention | in_memory |
| input | 2 item(s): system: BrightCart support policy: 1. Be empathetic, concise, and specific about the customer's order. 2. Do not promise refunds, credits, or delivery dates unless the policy context supports it. 3. For damaged-item replacements...; user: Draft a two-sentence agent reply for Maya Chen about delayed replacement order ORDER-8831. |
Prompt-cache input size
\{ "estimated_input_words": 3016, "target_minimum_tokens": 2048 \}
Result
Hi Maya, I’m sorry your replacement order ORDER-8831 is delayed. I’m checking the latest replacement and carrier status now so I can confirm the best next step for you without making you wait longer than necessary.
First request summary
\{ "id": "resp_3w6r6ipbqa5z2max35awv3i23i5sjmfa33zzw3vhpgqxcvchhkgq", "model": "openai.gpt-5.4", "status": "completed", "output_item_types": [ "message" ], "input_tokens": 3970, "output_tokens": 66, "total_tokens": 4036, "cached_input_tokens": 0, "reasoning_output_tokens": 0, "service_tier": "default" \}
Second request summary
\{ "id": "resp_zzjeqttoswdjdwwl56xpolvly23w4h2n5dtsdoddgxqwfbkn7npq", "model": "openai.gpt-5.4", "status": "completed", "output_item_types": [ "message" ], "input_tokens": 3970, "output_tokens": 48, "total_tokens": 4018, "cached_input_tokens": 0, "reasoning_output_tokens": 0, "service_tier": "default" \}
Response summary
| request | input_tokens | cached_input_tokens | output_tokens | total_tokens |
|---|---|---|---|---|
| first | 3970 | 0 | 66 | 4036 |
| second | 3970 | 0 | 48 | 4018 |
Key takeaway: cached_input_tokens is the metadata field to inspect for prompt-cache reuse.