9.2 Why Context Engineering is Important

Although models are getting faster and can handle larger data scales, we observe that: like humans, LLMs will "wander" or "get confused" at a certain point. Needle-in-a-haystack benchmarks reveal a phenomenon: context rot—as the number of tokens in the context window increases, the model's ability to accurately recall information from the context actually decreases.

Different models may have smoother degradation curves, but this characteristic appears in almost all models. Therefore, context must be viewed as a limited resource with diminishing marginal returns. Just as humans have limited working memory capacity, LLMs also have an "attention budget". Each new token consumes part of this budget, so we need to be more careful about which tokens should be provided to the LLM.

This scarcity is not accidental, but stems from the architectural constraints of LLMs. Transformers allow each token to establish associations with all tokens in the context, theoretically forming (n^2) pairwise attention relationships. As the context length grows, the model's ability to model these pairwise relationships is "stretched thin", naturally creating tension between "context scale" and "attention concentration". In addition, the model's attention patterns come from the training data distribution—short sequences are usually more common than long sequences, so the model has less experience with "full-context dependencies" and fewer specialized parameters.

Techniques such as position encoding interpolation can allow models to "adapt" to sequences longer than during training at inference time, but at the cost of some precision in understanding token positions. Overall, these factors together form a performance gradient rather than a "cliff-like" collapse: models are still powerful in long contexts, but compared to short contexts, their precision in information retrieval and long-range reasoning will decline.

Based on the above reality, conscious context engineering becomes a necessity for building robust agents.

9.2.1 The "Anatomy" of Effective Context

Under the constraint of "limited attention budget", the goal of excellent context engineering is: maximize the probability of obtaining expected results with as few but high signal density tokens as possible. In practice, we recommend engineering around the following components:

• System Prompt: Clear and straightforward language, with information hierarchy at "just right" height. Common pitfalls at two extremes:

  • Over-hardcoding: Writing complex, fragile if-else logic in prompts, with high long-term maintenance costs and fragility.
  • Too vague: Only providing macro goals and generalized guidance, lacking specific signals for expected output or assuming incorrect "shared context". It is recommended to organize prompts into sections (such as <background_information>, <instructions>, tool guidance, output description, etc.), separated by XML/Markdown. Regardless of format, the pursuit is the "minimum necessary information set" that can fully outline expected behavior ("minimum" does not equal "shortest"). First run with the best model on the minimum prompt, then add clear instructions and examples based on failure modes.

• Tools: Tools define the contract between the agent and the information/action space, and must promote efficiency: they must return token-friendly information while encouraging efficient agent behavior. Tools should:

  • Have single responsibilities with low overlap, clear interface semantics;
  • Be robust to errors;
  • Have clear and unambiguous parameter descriptions, fully leveraging the model's strengths in expression and reasoning. A common failure mode is "bloated tool sets": fuzzy functional boundaries, making the decision of "which tool to use" itself ambiguous. If human engineers can't tell which tool to use, don't expect agents to do better. Carefully identifying a "Minimum Viable Tool Set (MVTS)" can often significantly improve stability and maintainability in long-term interactions.

• Few-shot Examples: Always recommend providing examples, but don't recommend stuffing "all boundary conditions" into prompts. Please carefully select a set of diverse and typical examples that directly portray "expected behavior". For LLMs, good examples are worth a thousand words.

The overall guiding principle is: sufficient but compact information. As shown in Figure 9.2, this is dynamic retrieval entering runtime.

Figure 9.2 Calibrating the system prompt

9.2.2 Context Retrieval and Agentic Search

A concise definition: Agent = LLM autonomously calling tools in a loop. As the capabilities of underlying models increase, the autonomy level of agents can be improved: they can more independently explore complex problem spaces and recover from errors.

Engineering practice is gradually transitioning from "one-time retrieval before inference (embedding retrieval)" to "Just-in-time (JIT) context". The latter no longer preloads all relevant data, but maintains lightweight references (file paths, storage queries, URLs, etc.), dynamically loading required data through tools at runtime. This allows the model to write targeted queries, cache necessary results, and analyze large volumes of data with commands like <code>head</code>/<code>tail</code>—without stuffing entire data blocks into context at once. Its cognitive pattern is closer to humans: we don't memorize all information, but use external indexes like file systems, inboxes, bookmarks to extract on demand.

In addition to storage efficiency, metadata of references itself can help refine behavior: directory hierarchy, naming conventions, timestamps, etc., all implicitly convey "purpose and timeliness". For example, <code>tests/test_utils.py</code> and <code>src/core/test_utils.py</code> have different semantic implications.

Allowing agents to autonomously navigate and retrieve also enables progressive disclosure: each interaction step generates new context, which in turn guides the next decision—file size hints at complexity, naming hints at purpose, timestamps hint at relevance. Agents can build understanding layer by layer, keeping only the "currently necessary subset" in working memory, and using "note-taking" for supplementary persistence, thereby maintaining focus rather than being "dragged down by comprehensiveness".

The trade-off is: runtime exploration is often slower than pre-computed retrieval, and requires "opinionated" engineering design to ensure the model has the right tools and heuristics. Without guidance, agents may misuse tools, chase dead ends, or miss key information, causing context waste.

In many scenarios, a hybrid strategy is more effective: preload a small amount of "high-value" context to ensure speed, then allow agents to continue autonomous exploration on demand. The choice of boundaries depends on task dynamics and timeliness requirements. In engineering, you can preload files like "project convention descriptions (such as README/guides)", while providing primitives like <code>glob</code>, <code>grep</code>, allowing agents to retrieve specific files just-in-time, thereby bypassing the sunk costs of outdated indexes and complex syntax trees.

9.2.3 Context Engineering for Long-Horizon Tasks

Long-horizon tasks require agents to maintain coherence, context consistency, and goal orientation in action sequences that exceed the context window. For example, large codebase migrations, systematic research spanning hours. Expecting to infinitely increase the context window cannot cure the problems of "context pollution" and relevance degradation, so engineering methods directly facing these constraints are needed: Compaction, Structured note-taking, and Sub-agent architectures.

• Compaction

  • Definition: When a conversation approaches the context limit, perform high-fidelity summarization and restart a new context window with the summary to maintain long-range coherence.
  • Practice: Have the model compress and retain architectural decisions, unresolved defects, implementation details, discarding repetitive tool outputs and noise; the new window carries the compressed summary + a few recent highly relevant artifacts (such as "recently accessed files").
  • Tuning suggestions: First optimize recall (ensure no key information is missed), then optimize precision (remove redundant content); a safe "light-touch" compression is to clean up "tool calls and results in deep history".

• Structured note-taking

  • Definition: Also called "agent memory". Agents write key information to persistent storage outside the context at fixed frequencies, pulling it back on demand in subsequent stages.
  • Value: Maintain persistent state and dependencies with extremely low context overhead. For example, maintaining TODO lists, project NOTES.md, indexes of key conclusions/dependencies/blockers, maintaining progress and consistency across dozens of tool calls and multiple context resets.
  • Note: Equally effective in non-coding scenarios (such as long-term strategic tasks, goal management and statistical counting in games/simulations). Combined with <code>MemoryTool</code> from Chapter 8, file-based/vector-based external memory can be easily implemented and retrieved at runtime.

• Sub-agent architectures

  • Idea: The main agent is responsible for high-level planning and synthesis, while multiple specialized sub-agents each dig deep, call tools, and explore in "clean context windows", finally only returning condensed summaries (typically 1,000–2,000 tokens).
  • Benefits: Achieve separation of concerns. Complex search contexts remain internal to sub-agents, while the main agent focuses on integration and reasoning; suitable for complex research/analysis tasks requiring parallel exploration.
  • Experience: Public multi-agent research systems show that this pattern has significant advantages over single-agent baselines in complex research tasks.

Method trade-offs can follow these rules of thumb:

• Compaction: Suitable for tasks requiring long conversation continuity, emphasizing context "relay".
• Structured note-taking: Suitable for iterative development and research with milestones/phased results.
• Sub-agent architectures: Suitable for complex research and analysis that can benefit from parallel exploration.

Even as model capabilities continue to improve, "maintaining coherence and focus in long interactions" remains a core challenge in building robust agents. Careful and systematic context engineering will maintain its key value in the long term.