4.4 Reflection

In the ReAct and Plan-and-Solve paradigms we have already implemented, once the agent completes a task, its workflow ends. However, the initial answers they generate, whether action trajectories or final results, may contain errors or have room for improvement. The core idea of the Reflection mechanism is to introduce a post-hoc self-correction loop for the agent, enabling it to review its work, discover deficiencies, and iteratively optimize, just like humans do.

4.4.1 Core Idea of Reflection Mechanism

The inspiration for the Reflection mechanism comes from the human learning process: we proofread after completing a first draft and verify after solving a math problem. This idea is embodied in multiple studies, such as the Reflexion framework proposed by Shinn, Noah in 2023^[3]. Its core workflow can be summarized as a concise three-step loop: Execute -> Reflect -> Refine.

1. Execution: First, the agent attempts to complete the task using familiar methods (such as ReAct or Plan-and-Solve), generating a preliminary solution or action trajectory. This can be seen as a "first draft."
2. Reflection: Next, the agent enters the reflection phase. It calls an independent large language model instance, or one with special prompts, to play the role of a "reviewer." This "reviewer" examines the "first draft" generated in the first step and evaluates it from multiple dimensions, such as:
   • Factual Errors: Is there content that contradicts common sense or known facts?
   • Logical Flaws: Are there inconsistencies or contradictions in the reasoning process?
   • Efficiency Issues: Is there a more direct, more concise path to complete the task?
   • Missing Information: Are some key constraints or aspects of the problem overlooked? Based on the evaluation, it generates structured Feedback, pointing out specific problems and improvement suggestions.
3. Refinement: Finally, the agent uses the "first draft" and "feedback" as new context, calls the large language model again, and asks it to revise the first draft based on the feedback content, generating a more complete "revised draft."

As shown in Figure 4.3, this loop can be repeated multiple times until the reflection phase no longer finds new problems or reaches a preset iteration limit. We can formally express this iterative optimization process. Assuming $O_i$ is the output produced by the $i$-th iteration ($O_0$ is the initial output), the reflection model $\pi_{\text{reflect}}$ generates feedback $F_i$ for $O_i$: $$ F_i = \pi_{\text{reflect}}(\text{Task}, O_i) $$ Subsequently, the refinement model $\pi_{\text{refine}}$ combines the original task, the previous version's output, and feedback to generate a new version's output $O_{i+1}$: $$ O_{i+1} = \pi_{\text{refine}}(\text{Task}, O_i, F_i) $$

Figure 4.3 Execute-Reflect-Refine Iterative Loop in Reflection Mechanism

Compared to the previous two paradigms, the value of Reflection lies in:

• It provides the agent with an internal error correction loop, making it no longer completely dependent on external tool feedback (ReAct's Observation), thus able to correct higher-level logical and strategic errors.
• It transforms one-time task execution into a continuous optimization process, significantly improving the final success rate and answer quality for complex tasks.
• It builds a temporary "short-term memory" for the agent. The entire "execute-reflect-refine" trajectory forms a valuable experience record; the agent not only knows the final answer but also remembers how it iterated from a flawed first draft to the final version. Furthermore, this memory system can also be multimodal, allowing the agent to reflect on and revise outputs beyond text (such as code, images, etc.), laying the foundation for building more powerful multimodal agents.

4.4.2 Case Setting and Memory Module Design

To embody the Reflection mechanism in practice, we will introduce a memory management mechanism, because reflection usually corresponds to information storage and retrieval. If the context is long enough, having the "reviewer" directly obtain all information and then reflect often introduces a lot of redundant information. In this practical step, we mainly complete code generation and iterative optimization.

The goal task for this step is: "Write a Python function to find all prime numbers between 1 and n."

This task is an excellent scenario for testing the Reflection mechanism:

1. Clear Optimization Path Exists: The code initially generated by the large language model is likely a simple but inefficient recursive implementation.
2. Clear Reflection Points: Through reflection, problems like "excessively high time complexity" or "redundant calculations" can be discovered.
3. Clear Optimization Direction: Based on feedback, it can be optimized to a more efficient iterative version or a version using the memoization pattern.

The core of Reflection lies in iteration, and the prerequisite for iteration is the ability to remember previous attempts and received feedback. Therefore, a "short-term memory" module is essential for implementing this paradigm. This memory module will be responsible for storing the complete trajectory of each "execute-reflect" loop.

from typing import List, Dict, Any, Optional

class Memory:
    """
    A simple short-term memory module for storing the agent's action and reflection trajectory.
    """

    def __init__(self):
        """
        Initialize an empty list to store all records.
        """
        self.records: List[Dict[str, Any]] = []

    def add_record(self, record_type: str, content: str):
        """
        Add a new record to memory.

        Parameters:
        - record_type (str): Type of record ('execution' or 'reflection').
        - content (str): Specific content of the record (e.g., generated code or reflection feedback).
        """
        record = {"type": record_type, "content": content}
        self.records.append(record)
        print(f"📝 Memory updated, added a '{record_type}' record.")

    def get_trajectory(self) -> str:
        """
        Format all memory records into a coherent string text for building prompts.
        """
        trajectory_parts = []
        for record in self.records:
            if record['type'] == 'execution':
                trajectory_parts.append(f"--- Previous Attempt (Code) ---\n{record['content']}")
            elif record['type'] == 'reflection':
                trajectory_parts.append(f"--- Reviewer Feedback ---\n{record['content']}")

        return "\n\n".join(trajectory_parts)

    def get_last_execution(self) -> Optional[str]:
        """
        Get the most recent execution result (e.g., the latest generated code).
        Returns None if it doesn't exist.
        """
        for record in reversed(self.records):
            if record['type'] == 'execution':
                return record['content']
        return None

The design of this Memory class is relatively concise, with the main structure as follows:

• Uses a list records to store each action and reflection in order.
• The add_record method is responsible for adding new entries to memory.
• The get_trajectory method is the core; it "serializes" the memory trajectory into a text segment that can be directly inserted into subsequent prompts, providing complete context for the model's reflection and optimization.
• get_last_execution makes it convenient to obtain the latest "first draft" for reflection.

4.4.3 Coding Implementation of Reflection Agent

With the Memory module as a foundation, we can now proceed to build the core logic of ReflectionAgent. The entire agent's workflow will revolve around the "execute-reflect-refine" loop we discussed earlier and guide the large language model to play different roles through carefully designed prompts.

(1) Prompt Design

Unlike previous paradigms, the Reflection mechanism requires multiple prompts for different roles to work together.

1. Initial Execution Prompt: This is the prompt for the agent's first attempt to solve the problem, with relatively straightforward content, only requiring the model to complete the specified task.

INITIAL_PROMPT_TEMPLATE = """
You are a senior Python programmer. Please write a Python function according to the following requirements.
Your code must include a complete function signature, docstring, and follow PEP 8 coding standards.

Requirement: {task}

Please output the code directly without any additional explanations.
"""

2. Reflection Prompt: This prompt is the soul of the Reflection mechanism. It instructs the model to play the role of a "code reviewer," critically analyze the code generated in the previous round, and provide specific, actionable feedback.

REFLECT_PROMPT_TEMPLATE = """
You are an extremely strict code review expert and senior algorithm engineer with ultimate requirements for code performance.
Your task is to review the following Python code and focus on finding its main bottlenecks in <strong>algorithm efficiency</strong>.

# Original Task:
{task}

# Code to Review:
```python
{code}
```

Please analyze the time complexity of this code and consider whether there is an <strong>algorithmically superior</strong> solution to significantly improve performance.
If one exists, please clearly point out the deficiencies of the current algorithm and propose specific, feasible algorithm improvement suggestions (e.g., using sieve method instead of trial division).
Only if the code has reached optimality at the algorithm level can you answer "no improvement needed."

Please output your feedback directly without any additional explanations.
"""

3. Refinement Prompt: After receiving feedback, this prompt will guide the model to revise and optimize the original code based on the feedback content.

REFINE_PROMPT_TEMPLATE = """
You are a senior Python programmer. You are optimizing your code based on feedback from a code review expert.

# Original Task:
{task}

# Your Previous Code Attempt:
{last_code_attempt}
Reviewer's Feedback:
{feedback}

Please generate an optimized new version of the code based on the reviewer's feedback.
Your code must include a complete function signature, docstring, and follow PEP 8 coding standards.
Please output the optimized code directly without any additional explanations.
"""

(2) Agent Encapsulation and Implementation

Now, we will integrate this set of prompt logic and the Memory module into the ReflectionAgent class.

# Assume llm_client.py and memory.py are already defined
# from llm_client import HelloAgentsLLM
# from memory import Memory

class ReflectionAgent:
    def __init__(self, llm_client, max_iterations=3):
        self.llm_client = llm_client
        self.memory = Memory()
        self.max_iterations = max_iterations

    def run(self, task: str):
        print(f"\n--- Starting to Process Task ---\nTask: {task}")

        # --- 1. Initial Execution ---
        print("\n--- Performing Initial Attempt ---")
        initial_prompt = INITIAL_PROMPT_TEMPLATE.format(task=task)
        initial_code = self._get_llm_response(initial_prompt)
        self.memory.add_record("execution", initial_code)

        # --- 2. Iterative Loop: Reflection and Refinement ---
        for i in range(self.max_iterations):
            print(f"\n--- Iteration {i+1}/{self.max_iterations} ---")

            # a. Reflection
            print("\n-> Performing Reflection...")
            last_code = self.memory.get_last_execution()
            reflect_prompt = REFLECT_PROMPT_TEMPLATE.format(task=task, code=last_code)
            feedback = self._get_llm_response(reflect_prompt)
            self.memory.add_record("reflection", feedback)

            # b. Check if stopping is needed
            if "no improvement needed" in feedback.lower():
                print("\n✅ Reflection considers code needs no improvement, task completed.")
                break

            # c. Refinement
            print("\n-> Performing Refinement...")
            refine_prompt = REFINE_PROMPT_TEMPLATE.format(
                task=task,
                last_code_attempt=last_code,
                feedback=feedback
            )
            refined_code = self._get_llm_response(refine_prompt)
            self.memory.add_record("execution", refined_code)

        final_code = self.memory.get_last_execution()
        print(f"\n--- Task Completed ---\nFinal Generated Code:\n```python\n{final_code}\n```")
        return final_code

    def _get_llm_response(self, prompt: str) -> str:
        """A helper method for calling LLM and getting complete streaming response."""
        messages = [{"role": "user", "content": prompt}]
        response_text = self.llm_client.think(messages=messages) or ""
        return response_text

4.4.4 Running Instance and Analysis

The complete code can also be found in the code folder of this book's accompanying code repository; here we provide an output instance.

--- Starting to Process Task ---
Task: Write a Python function to find all prime numbers between 1 and n.

--- Performing Initial Attempt ---
🧠 Calling xxxxxx model...
✅ Large language model response successful:
```python
def find_primes(n):
    ...
    return primes
```
📝 Memory updated, added an 'execution' record.

--- Iteration 1/2 ---

-> Performing Reflection...
🧠 Calling xxxxxx model...
✅ Large language model response successful:
The current code has a time complexity of O(n * sqrt(n)). While this implementation is acceptable for smaller n values, performance will significantly degrade when n is very large. The main bottleneck is that each number needs trial division checking, leading to high time overhead.

It is recommended to use the Sieve of Eratosthenes algorithm, which has a time complexity of O(n log(log n)) and can significantly improve the efficiency of finding prime numbers.

Improved code as follows:
```python
def find_primes(n):
    ...
    return primes
```
📝 Memory updated, added a 'reflection' record.

-> Performing Refinement...
🧠 Calling xxxxxx model...
✅ Large language model response successful:
```python
def find_primes(n):
    ...
    return primes
```
📝 Memory updated, added an 'execution' record.

--- Iteration 2/2 ---

-> Performing Reflection...
🧠 Calling xxxxxx model...
✅ Large language model response successful:
The current code uses the Sieve of Eratosthenes with time complexity O(n log log n) and space complexity O(n). This algorithm is already very efficient for finding all prime numbers between 1 and n, and usually requires no further optimization. However, in some specific scenarios, the following improvements can be considered:

1. <strong>Segmented Sieve</strong>: Suitable for cases where n is very large but memory is limited. Divide the interval into multiple small segments, process each segment separately with the sieve method, reducing memory usage.
2. <strong>Odd Number Sieve</strong>: Except for 2, all prime numbers are odd. When initializing the `is_prime` array, only mark odd numbers, which can reduce space complexity by half while reducing some unnecessary calculations.

However, these improvements are not necessary for most application scenarios because the standard Sieve of Eratosthenes is already efficient enough. Therefore, in general cases, <strong>no improvement needed</strong>.
📝 Memory updated, added a 'reflection' record.

✅ Reflection considers code needs no improvement, task completed.

--- Task Completed ---
Final Generated Code:
```python
def find_primes(n):
    """
    Finds all prime numbers between 1 and n using the Sieve of Eratosthenes algorithm.

    :param n: The upper limit of the range to find prime numbers.
    :return: A list of all prime numbers between 1 and n.
    """
    if n < 2:
        return []

    is_prime = [True] * (n + 1)
    is_prime[0] = is_prime[1] = False

    p = 2
    while p * p <= n:
        if is_prime[p]:
            for i in range(p * p, n + 1, p):
                is_prime[i] = False
        p += 1

    primes = [num for num in range(2, n + 1) if is_prime[num]]
    return primes
```

This running instance demonstrates how the Reflection mechanism drives the agent to perform deep optimization:

1. Effective "Criticism" is the Prerequisite for Optimization: In the first round of reflection, because we used an "extremely strict" and "focused on algorithm efficiency" prompt, the agent was not satisfied with the functionally correct initial code but precisely pointed out its O(n * sqrt(n)) time complexity bottleneck and proposed algorithm-level improvement suggestions—the Sieve of Eratosthenes.
2. Iterative Improvement: After receiving clear feedback, the agent successfully implemented a more efficient sieve method in the refinement phase, reducing algorithm complexity to O(n log log n), completing the first meaningful self-iteration.
3. Convergence and Termination: In the second round of reflection, facing the already efficient sieve method, the agent demonstrated deeper knowledge. It not only affirmed the current algorithm's efficiency but even mentioned more advanced optimization directions like segmented sieve, but ultimately made the correct judgment of "no improvement needed in general cases." This judgment triggered our termination condition, allowing the optimization process to converge.

This case fully proves that a well-designed Reflection mechanism's value lies not only in fixing errors but more importantly in driving solutions to achieve step-wise improvements in quality and efficiency, making it one of the key technologies for building complex, high-quality agents.

4.4.5 Cost-Benefit Analysis of Reflection Mechanism

Although the Reflection mechanism performs excellently in improving task solution quality, this capability is not without cost. In practical applications, we need to weigh the benefits it brings against the corresponding costs.

(1) Main Costs

1. Increased Model Call Overhead: This is the most direct cost. Each iteration requires at least two additional large language model calls (one for reflection, one for refinement). If iterating multiple rounds, API call costs and computational resource consumption will increase exponentially.

2. Significantly Increased Task Latency: Reflection is a serial process; each round of refinement must wait for the previous round's reflection to complete. This significantly extends the total task time, making it unsuitable for scenarios with high real-time requirements.

3. Increased Prompt Engineering Complexity: As our case demonstrates, the success of Reflection largely depends on high-quality, targeted prompts. Designing and debugging effective prompts for different stages like "execution," "reflection," and "refinement" requires more development effort.

(2) Core Benefits

1. Leap in Solution Quality: The greatest benefit is that it can iteratively optimize a "qualified" initial solution into an "excellent" final solution. This improvement from functionally correct to performance-efficient, from rough logic to rigorous logic, is crucial in many critical tasks.

2. Enhanced Robustness and Reliability: Through internal self-correction loops, the agent can discover and fix potential logical flaws, factual errors, or improper boundary case handling in the initial solution, greatly improving the reliability of the final result.

In summary, the Reflection mechanism is a typical "cost for quality" strategy. It is very suitable for scenarios that have extremely high requirements for the quality, accuracy, and reliability of final results, and have relatively relaxed requirements for task completion real-time performance. For example:

• Generating critical business code or technical reports.
• Conducting complex logical reasoning in scientific research.
• Decision support systems requiring deep analysis and planning.

Conversely, if the application scenario requires quick responses, or a "roughly correct" answer is already sufficient, using lighter ReAct or Plan-and-Solve paradigms may be a more cost-effective choice.