12.1 Agent Evaluation Fundamentals

12.1.1 Why Agent Evaluation is Needed

We now have SimpleAgent, which already possesses powerful reasoning and tool invocation capabilities. Let's look at a typical usage scenario:

from hello_agents import SimpleAgent, HelloAgentsLLM
from hello_agents.tools import SearchTool

# Create LLM and agent
llm = HelloAgentsLLM()

# Create a system prompt emphasizing tool use
system_prompt = """You are an AI assistant that can use search tools to obtain the latest information.

When you need to search for information, please use the following format:
[TOOL_CALL:search:search keywords]

For example:
- [TOOL_CALL:search:latest AI news]
- [TOOL_CALL:search:Python programming tutorial]

Please use the search tool to obtain the latest information before answering questions."""

agent = SimpleAgent(name="AI Assistant", llm=llm, system_prompt=system_prompt)

# Add search tool
agent.add_tool(SearchTool())

# Example: Use search tool to answer questions
response = agent.run("What are the latest AI technology development trends?")
print(f"\nAnswer: {response}")

This agent can work normally, but we face a core problem: How to objectively evaluate its performance? When we optimize prompts or change LLM models, how do we know if there's real improvement? Before deploying to production environment, how do we ensure agent reliability? These questions all need to be solved through systematic evaluation.

The core value of agent evaluation lies in providing standardized methods to measure agent capabilities. Through evaluation, we can quantify agent performance with specific numerical metrics, objectively compare the merits of different design solutions, promptly discover agent weaknesses in specific scenarios, and prove agent reliability to users.

Unlike traditional software testing, agent evaluation faces unique challenges. First is output uncertainty - the same question may have multiple correct answers, making it difficult to judge with simple right or wrong. Second is diversity of evaluation criteria - different tasks require different evaluation methods; tool invocation needs to check function signatures, while Q&A tasks need to evaluate semantic similarity. Finally is high evaluation cost - each evaluation requires numerous API calls, potentially costing hundreds of yuan or more.

To address these challenges, academia and industry have proposed multiple standardized Benchmarks. These benchmarks provide unified datasets, evaluation metrics, and scoring methods, enabling us to evaluate and compare different agent systems under the same standards.

12.1.2 Overview of Mainstream Evaluation Benchmarks

The agent evaluation field has seen the emergence of multiple influential benchmark tests. Below are some mainstream evaluation benchmarks and metrics:

(1) Tool Invocation Capability Evaluation

Tool invocation is one of the core capabilities of agents. Agents need to understand user intent, select appropriate tools, and correctly construct function calls. Related evaluation benchmarks include:

• BFCL (Berkeley Function Calling Leaderboard)^[1]: Launched by UC Berkeley, includes 1120+ test samples, covering four categories: simple, multiple, parallel, irrelevance, uses AST matching algorithm for evaluation, moderate dataset size, active community.
• ToolBench^[2]: Launched by Tsinghua University, includes 16000+ real API call scenarios, covering complex tool usage scenarios in the real world.
• API-Bank^[3]: Launched by Microsoft Research, includes 53 commonly used API tools, focuses on evaluating agent understanding and invocation of API documentation.

(2) General Capability Evaluation

Evaluates agent comprehensive performance in real-world tasks, including multi-step reasoning, knowledge application, multimodal understanding, etc.:

• GAIA (General AI Assistants)^[4]: Jointly launched by Meta AI and Hugging Face, includes 466 real-world problems, divided into Level 1/2/3 difficulty levels, evaluates multi-step reasoning, tool use, file processing, web browsing capabilities, uses Quasi Exact Match algorithm, tasks are realistic and comprehensive.
• AgentBench^[5]: Launched by Tsinghua University, includes 8 tasks in different domains, comprehensively evaluates agent general capabilities.
• WebArena^[6]: Launched by CMU, evaluates agent task completion and web interaction capabilities in real web environments.

(3) Multi-Agent Collaboration Evaluation

Evaluates the ability of multiple agents to work collaboratively:

• ChatEval^[7]: Evaluates quality of multi-agent dialogue systems.
• SOTOPIA^[8]: Evaluates agent interaction capabilities in social scenarios.
• Custom Collaboration Scenarios: Evaluation tasks designed according to specific application scenarios.

(4) Common Evaluation Metrics

Different benchmarks use different evaluation metrics, common ones include:

• Accuracy Metrics: Accuracy, Exact Match, F1 Score, used to measure answer correctness.
• Efficiency Metrics: Response Time, Token Usage, used to measure execution efficiency.
• Robustness Metrics: Error Rate, Failure Recovery, used to measure fault tolerance.
• Collaboration Metrics: Communication Efficiency, Task Completion, used to measure collaboration effectiveness.

12.1.3 HelloAgents Evaluation System Design

Considering learning curve and practicality, this chapter will focus on the following evaluation scenarios:

1. BFCL: Evaluate tool invocation capability

   • Selection rationale: Moderate dataset size, clear evaluation metrics, active community
   • Applicable scenarios: Evaluate agent function call accuracy

2. GAIA: Evaluate general AI assistant capability

   • Selection rationale: Realistic tasks, difficulty grading, strong comprehensiveness
   • Applicable scenarios: Evaluate agent comprehensive problem-solving capability

3. Data Generation Quality Evaluation: Evaluate LLM-generated data quality

   • Selection rationale: Through this case, experience complete demonstration of using Agent to create data and evaluate data
   • Applicable scenarios: Evaluate quality of generated training data and test data
   • Evaluation methods: LLM Judge, Win Rate, manual verification

Through these three evaluation scenarios, we will build a complete evaluation system. Figure 12.1 shows our evaluation system construction approach.

Figure 12.1 HelloAgents Evaluation System Architecture

12.1.4 Chapter Learning Objectives and Quick Experience

Let's first look at the learning content of Chapter 12:

hello_agents/
├── evaluation/                         # Evaluation module
│   └── benchmarks/                     # Evaluation benchmark implementation
│       ├── bfcl/                       # BFCL evaluation implementation
│       │   ├── dataset.py              # BFCL dataset loader
│       │   ├── evaluator.py            # BFCL evaluator (AST matching)
│       │   ├── metrics.py              # BFCL-specific metrics
│       │   └── ast_matcher.py          # AST matching algorithm
│       ├── gaia/                       # GAIA evaluation implementation
│       │   ├── dataset.py              # GAIA dataset loader
│       │   ├── evaluator.py            # GAIA evaluator (quasi exact match)
│       │   ├── metrics.py              # GAIA-specific metrics
│       │   └── quasi_exact_match.py    # Quasi exact match algorithm
│       └── data_generation/            # Data generation evaluation implementation
│           ├── dataset.py              # AIME dataset loader
│           ├── llm_judge.py            # LLM Judge evaluator
│           └── win_rate.py             # Win Rate evaluator
└── tools/builtin/                      # Built-in tools module
    ├── bfcl_evaluation_tool.py         # BFCL evaluation tool
    ├── gaia_evaluation_tool.py         # GAIA evaluation tool
    ├── llm_judge_tool.py               # LLM Judge tool
    └── win_rate_tool.py                # Win Rate tool

For this chapter's content, the learning objective is to master the ability to apply evaluation tools. Let's first prepare the development environment:

# Install HelloAgents framework (Chapter 12 version)
pip install "hello-agents[evaluation]==0.2.7"

# Set environment variables
export HF_TOKEN="your_huggingface_token"     # For GAIA dataset (setup steps will follow)

# Since the official `bfcl-eval` package requires numpy<=2.0.0, which conflicts with HelloAgents main dependencies, separate installation is needed
pip install "numpy==1.26.4" bfcl-eval

In the following sections, we will deeply learn the detailed usage and introduction of each evaluation method.