11.5 Model Evaluation and Analysis

After training is complete, we need to comprehensively evaluate model performance, not only looking at accuracy as a single metric, but also deeply analyzing model's reasoning quality, error patterns, generalization ability, etc. This section will introduce how to systematically evaluate and analyze Agentic RL models.

11.5.1 Evaluation Metric System

A good evaluation system should be multi-dimensional, measuring model capabilities from different angles. We divide evaluation metrics into three categories: accuracy metrics, efficiency metrics, and quality metrics.

(1) Accuracy Metrics

Accuracy metrics measure whether the model can arrive at correct answers.

Accuracy: Most basic metric, proportion of completely correct answers. Calculation formula: $$ \text{Accuracy} = \frac{\text{Number of correct answers}}{\text{Total number of questions}} $$

Advantages are simple and intuitive, easy to understand and compare. Disadvantages are inability to distinguish "nearly correct" from "completely wrong", may be too coarse for complex tasks.

Top-K Accuracy: Generate K answers, count as correct if at least one is correct. Calculation formula: $$ \text{Accuracy@K} = \frac{\text{Number of questions with at least one correct answer}}{\text{Total number of questions}} $$

This metric reflects the model's "potential", i.e., whether correct answers can be found through multiple sampling.

Numerical Error: For mathematical problems, can calculate error between predicted and true values. Calculation formula:

$$ \text{Error} = \frac{1}{N} \sum_{i=1}^{N} |y_i - \hat{y}_i| $$

This metric can distinguish "nearly correct" (e.g., predicted 72.5, actual 72) from "completely wrong" (e.g., predicted 100, actual 72).

(2) Efficiency Metrics

Efficiency metrics measure the cost of generating answers.

Average Length: Average number of tokens in generated answers. Calculation formula:

$$ \text{Avg Length} = \frac{1}{N} \sum_{i=1}^{N} |y_i| $$

Shorter answers mean lower inference cost and faster response speed.

Reasoning Steps: Number of reasoning steps contained in answers. Calculation formula:

$$ \text{Avg Steps} = \frac{1}{N} \sum_{i=1}^{N} s_i $$

Appropriate number of steps (2-5 steps) indicates model can systematically decompose problems; too many steps may indicate redundant reasoning.

Inference Time: Time required to generate one answer. This metric is important in actual deployment, affecting user experience.

(3) Quality Metrics

Quality metrics measure readability and explainability of answers.

Format Correctness: Whether answers conform to expected format (e.g., containing markers like "Step 1", "Final Answer"). Calculation formula: $$ \text{Format Correctness} = \frac{\text{Number of correctly formatted answers}}{\text{Total number of answers}} $$

Correct format is a basic requirement; answers with confused format are difficult to use even if results are correct.

Reasoning Coherence: Whether reasoning steps are logically coherent. This metric usually requires manual evaluation or specialized evaluation models.

Explainability: Whether answers are easy to understand and verify. Answers with clear steps are more explainable than answers that directly give results.

As shown in Table 11.7, comparison of different metrics.

Table 11.7 Evaluation Metric Comparison

11.5.2 Evaluation Practice

HelloAgents provides comprehensive evaluation functionality, capable of calculating multiple metrics at once.

from hello_agents.tools import RLTrainingTool

rl_tool = RLTrainingTool()

# Comprehensive evaluation
print("=" * 50)
print("Comprehensive GRPO Model Evaluation")
print("=" * 50)

result = rl_tool.run({
    "action": "evaluate",
    "model_path": "./models/grpo_full",
    "max_samples": 200,
    "use_lora": True,

    # Evaluation configuration
    "metrics": [
        "accuracy",           # Accuracy
        "accuracy_at_k",      # Top-K accuracy
        "average_length",     # Average length
        "average_steps",      # Average steps
        "format_correctness", # Format correctness
    ],
    "k": 3,  # Top-3 accuracy
})

# Parse results
eval_data = json.loads(result)

# Print results
print(f"\nEvaluation results:")
print(f"  Accuracy: {eval_data['accuracy']}")
print(f"  Average reward: {eval_data['average_reward']}")
print(f"  Test samples: {eval_data['num_samples']}")

We can compare performance of pretrained model, SFT model, and GRPO model:

# Evaluate three models
models = [
    ("Pretrained Model", "Qwen/Qwen3-0.6B", False),
    ("SFT Model", "./models/sft_full", True),
    ("GRPO Model", "./models/grpo_full", True),
]

results = []
for name, path, use_lora in models:
    print(f"\nEvaluating {name}...")
    result = rl_tool.run({
        "action": "evaluate",
        "model_path": path,
        "max_samples": 200,
        "use_lora": use_lora,
        "metrics": ["accuracy", "average_length", "format_correctness"],
    })
    results.append((name, result))

# Print comparison table
print("\n" + "=" * 70)
print(f"{'Model':<15} {'Accuracy':<12} {'Avg Length':<15} {'Format Correct':<12}")
print("=" * 70)
for name, result in results:
    print(f"{name:<15} {result['accuracy']:<12.2%} {result['average_length']:<15.1f} {result['format_correctness']:<12.2%}")
print("=" * 70)

11.5.3 Error Analysis

Knowing accuracy alone is not enough; we need to deeply analyze what types of problems the model is prone to errors on, thereby guiding subsequent improvements. Model errors can be divided into four categories: calculation errors (reasoning steps correct but calculation wrong, e.g., "48/2=25", indicating insufficient numerical calculation ability), reasoning errors (reasoning logic errors leading to wrong problem-solving approach, e.g., adding first then dividing instead of dividing first then adding, indicating insufficient logical reasoning ability), comprehension errors (not correctly understanding the problem, e.g., question asks for "total" but only calculated part, indicating insufficient language understanding ability), format errors (answer correct but format doesn't meet requirements, e.g., missing "Final Answer:" marker, indicating insufficient format learning).

Error analysis example:

from hello_agents.tools import RLTrainingTool

rl_tool = RLTrainingTool()

# Evaluate and collect error samples
result = rl_tool.run({
    "action": "evaluate",
    "model_path": "./models/grpo_full",
    "max_samples": 200,
    "use_lora": True,
    "return_details": True,  # Return detailed results
})

# Analyze error samples
errors = result['errors']  # Error sample list
print(f"Total errors: {len(errors)}")

# Classify by error type
error_types = {
    "Calculation Error": 0,
    "Reasoning Error": 0,
    "Comprehension Error": 0,
    "Format Error": 0,
}

for error in errors:
    question = error['question']
    prediction = error['prediction']
    ground_truth = error['ground_truth']

    # Simple error classification logic (may need more complex analysis in practice)
    if "Final Answer:" not in prediction:
        error_types["Format Error"] += 1
    elif "Step" in prediction:
        # Has reasoning steps, may be calculation or reasoning error
        # More detailed analysis needed here
        error_types["Calculation Error"] += 1
    else:
        error_types["Comprehension Error"] += 1

# Print error distribution
print("\nError type distribution:")
for error_type, count in error_types.items():
    percentage = count / len(errors) * 100
    print(f"  {error_type}: {count} ({percentage:.1f}%)")

Output example:

Total errors: 76

Error type distribution:
  Calculation Error: 32 (42.1%)
  Reasoning Error: 18 (23.7%)
  Comprehension Error: 22 (28.9%)
  Format Error: 4 (5.3%)

As can be seen, calculation errors are the main error type (42.1%), indicating the model's numerical calculation ability needs strengthening. Format errors are rare (5.3%), indicating SFT training was effective. We can also analyze the model's performance on problems of different difficulty:

# Group by number of reasoning steps
step_groups = {
    "Easy (1-2 steps)": [],
    "Medium (3-4 steps)": [],
    "Hard (5+ steps)": [],
}

for sample in result['details']:
    steps = sample['ground_truth_steps']  # Number of steps in true answer
    correct = sample['correct']

    if steps <= 2:
        step_groups["Easy (1-2 steps)"].append(correct)
    elif steps <= 4:
        step_groups["Medium (3-4 steps)"].append(correct)
    else:
        step_groups["Hard (5+ steps)"].append(correct)

# Calculate accuracy for each group
print("\nAccuracy at different difficulty levels:")
for group_name, results in step_groups.items():
    if len(results) > 0:
        accuracy = sum(results) / len(results)
        print(f"  {group_name}: {accuracy:.2%} ({len(results)} samples)")

Output example:

Accuracy at different difficulty levels:
  Easy (1-2 steps): 78.50% (85 samples)
  Medium (3-4 steps): 58.30% (96 samples)
  Hard (5+ steps): 31.60% (19 samples)

As can be seen, the model performs well on easy problems (78.5%) but poorly on hard problems (31.6%). This indicates the model's multi-step reasoning ability needs improvement.

11.5.4 Improvement Directions

Based on evaluation and analysis results, we can determine improvement directions for the model, as shown in Figure 11.8.

Figure 11.8 Model Improvement Iteration Process

This is a continuous iteration process: train model → evaluate performance → analyze errors → identify problems → select improvement direction → retrain. Through multiple iterations, model performance will continuously improve.