Evaluation Reference: Metrics and Implementation

This document provides implementation details for evaluation metrics and evaluation systems.

Core Metric Definitions

Factual Accuracy

Factual accuracy measures whether claims in agent output match ground truth.

Excellent (1.0): All claims verified against ground truth, no errors
Good (0.8): Minor errors that do not affect main conclusions
Acceptable (0.6): Major claims correct, minor inaccuracies present
Poor (0.3): Significant factual errors in key claims
Failed (0.0): Fundamental factual errors that invalidate output

Calculation approach:

• Extract claims from output
• Verify each claim against ground truth
• Weight claims by importance (major claims more weight)
• Calculate weighted average of claim accuracy

Completeness

Completeness measures whether output covers all requested aspects.

Excellent (1.0): All requested aspects thoroughly covered
Good (0.8): Most aspects covered with minor gaps
Acceptable (0.6): Key aspects covered, some gaps
Poor (0.3): Major aspects missing from output
Failed (0.0): Fundamental aspects not addressed