# Built-in Evaluators Reference

Complete reference for Microsoft Foundry's built-in evaluators using the `azure-ai-projects` SDK.

## Discovering Evaluators

### List All Built-in Evaluators

```python
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
):
    evaluators = project_client.evaluators.list_latest_versions(type="builtin")
    for e in evaluators:
        print(f"{e.name}: {e.description}")
        print(f"  Categories: {e.categories}")
```

### Get Evaluator Schema

Before using an evaluator, query its schema to discover required inputs:

```python
evaluator = project_client.evaluators.get_version(
    name="builtin.task_adherence",
    version="latest"
)
print(f"Init Parameters: {evaluator.definition.init_parameters}")
print(f"Data Schema: {evaluator.definition.data_schema}")
print(f"Metrics: {evaluator.definition.metrics}")
```

## Using Built-in Evaluators

All built-in evaluators use the `azure_ai_evaluator` type with `builtin.` prefix:

```python
testing_criteria = [
    {
        "type": "azure_ai_evaluator",
        "name": "my_coherence_check",          # Your custom name for results
        "evaluator_name": "builtin.coherence", # The actual evaluator
        "data_mapping": {
            "query": "{{item.query}}",
            "response": "{{item.response}}"
        },
        "initialization_parameters": {
            "deployment_name": "gpt-4o-mini"   # Required for LLM-based evaluators
        }
    }
]
```

## Quality Evaluators

### builtin.coherence

Measures logical flow and consistency of the response.

```python
{
    "type": "azure_ai_evaluator",
    "name": "coherence",
    "evaluator_name": "builtin.coherence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}
```

**Inputs:** query, response  
**Output:** Score 1-5 (5 = highly coherent)

### builtin.fluency

Measures grammatical correctness and natural language quality.

```python
{
    "type": "azure_ai_evaluator",
    "name": "fluency",
    "evaluator_name": "builtin.fluency",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}
```

**Inputs:** query, response  
**Output:** Score 1-5 (5 = perfectly fluent)

### builtin.relevance

Measures how well the response addresses the query given context.

```python
{
    "type": "azure_ai_evaluator",
    "name": "relevance",
    "evaluator_name": "builtin.relevance",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "context": "{{item.context}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}
```

**Inputs:** query, response, context  
**Output:** Score 1-5 (5 = highly relevant)

### builtin.groundedness

Measures whether the response is factually grounded in the provided context.

```python
{
    "type": "azure_ai_evaluator",
    "name": "groundedness",
    "evaluator_name": "builtin.groundedness",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "context": "{{item.context}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}
```

**Inputs:** query, response, context  
**Output:** Score 1-5 (5 = fully grounded)

### builtin.response_completeness

Measures whether the response fully addresses all aspects of the query.

```python
{
    "type": "azure_ai_evaluator",
    "name": "response_completeness",
    "evaluator_name": "builtin.response_completeness",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}
```

**Inputs:** query, response  
**Output:** Score 1-5

## Safety Evaluators

Safety evaluators detect harmful content. They don't require `deployment_name`.

### builtin.violence

Detects violent content.

```python
{
    "type": "azure_ai_evaluator",
    "name": "violence",
    "evaluator_name": "builtin.violence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}
```

**Inputs:** query, response  
**Output:** pass/fail with severity score

### builtin.sexual

Detects inappropriate sexual content.

```python
{
    "type": "azure_ai_evaluator",
    "name": "sexual",
    "evaluator_name": "builtin.sexual",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}
```

### builtin.self_harm

Detects content promoting or describing self-harm.

```python
{
    "type": "azure_ai_evaluator",
    "name": "self_harm",
    "evaluator_name": "builtin.self_harm",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}
```

### builtin.hate_unfairness

Detects biased or hateful content.

```python
{
    "type": "azure_ai_evaluator",
    "name": "hate_unfairness",
    "evaluator_name": "builtin.hate_unfairness",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}
```

## Agent Evaluators

Agent evaluators assess AI agent behavior and tool usage.

### builtin.task_adherence

Evaluates whether the agent follows its system instructions.

```python
{
    "type": "azure_ai_evaluator",
    "name": "task_adherence",
    "evaluator_name": "builtin.task_adherence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}
```

**Note:** Use `{{sample.output_items}}` for agent responses to include tool call information.

### builtin.intent_resolution

Evaluates whether the agent correctly understood user intent.

```python
{
    "type": "azure_ai_evaluator",
    "name": "intent_resolution",
    "evaluator_name": "builtin.intent_resolution",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_text}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}
```

### builtin.task_completion

Evaluates whether the agent completed the task end-to-end.

```python
{
    "type": "azure_ai_evaluator",
    "name": "task_completion",
    "evaluator_name": "builtin.task_completion",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}
```

### builtin.tool_call_accuracy

Evaluates whether tool calls are correct (selection + parameters).

```python
{
    "type": "azure_ai_evaluator",
    "name": "tool_call_accuracy",
    "evaluator_name": "builtin.tool_call_accuracy",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}
```

### builtin.tool_call_success

Evaluates whether tool calls executed without failures.

```python
{
    "type": "azure_ai_evaluator",
    "name": "tool_call_success",
    "evaluator_name": "builtin.tool_call_success",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"}
}
```

### builtin.tool_selection

Evaluates whether the correct tools were selected.

```python
{
    "type": "azure_ai_evaluator",
    "name": "tool_selection",
    "evaluator_name": "builtin.tool_selection",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}
```

## NLP Evaluators

NLP evaluators compare responses to ground truth without requiring an LLM.

### builtin.f1_score

Token-level F1 score between response and ground truth.

```python
{
    "type": "azure_ai_evaluator",
    "name": "f1",
    "evaluator_name": "builtin.f1_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}
```

**Output:** Score 0-1

### builtin.bleu_score

BLEU score for generation quality.

```python
{
    "type": "azure_ai_evaluator",
    "name": "bleu",
    "evaluator_name": "builtin.bleu_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}
```

### builtin.rouge_score

ROUGE score for summarization quality.

```python
{
    "type": "azure_ai_evaluator",
    "name": "rouge",
    "evaluator_name": "builtin.rouge_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}
```

### builtin.similarity

Semantic similarity between response and ground truth.

```python
{
    "type": "azure_ai_evaluator",
    "name": "similarity",
    "evaluator_name": "builtin.similarity",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "ground_truth": "{{item.ground_truth}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}
```

## Evaluator Sets by Use Case

### Quick Health Check

```python
testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "coherence", "evaluator_name": "builtin.coherence", ...},
    {"type": "azure_ai_evaluator", "name": "fluency", "evaluator_name": "builtin.fluency", ...},
    {"type": "azure_ai_evaluator", "name": "violence", "evaluator_name": "builtin.violence", ...},
]
```

### Safety Audit

```python
testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "violence", "evaluator_name": "builtin.violence", ...},
    {"type": "azure_ai_evaluator", "name": "sexual", "evaluator_name": "builtin.sexual", ...},
    {"type": "azure_ai_evaluator", "name": "self_harm", "evaluator_name": "builtin.self_harm", ...},
    {"type": "azure_ai_evaluator", "name": "hate_unfairness", "evaluator_name": "builtin.hate_unfairness", ...},
]
```

### Agent Evaluation

```python
testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "task_adherence", "evaluator_name": "builtin.task_adherence", ...},
    {"type": "azure_ai_evaluator", "name": "intent_resolution", "evaluator_name": "builtin.intent_resolution", ...},
    {"type": "azure_ai_evaluator", "name": "tool_call_accuracy", "evaluator_name": "builtin.tool_call_accuracy", ...},
]
```

### RAG Evaluation

```python
testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "groundedness", "evaluator_name": "builtin.groundedness", ...},
    {"type": "azure_ai_evaluator", "name": "relevance", "evaluator_name": "builtin.relevance", ...},
    {"type": "azure_ai_evaluator", "name": "response_completeness", "evaluator_name": "builtin.response_completeness", ...},
]
```

## Data Mapping Reference

| Data Source | Response Mapping | Use Case |
|-------------|------------------|----------|
| JSONL dataset | `{{item.response}}` | Pre-recorded query/response pairs |
| Agent target | `{{sample.output_text}}` | Plain text response |
| Agent target | `{{sample.output_items}}` | Structured JSON with tool calls |

**When to use `sample.output_items`:**
- Tool-related evaluators (tool_call_accuracy, tool_selection, etc.)
- Task adherence evaluator
- Any evaluator needing tool call context

## Related Documentation

- [Azure AI Projects Samples](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/ai/azure-ai-projects/samples/evaluations)
- [Agent Evaluators](https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-evaluators/agent-evaluators)
- [RAG Evaluators](https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-evaluators/rag-evaluators)
- [Risk and Safety Evaluators](https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-evaluators/risk-safety-evaluators)