Answer and Context Relevancy Evaluations in LlamaIndex

This example demonstrates how to evaluate the quality of answers generated by a LlamaIndex RAG system and the relevancy of the retrieved context using AnswerRelevancyEvaluator and ContextRelevancyEvaluator. Evaluating these aspects is crucial for ensuring the effectiveness of LLM applications, especially those relying on external knowledge.

The process involves:

1. Setting up a basic RAG pipeline with a VectorStoreIndex.
2. Generating synthetic questions and reference answers using RagDatasetGenerator (though for simplicity, a small predefined dataset is used here).
3. Running the RAG pipeline to get predictions (answers and retrieved context).
4. Using AnswerRelevancyEvaluator to assess if the generated answers directly address the user's query.
5. Using ContextRelevancyEvaluator to check if the retrieved context is pertinent to the question asked.
6. Displaying the evaluation results, which include a score (0-1) and feedback from an LLM acting as a judge.

Requirements

• Python 3.10+
• An OpenAI API key set as an environment variable (OPENAI_API_KEY)

Steps for running this example in the shell

1. Install dependencies:

python -m venv .venv

# macOS/Linux:
source .venv/bin/activate

# Windows Command Prompt:
.venv\Scripts\activate.bat

# Windows PowerShell:
.venv\Scripts\Activate.ps1

pip install -r requirements.txt

2. Export your API key as an environment variable:

export OPENAI_API_KEY="sk-..." # Windows cmd: set OPENAI_API_KEY="sk-..." # Windows PowerShell: $env:OPENAI_API_KEY="sk-..."

3. Run the script:

python answer_context_evaluation.py

Output

The script will output the results of the answer relevancy and context relevancy evaluations. For each query, it will show the score and feedback provided by the LLM evaluator, giving insights into how well the RAG system performed.