LLM evals with Promptfoo
This example uses Promptfoo to compare two support-reply prompts on the same test set. It combines deterministic assertions with an llm-rubric judge so you can compare prompt quality directly.
Requirements
This example requires Node.js and an OpenAI API key.
Steps for running this example
- Set your API key:
bash
export OPENAI_API_KEY="sk-..."- Run the evaluation:
bash
npx promptfoo@latest eval -c promptfooconfig.yaml- Optional: open the web viewer:
bash
npx promptfoo@latest viewOutput
Promptfoo prints a comparison matrix for both prompt variants and shows the model-graded and deterministic scores for each test case.
Running 4 test cases (up to 4 at a time)...
┌────────────────────────────────────┬────────────────────────────────────┬────────────────────────────────────┬────────────────────────────────────┐
│ policy │ ticket │ [openai:chat:gpt-4o-mini] │ [openai:chat:gpt-4o-mini] │
│ │ │ support-reply-baseline.txt: You │ support-reply-guardrailed.txt: You │
│ │ │ are a support assistant. │ are a support assistant. │
│ │ │ Use only the policy to answer the │ Use only the policy to answer the │
│ │ │ ticket. │ ticket. │
│ │ │ If the policy is insufficient, say │ Do not promise refunds or account │
│ │ │ that a human review is needed. │ changes unless the policy │
│ │ │ Ticket: │ explicitly allows it. │
│ │ │ {{ticket}} │ Ticket: │
│ │ │ Policy: │ {{ticket}} │
│ │ │ {{policy}} │ Policy: │
│ │ │ Write a concis... │ {{polic... │
├────────────────────────────────────┼────────────────────────────────────┼────────────────────────────────────┼────────────────────────────────────┤
│ Duplicate charges must be │ I was charged twice for order │ [FAIL] Thank you for reaching out. │ [PASS] Thank you for reaching out. │
│ acknowledged, escalated to │ │ Your duplicate charge will be │ I will escalate your concern │
│ billing, and verified before a │ │ escalated to billing for │ regarding the duplicate charges to │
│ refund is promised. │ │ verification, and a human review │ our billing team for verification. │
│ │ │ will be conducted to address your │ │
│ │ │ concern. │ │
├────────────────────────────────────┼────────────────────────────────────┼────────────────────────────────────┼────────────────────────────────────┤
│ Ask the user to request a fresh │ My password reset link expired │ [PASS] Please request a fresh │ [PASS] Please request a fresh │
│ reset link and avoid claiming the │ before I could use it. │ password reset link, as expired │ password reset link to proceed │
│ account was restored. │ │ links cannot be reused. If you │ with resetting your password. │
│ │ │ need further assistance, a human │ │
│ │ │ review may be necessary. │ │
└────────────────────────────────────┴────────────────────────────────────┴────────────────────────────────────┴────────────────────────────────────┘
✓ Eval complete
Total Tokens: 1,433
Eval: 402 (308 prompt, 94 completion)
Grading: 1,031 (886 prompt, 145 completion)
Results:
✓ 3 passed (75.00%)
✗ 1 failed (25.00%)
0 errors (0%)
Duration: 3s (concurrency: 4)