Skill Evaluation Test Harness

Quick Start

cd tests
pnpm install
pnpm harness --list                              # List available skills
pnpm harness azure-ai-projects-py --mock --verbose # Run evaluation
pnpm test                                        # Run unit tests

Overview

A TypeScript test framework for evaluating AI-generated code against acceptance criteria defined in skill files. Powered by the GitHub Copilot SDK.

Workflow:

1. Load acceptance criteria from tests/scenarios/<skill>/acceptance-criteria.md
2. Run test scenarios from tests/scenarios/<skill>/scenarios.yaml
3. Generate code using GitHub Copilot SDK (or mock responses)
4. Evaluate code against correct/incorrect patterns
5. Report results via console, markdown, or JSON

Architecture

tests/
├── harness/
│   ├── types.ts              # Type definitions
│   ├── criteria-loader.ts    # Parses acceptance-criteria.md
│   ├── evaluator.ts          # Validates code against patterns
│   ├── copilot-client.ts     # Wraps Copilot SDK (with mock fallback)
│   ├── runner.ts             # Main CLI runner
│   ├── ralph-loop.ts         # Iterative improvement loop
│   ├── feedback-builder.ts   # LLM-actionable feedback generator
│   ├── index.ts              # Package exports
│   └── reporters/
│       ├── console.ts        # Pretty console output
│       └── markdown.ts       # Markdown report generation
│
├── scenarios/