6. Observability Stack

The open-source baseline that works in production, documented by independent organizations:

OpenLLMetry (Traceloop)       ← instrumentation layer (Python + TypeScript)
      +
OpenInference (Arize)         ← semantic schema for LLM/agent attributes
      ↓
Langfuse or Arize Phoenix      ← tracing backend + eval storage
      +
DeepEval or LangWatch Scenario ← quality evaluation

For enterprise with governance requirements, add Strata Maverics or Entra Agent ID in the identity layer.

OpenTelemetry GenAI conventions (August 2026 status)

The canonical OpenTelemetry GenAI registry defines shared fields for conversations, system instructions, tool definitions, tool-call IDs, arguments, results, workflow names, token usage, and operations such as execute_tool, invoke_agent, and invoke_workflow. The GenAI conventions are still evolving and several definitions have moved to the dedicated semantic-conventions repository, so pin the version used by production instrumentation.

Do not record full content by default. System instructions, retrieval queries, tool arguments, and tool results may contain secrets or personal data. Prefer identifiers, sizes, hashes, allowlisted fields, and explicit redaction. Enable payload capture only for a bounded diagnostic purpose with retention and access controls.

What to instrument first

1. Every LLM call: latency, input tokens, output tokens, model name, finish reason.
2. Every tool call: tool name, arguments (redacted if sensitive), result success/failure, latency.
3. Session-level: total tokens per session, session duration, task completion (binary).
4. Eval scores: task completion rate, tool correctness rate (correct tools used / total tools used).

Operational measures beyond telemetry

Token and latency traces explain resource use. They do not establish that the agent produced acceptable work. Add outcome and control-flow measures that connect a run to its reviewed result.

Measure	Why it matters	Source prompt
Turns to accepted completion	Reveals friction hidden by a final pass/fail score	Patrick Debois discusses turn count as an agent-enablement signal
Manual takeovers and human review comments	Measures where the harness still depends on rescue	The AI Native DevCon interview proposes tracking takeovers and pull-request comments
Requirement coverage	Prevents a broad success score from hiding one missed contract	Executable-spec verifiers can report one verdict per requirement
Evidence completeness	Requires commands, outputs, tests, screenshots, or traces for claims of completion	Simon Willison demonstrates a Markdown proof artifact built from commands and outputs
Latency distribution	Captures slow tails that a mean or median hides	Amit Kushwaha argues for distribution-aware agent benchmarks
Controlled recovery	Tests whether state survives an interruption without duplicated or lost work	Interrupt the run at a defined point, then inspect its resumed trajectory
Repeated-run reliability	Separates a task solved once from behavior that succeeds consistently	Towards a Science of AI Agent Reliability defines consistency, robustness, predictability, and safety as separate dimensions
Harness-model pair	Prevents a model name from absorbing harness-specific cost and failure behavior	The Scaffold Effect reports harness-specific cost and failure fingerprints under fixed models

Report denominators and distributions. "Eight accepted tasks out of ten" is inspectable. "The agent usually succeeds" is not. For latency and cost, include at least the median and a tail percentile when the sample supports it. For small pilots, publish every result instead of implying statistical precision.

Datadog, Honeycomb, New Relic, and MLflow all support OTel GenAI conventions. Arize Phoenix processes 1 trillion spans per month across DoorDash, Instacart, Reddit, Uber, and Booking.com, making it the most documented production scale for an open-source option in this space.

LLM-as-judge limitations

JudgeBiasBench (arXiv 2604.23178, Hongli Zhou et al., April 2026) measured more than 50% error rates for frontier models on advanced bias detection tests. Style bias is the dominant pattern: scores of 0.76-0.92, versus position bias below 0.04. The practical consequence: an LLM judge approves stylistically polished but incorrect outputs significantly more often than it should. True negative rates for identifying invalid outputs typically sit below 25%.

No commercial platform (Arize, LangWatch, Openlayer, Langfuse as of May 2026) publicly documents how it neutralizes style bias in its evaluators.

The working solution: use LLM-as-judge for qualitative dimensions that cannot be evaluated deterministically (tone, explanation quality, context sensitivity). Use code-based checks for anything that can be evaluated mechanically (tool selection correctness, structured output schema compliance, regression tests on known examples). Do not use LLM-as-judge alone as the quality gate for production traffic.