9. LLMOps & Observability
You cannot optimize what you do not measure. The LLMOps tooling category provides the instrumentation layer: tracing, cost tracking, quality evaluation, and drift detection for LLM-powered systems.
Local Session Inspectors: claude-devtools and tokview
Before reaching for a hosted platform, two zero-cloud tools read Claude Code's own local session logs (~/.claude/projects/**/*.jsonl) directly and turn them into per-turn attribution, no proxy or code change required for historical data.
| Tool | Mechanism | Distinguishing feature |
|---|---|---|
claude-devtools (github.com/matt1398/claude-devtools, MIT, brew install --cask claude-devtools) | Electron desktop app that parses local session transcripts | Per-turn token attribution across 7 categories: CLAUDE.md (global/project/directory), skill activations, @-mentioned files, tool call I/O, extended thinking, team/subagent overhead, user text. Finer-grained than the native /context command's three-segment bar. |
tokview (github.com/headroomlabs-ai/tokview, MIT, uv tool install token-viewer) | Local proxy plus terminal/browser dashboard; tokview import claude backfills from existing JSONL history with no proxy needed for past sessions | Attribution down to the individual tool call, live as the agent runs, plus estimated-cost tracking for subscription (non-API) usage |
Both run entirely on local files: no telemetry, no login, no account. tokview ships from headroomlabs-ai, the same team behind Headroom (§3): treat its hotspot suggestions as a lead-in to Headroom's own compression tooling rather than a vendor-neutral recommendation, the same caveat this guide already applies to Headroom's self-reported figures.
For a codebase small enough to inspect by hand, the same 7-category breakdown can be approximated with a 20-line script that sums usage.input_tokens per tool_use block across a session's JSONL file. No install, no dependency, and it teaches what the categories actually mean before reaching for a packaged dashboard.
Langfuse
The leading open-source option. Langfuse traces LLM calls across complex multi-step agent workflows, capturing input/output at each step, latency, cost per call, and the full execution tree.
| Attribute | Details |
|---|---|
| Source | github.com/langfuse/langfuse |
| License | Open-source (MIT) |
| Deployment | Self-hosted or cloud |
| Best for | Self-hosting requirements, cost analysis, trace debugging |
Key features for context optimization: per-session token cost breakdowns, trace comparison (which prompt variant is cheaper?), and custom evaluation metrics you can run on stored traces without rerunning the agent.
LangSmith
LangSmith is Anthropic-adjacent (LangChain ecosystem) and the standard choice if you are building on LangChain or LangGraph. It excels at debugging chained operations where understanding the execution graph is as important as the individual LLM calls.
| Attribute | Details |
|---|---|
| Source | smith.langchain.com |
| Best for | LangChain/LangGraph workloads, chain debugging, A/B testing |
| Features | Dataset management, automated evaluation, regression testing |
Arize Phoenix
Phoenix specializes in RAG quality evaluation, implementing the RAG Triad natively. It traces retrieval operations alongside generation, so you can correlate retrieval quality with final answer quality.
| Attribute | Details |
|---|---|
| Source | github.com/Arize-ai/phoenix |
| License | Open-source |
| Specialty | RAG evaluation, LLM-as-judge metrics, embedding drift |
Particularly useful for: identifying when retrieval is the bottleneck (context relevance failures) versus generation (groundedness failures). This distinction determines whether you fix the retriever or the prompt.
Maxim AI
Maxim AI focuses on continuous evaluation: running automated evals against every production trace, not just offline test sets. It supports LLM-as-a-judge workflows (using an LLM to score another LLM's output) and A/B testing of prompt variants against production traffic.
| Attribute | Details |
|---|---|
| Source | getmaxim.ai |
| Best for | Continuous eval, A/B testing in production, regression detection |
TruLens
TruLens implements the RAG Triad evaluation framework as an open-source library. It can be embedded directly in your application code, running evaluations inline as part of the application rather than in a separate observability platform.
| Attribute | Details |
|---|---|
| Source | github.com/truera/trulens |
| License | Open-source |
| Best for | Inline RAG evaluation, library integration, RAG Triad scoring |
Choosing an Observability Tool
| Need | Recommended |
|---|---|
| Self-host everything, cost analysis | Langfuse |
| LangChain ecosystem, chain debugging | LangSmith |
| RAG quality evaluation specifically | Arize Phoenix |
| Continuous prod eval, A/B testing | Maxim AI |
| Embed RAG Triad in app code | TruLens |
These tools are not mutually exclusive. Langfuse for tracing plus Phoenix for RAG evaluation is a common combination.