15.4 Production Considerations

Telemetry and Observability

A production Agent Harness must have comprehensive observability. Claude Code's approach is worth emulating: instrument key paths to record metrics such as message count, tool result count, query chain ID, and query depth.

A production-grade Harness should track at minimum the following metrics:

• Latency distribution: Model call latency and tool execution latency per loop turn
• Token budget: Input/output token consumption trends, compression frequency
• Error rate: Failure rates categorized by error type (model errors, tool errors, permission denials)
• Behavioral metrics: Average tool calls per turn, average turns to completion, user interruption rate

Observability implementation roadmap:

flowchart TD
    subgraph L1["Layer 1: Structured Logs (Effort: Low, Value: High)"]
        L1A["Per tool call:<br/>{tool, duration, input_summary, output_size, status}"]
        L1B["Per loop turn:<br/>{turn_number, total_tokens, tool_count, duration}"]
        L1C["Output to stderr or file<br/>JSON Lines format"]
        L1A --> L1B --> L1C
    end

    subgraph L2["Layer 2: Metric Aggregation (Effort: Medium, Value: Medium)"]
        L2A["Set up Prometheus/Grafana monitoring stack"]
        L2B["Define key metrics:<br/>P50/P95/P99 latency, error rate, token consumption rate"]
        L2C["Configure alert thresholds"]
        L2A --> L2B --> L2C
    end

    subgraph L3["Layer 3: Distributed Tracing (Effort: High, Value: High)"]
        L3A["Integrate OpenTelemetry"]
        L3B["Assign trace_id, span_id"]
        L3C["Implement request chain visualization"]
        L3A --> L3B --> L3C
    end

    subgraph L4["Layer 4: Anomaly Detection (Effort: High, Value: Incremental)"]
        L4A["Establish normal behavior baseline"]
        L4B["Automated anomaly detection and alerting"]
        L4C["Support post-mortem analysis and root cause identification"]
        L4A --> L4B --> L4C
    end

    L1 --> L2 --> L3 --> L4

    classDef l1Node fill:#d4edda,stroke:#27ae60,stroke-width:1px
    classDef l2Node fill:#e8f4f8,stroke:#2980b9,stroke-width:1px
    classDef l3Node fill:#fff3cd,stroke:#f39c12,stroke-width:1px
    classDef l4Node fill:#f8d7da,stroke:#c0392b,stroke-width:1px

    class L1A,L1B,L1C l1Node
    class L2A,L2B,L2C l2Node
    class L3A,L3B,L3C l3Node
    class L4A,L4B,L4C l4Node

Multi-Environment Adaptation: CLI / IDE / SDK / Server

Claude Code's dependency injection pattern provides an elegant multi-environment adaptation solution. By abstracting I/O operations into dependency injection interfaces (such as model calls, message compression, UUID generation, etc.), the core loop doesn't need to know what environment it's running in.

In a CLI environment, production dependencies provide real file system access and model calls. In tests, mock implementations can be injected. In SDK mode, different UI feedback mechanisms can be substituted. This "core + adapter" architecture allows the same Harness logic to run seamlessly across different platforms.

Key environment difference points include:

Environment	Permission Interaction	UI Rendering	Hook Execution	Session Storage
CLI	Terminal prompt	Ink/React	Shell commands	Local files
IDE (VS Code)	Dialog popup	WebView	Same	Same
SDK	Callback functions	None	Same	In-memory
Server	Auto-deny/allow	None	HTTP endpoints	Database

Multi-environment adaptation architecture pattern:

flowchart TD
    CORE["Core Agent Loop<br/>Environment-agnostic"]

    CORE --> DEPS["AgentDeps<br/>Dependency Injection Interface"]

    DEPS --> CLI["CLI Adapter<br/>Terminal rendering<br/>User confirmation<br/>Local storage"]
    DEPS --> SDK["SDK Adapter<br/>Callback notifications<br/>Programmatic control<br/>In-memory storage"]
    DEPS --> SERVER["Server Adapter<br/>HTTP API<br/>Auto-approval<br/>Database storage"]

    classDef coreNode fill:#e8f4f8,stroke:#2980b9,stroke-width:2px
    classDef depsNode fill:#fff3cd,stroke:#f39c12,stroke-width:2px
    classDef adapterNode fill:#d4edda,stroke:#27ae60,stroke-width:1px

    class CORE coreNode
    class DEPS depsNode
    class CLI,SDK,SERVER adapterNode

The core principle of this architecture is "the core doesn't know where it's running." If any if (isCLI) ... else if (isSDK) ... branch logic appears in the core code, it means dependency injection isn't thorough enough -- environment differences should be encapsulated in adapters, not handled through conditional logic in core code.

Security Auditing

An Agent Harness's security boundary is more complex than traditional applications because LLM output is unpredictable. Claude Code takes a multi-layered defense approach to security:

1. Tool permission tiers: Each tool declares isReadOnly, isDestructive, and isConcurrencySafe; the system determines the approval flow based on these annotations
2. Workspace trust check: shouldSkipHookDueToTrust() ensures hooks don't execute in untrusted workspaces
3. Input sanitization: The backfillObservableInput method cleans tool input before passing it to observers, without modifying the API-bound original input (to avoid breaking prompt caching)
4. Budget ceiling: The maxBudgetUsd parameter limits token costs per query

Security threat model for Agent systems:

Agent systems face threats fundamentally different from traditional applications. Traditional application threats come from external attackers (injecting malicious input), while Agent system threats come from the LLM itself -- the model may execute unintended operations due to prompt injection.

flowchart TD
    subgraph Threat Model
        T1["Prompt Injection<br/>Malicious content injected via files/web pages"]
        T2["Tool Abuse<br/>Model calls tools it shouldn't"]
        T3["Resource Exhaustion<br/>Infinite loops consuming tokens/API quotas"]
        T4["Data Leakage<br/>Tool results contain sensitive information"]
        T5["Supply Chain Attack<br/>Malicious hooks/MCP servers"]
    end

    subgraph Defenses
        D1["Tool result isolation + permission checks"]
        D2["Permission pipeline + user confirmation"]
        D3["Budget ceiling + maximum turn limit"]
        D4["Input sanitization + workspace trust check"]
        D5["Hook allowlist + sandboxed execution"]
    end

    T1 -->|"Defense"| D1
    T2 -->|"Defense"| D2
    T3 -->|"Defense"| D3
    T4 -->|"Defense"| D4
    T5 -->|"Defense"| D5

    classDef threatNode fill:#f8d7da,stroke:#c0392b,stroke-width:1px
    classDef defenseNode fill:#d4edda,stroke:#27ae60,stroke-width:1px

    class T1,T2,T3,T4,T5 threatNode
    class D1,D2,D3,D4,D5 defenseNode

Security audit checklist:

• [ ] Every tool has correct isReadOnly/isDestructive/isConcurrencySafe annotations
• [ ] The permission pipeline completes all checks before tool execution (no bypass paths)
• [ ] Budget ceiling is checked on every loop turn, not just at session end
• [ ] Sensitive information in tool results is sanitized before passing to the model
• [ ] Hook commands execute in trusted workspaces; untrusted workspaces are automatically skipped
• [ ] Sub-Agent permissions do not exceed parent Agent's permission scope
• [ ] All permission decisions have audit logs