Compaction — TODO

Design

Messages array stays flat. A compactionSummary custom message type is inserted at the cut point. Old messages are preserved for display but excluded from LLM context.

m1, m2, m3, m4, m5, [compactionSummary], m6, m7
                      ^--- LLM context starts here

• Display: filter out compactionSummary messages, user sees linear history as-is
• LLM context: find last compactionSummary in array, send it + everything after it
• Re-compaction: no iterative logic needed — previous compaction summary is already in the LLM context window, gets naturally folded into the next summary
• UI hint: optional subtle divider at the compaction point (not the summary content itself)

DB/Schema

No changes. compactionSummary is a custom AgentMessage that serializes into the existing agentMessages: AgentMessage[] array in IndexedDB.

Implementation Steps

1. Define custom message type (SDK)

• Declaration-merge CustomAgentMessages to add compactionSummary role
• Shape: \{ role: "compactionSummary"; summary: string; timestamp: number \}

2. convertToLlm handling (SDK)

• In the agent's convertToLlm hook, convert compactionSummary to a UserMessage with summary text wrapped in <compaction_summary> tags
• Already have a convertToLlm in the Agent constructor — extend it

3. Context building (SDK — runtime.ts)

• Use the agent's transformContext hook (or add one)
• Before sending messages to LLM: find the last compactionSummary in the array, slice from there
• Everything before it stays in agent.state.messages (for persistence/display) but is excluded from LLM context

4. Overflow detection (SDK — runtime.ts)

• In handleAgentEvent for message_end: check if the assistant message indicates context overflow
  • stopReason === "error" + error message contains overflow indicators (pi-mono has isContextOverflow())
  • OR usage.totalTokens > contextWindow - reserveTokens (threshold)
• pi-mono ref: isContextOverflow(assistantMessage, contextWindow)

5. Compaction flow (SDK — runtime.ts)

• On overflow trigger:
  1. Get current LLM context messages (from last compaction point or start of array)
  2. Find cut point: walk backwards keeping ~20k tokens of recent messages (chars/4 heuristic)
  3. Send old messages (before cut point) to LLM with summarization prompt via completeSimple()
  4. Insert compactionSummary message at cut point in agent.state.messages
  5. agent.replaceMessages(updatedMessages)
  6. If triggered by overflow, auto-retry the failed prompt

6. Display filtering (SDK — message-utils.ts)

• agentMessagesToChatMessages() already converts AgentMessage[] → ChatMessage[] for UI
• Filter out compactionSummary messages so they never appear in the chat UI

UI Concerns

State During Compaction

• Compaction is an async LLM call that happens between agent turns
• Need a new state flag: isCompacting: boolean on RuntimeState
• UI should show a loading/status indicator (e.g. "Compacting context...") while compaction runs
• Input should be disabled during compaction (same as during streaming)
• isStreaming stays false during compaction — it's a separate state

Event Flow (overflow case)

1. agent streams response
2. message_end → stopReason: "error" (overflow)
3. agent_end fires → onStreamingEnd() saves session
4. detect overflow in message_end or agent_end handler
5. set isCompacting = true, emit state update
6. remove error message from agent state
7. run compaction (LLM call for summary)
8. insert compactionSummary, replaceMessages, save session
9. set isCompacting = false
10. auto-retry: re-send the last user prompt

Event Flow (threshold case)

1. agent streams response successfully
2. message_end → usage shows context near limit
3. agent_end fires → onStreamingEnd() saves session
4. detect threshold breach
5. set isCompacting = true
6. run compaction (LLM call for summary)
7. insert compactionSummary, replaceMessages, save session
8. set isCompacting = false
9. NO auto-retry — user continues manually

UI Updates Needed

• RuntimeState: add isCompacting: boolean
• ChatInterface: disable input when isCompacting
• ChatInterface: show compaction indicator (toast, inline status, or subtle banner)
• Stats bar: update lastInputTokens after compaction to reflect reduced context