feat: cap tool output to fit the model context window - #26637
Merged
Conversation
Large tool results, most often multi-megabyte MCP responses, were persisted and replayed to the model verbatim, overflowing the prompt and wedging the chat. Cap each local tool result in executeSingleTool to a token-aware byte budget (a fraction of the model context window) with UTF-8-safe head-and-tail truncation that preserves the start and end of the output. Covers built-in, global MCP, and workspace MCP tools; binary media data is left untouched.
johnstcn
reviewed
Jun 23, 2026
johnstcn
left a comment
johnstcn
left a comment
Member
There was a problem hiding this comment.
I think a truncated and incomplete tool call result is less likely to be useful than an explicit error saying e.g. "tool call was successful but output would exceed token budget". In this case, the model will have a signal that it does not need to retry the tool call.
Member
Author
|
@johnstcn I considered that, but I think that might be dangerous. e.g. you run a bash command, the model thinks it never ran because of the tool-output. It tries to run again and state is mutated. |
johnstcn
approved these changes
Jun 23, 2026
johnstcn
left a comment
johnstcn
left a comment
Member
There was a problem hiding this comment.
It's impossible to know until we try it, I suppose!
make gen adds the new coderd_chatd_tool_result_truncated_total metric to the scanned metrics fixture and the Prometheus metrics reference.
Docs preview📖 View docs preview for |
kylecarbs
changed the title
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Local tool results were persisted and replayed to the model verbatim, with no size cap. A single oversized result, most often a multi-megabyte response from an MCP tool, overflows the prompt on the next request. Every retry rebuilds the same history and fails the same way, leaving the chat wedged in
error. Auto-compaction is reactive (token usage is only known after a response), so it can't catch a single result that blows the very next request.Fix
Cap every locally-executed tool result at its single choke point,
executeSingleToolinchatloop, so the cap covers built-in tools, global (deployment-pinned) MCP, workspace MCP, and provider runners uniformly. Because this runs before the result is published to the live stream and before it is committed, the SSE preview, the persisted message, and the model replay all see the same bounded output.The budget is token-aware: a single tool result may use at most half the model's context window (
~4 bytes/token), with a16KBfloor and a64KBdefault when the window is unknown. Truncation keeps the head and tail of the output and replaces the middle with a marker telling the model how much was removed and to narrow its query; it is UTF-8 safe and never exceeds the budget. Binary mediaDatais passed through untouched (only the text payload is bounded).A
coderd_chatd_tool_result_truncated_total{provider,model,tool_name}counter and a warning log record each truncation.Out of scope
executeSingleTool./tool-resultsAPI are validated as JSON elsewhere.Implementation notes
coderd/x/chatd/chatloop/tooltruncate.go:toolResultByteBudget(contextLimitTokens)andtruncateToolResultText(text, maxBytes)(pure, unit-tested).chatloop.go: addedContextLimittoExecuteLocalToolsOptions; threaded a computed byte budget throughexecuteToolsintoexecuteSingleTool, whereresp.Contentis capped for the text, media-text, and error branches.generation.go: passesContextLimit: prepared.ContextLimitFallback(the model's configured context limit).metrics.go: newToolResultTruncatedTotalcounter +RecordToolResultTruncated.tooltruncate.go(toolResultContextDivisor = 2,bytesPerTokenEstimate,minToolResultBytes,defaultToolResultBytes).Verified:
go build ./coderd/x/chatd/...,go test ./coderd/x/chatd/chatloop/..., and thechatdtest binary compiles.Resolves CODAGT-678
Generated by Coder Agents on behalf of @kylecarbs.