🌐 US-Proxy
class="logged-out env-production page-responsive" style="word-wrap: break-word;" >
Skip to content

feat: allow admins to override the chat compaction model - #27151

Merged
ibetitsmike merged 13 commits into
mainfrom
mike/compaction-model-override
Jul 14, 2026
Merged

feat: allow admins to override the chat compaction model#27151
ibetitsmike merged 13 commits into
mainfrom
mike/compaction-model-override

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Chat compaction currently summarizes the conversation with the same model that runs the chat turn, and admins have no way to route compaction to a stronger model when the chat model produces poor summaries.

This adds a deployment-level compaction model override, configured on the AI Settings > Coder Agents page next to the existing title generation override (site config key agents_chat_compaction_model_override, served by the existing /api/experimental/chats/config/model-override/{context} endpoints). When set, compaction summaries are generated with the override model while the chat keeps its own model for the continuation turn.

Behavior details:

  • An unset, malformed, stale, or credential-less override falls back to the chat model with a warning log. A usable override that fails at route or client construction fails the generation visibly instead of silently degrading.
  • The compaction trigger uses the stricter of the chat and override models' context limits so a smaller override model still receives a prompt that fits its window.
  • On a provider mismatch, a copy of the compaction prompt has provider-executed tool history flattened into plain text (matching coder/mux's request sanitization) and unsupported file parts replaced with placeholders before it is sent to the override model.
  • Metrics and chat debug runs record the model that actually produced the summary, including the still-over-limit terminal error, which is recorded before the override client is built and uses the override identity resolved at prepare time.

Closes CODAGT-339

Mux authored this PR on behalf of Mike.

Adds a deployment-wide 'compaction' context to the chat model override
framework. When set, chatd generates compaction summaries with the
override model instead of the chat model. Thresholds use the stricter
of the two context limits, the prompt is re-sanitized for the override
provider, and metrics plus debug runs record the model that actually
generated the summary.
@linear-code

linear-code Bot commented Jul 10, 2026

Copy link
Copy Markdown

CODAGT-339

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 89f78c7369

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/generation_preparer.go Outdated
Resolving the compaction model override previously built the route and
model client on every generation prepare, so a configured but
temporarily broken override could fail ordinary turns that never
compact. Split the resolver: the stored config (whose context limit
feeds the compaction trigger) is resolved on every prepare with soft
fallback, while route and client construction happen only once
compaction is known to run.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c488f6b6f5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codersdk/chats.go
Adding the compaction context to AllChatModelOverrideContexts extends
the invalid-context error detail, so the UnknownContextReturns400
expectations must list it.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 358739808c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/generation_preparer.go Outdated
compactionNeeded at prepare time does not guarantee compaction runs:
decideGenerationAction can return FinishTurn first (completed step,
stop-after tool, max steps), and building the override client during
prepare failed such turns on hard construction errors. Carry the
resolved override config through generationCompaction and build the
client, swap identity, and sanitize the prompt inside
generateCompaction, which only runs for the compact action.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: b866d4fde3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

# Conflicts:
#	coderd/x/chatd/ARCHITECTURE.md
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: c95e1053ac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ction

Align the cross-provider compaction prompt pass with coder/mux's
flattenProviderExecutedToolParts: rewrite provider-executed tool calls
and results into plain text parts instead of dropping them, so the
summary keeps server-side tool content (for example web_search results)
while still shedding the provider-specific wire shape other providers
reject. Anomalous provider-executed parts outside assistant messages
are still dropped.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f9bbc7b181

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/generation.go Outdated
The compaction override resolver folds the admin-configured reasoning
effort into the override config's call options, but the summary call
never consumed them. Convert the override's provider options and
resolved reasoning effort when building the override client and pass
them on the summary fantasy.Call, so a compaction model's configured
effort actually applies.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: b32de95342

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…anitize.go

The file never defined the compaction prompt; the summary prompt text
lives in chatloop/compaction.go. It sanitizes the already-built prompt
messages for the compaction override model, so name it after its entry
point sanitizeCompactionPrompt and the chatsanitize package it wraps.
…the override model

The still-over-limit terminal error is recorded at decision time, before
the compact action builds the override client, so it used the chat model
labels from the prepared options while the compact action's own events
used the built override client's labels. A failing override compaction
then split across two metric series: success on the override model,
error on the chat model.

Prepare-time override resolution already computes the normalized
provider and validates ResolveModelWithProviderHint, so keep that
identity in resolvedCompactionOverride and source every compaction
metric label from one compactionMetricIdentity helper.
@ibetitsmike
ibetitsmike marked this pull request as ready for review July 14, 2026 08:56
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@coder-tasks

coder-tasks Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Documentation Check

Updates Needed

  • docs/ai-coder/agents/models.md - The Model overrides section documents the admin-only override contexts (General, Explore, Title generation, Root). This PR adds a new admin-only Compaction override (site config agents_chat_compaction_model_override, UI title "Compaction model" on Admin settings > AI > Coder Agents), but the "configurable contexts" table and surrounding text don't mention it. Add a Compaction / Admin only row describing that it summarizes long conversations when they approach the context limit, and note the fallback to the chat model when unset.

Automated review via Coder Agents

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e4a684d89b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/generation_preparer.go
The compaction trigger uses the stricter of the chat and override
models' context limits, but the still-over-limit check compared
against the chat model's limit. A stricter override then looped
through repeated compactions instead of failing the turn.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 9dc171de10

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike merged commit a567f6a into main Jul 14, 2026
28 of 29 checks passed
@ibetitsmike
ibetitsmike deleted the mike/compaction-model-override branch July 14, 2026 14:57
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 14, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants