feat: notify users when AI spend crosses the budget threshold - #27346
Conversation
e4e2867 to
239eaf0
Compare
|
/coder-agents-review |
|
Chat: Review posted | View chat Review historydeep-review v0.9.0 | Round 2 | Last posted: Round 2, 18 findings (3 P2, 4 P3, 6 Nit, 5 Note), COMMENT. Review Finding inventoryFinding inventory - PR #27346Findings
Contested and acknowledgedCRF-1 (P2, aibridgedserver.go:439) - edge-triggered best-effort delivery
CRF-15 (Note, aibridgedserver.go:147) - nil Enqueuer no-op
Round logRound 1Netero-only first pass: no findings (2 Notes, mechanical floor clean), proceeded to panel. Round 2 updateBLOCKED by churn guard. New head bef2d0a (base 60c20be); +940 -3, migration renumbered 000551 -> 000552. Author (evgeniy-scherbina) plus human reviewers (ssncferreira, johnstcn) worked through every finding. Churn guard classification: 9 addressed (CRF-2 clock arm, CRF-3, CRF-4, CRF-5, CRF-8, CRF-10, CRF-11, CRF-12, CRF-13), 2 acknowledged with reasoning (CRF-1, CRF-6), 1 contested (CRF-15), 1 deferred with ticket (CRF-7 -> AIGOV-448), 5 silent (CRF-9, CRF-14, CRF-16, CRF-17, CRF-18) with content-free replies ("I think it's fine", "it's okay") and no code change. Any silent finding blocks the panel: no reviewers spawned, no Netero, no fix verification this round. Posted a COMMENT listing the 5 silent findings and what unblocks each. Fix verification deferred to the next round once the silent items get a substantive response or change. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
This is a clean, well-scoped feature: edge-triggered detection folded into the existing spend transaction, per-day DB dedupe, no new config surface, and both the AGPL and enterprise call sites wired. Test density is high (67%) and the boundary tables (exactly-on, just-below, already-at, already-above, crosses-both) genuinely fail if the crossing math breaks. The limit-reached copy even names its real consequence, and enforcement agrees with it: the 100% crossing fires at the same spend >= limit boundary that IsBudgetExceeded blocks on.
The panel's central concern is that the notification signal, which is the entire point of this PR, is not durable. It is edge-triggered (fires on exactly one interception) and both detected best-effort, so it can be lost or misfired three ways: a transient failure on the crossing interception (CRF-1), an unsound oldSpend = newSpend - cost reconstruction under cross-day concurrency and a createdAt-vs-clock.Now() period mismatch (CRF-2), and enforcement keeps blocking the user either way, so the failure mode is "blocked with no explanation." These converged from many angles (Knov, Mafuuu, Nami, Meruem at P2; Knuckle, Pariston, Hisoka, Komugi, Melody, Chopper below). I verified the load-bearing premises against the code: InTx(..., nil) runs at READ COMMITTED, IncrementUserAIDailySpend only row-locks the single (user, group, day) row, the spend day comes from createdAt while the period comes from s.clock.Now(), and subjectNotifier genuinely cannot read groups (which is why the path reaches for the wildcard actor in CRF-3). The durable fix for CRF-1 and CRF-2 is the same shape: make detection level-triggered from observed state (compare the highest threshold already notified this period against current spend, let the per-day dedupe drop repeats) and serialize the read+decision per user/period, rather than trusting a one-interception delta.
None of these block a merge on their own (no P0/P1), but CRF-1 through CRF-3 deserve a decision before this ships, not a follow-up assumption. Counts: 3 P2, 4 P3, 6 Nit, 5 Note.
Process: the PR description is empty (just the AI-contribution comment), so the reasoning the diff cannot show, why 85%, why best-effort inside the transaction, how duplicates are deduped, and the per-user-override behavior, is lost to the next reader. Per the repo PR style guide, please add it. The commit subject itself is good.
Bisky put the test gap best: "the suite is afraid of one thing, and it's the thing this PR is actually about" - no test exercises a failing detect or a failing notify, so changing return nil to return detectErr inside the transaction would roll back committed spend and no test would turn red.
This review was generated by Coder Agents.
🤖 This review was automatically generated with Coder Agents.
| This limit is set by your group **{{.Labels.group_name}}**. | ||
| {{- else if eq .Labels.limit_source "user_override"}} | ||
|
|
||
| This limit is set specifically for your account. |
There was a problem hiding this comment.
The user does not need to know whether their limit is set via a Group or a User Override, that is something the Admin needs to know. I think we can drop this sentence and the user_override label.
In fact, I'm not sure whether users even need to know what their effective group is 🤔
There was a problem hiding this comment.
@ssncferreira We discussed this before, and Agent raised the same concern.
I think there are two reasonable options:
- Keep it as is. I think it's fine—the notification still gives the user useful information about why they were blocked and what action their manager or admin needs to take to unblock them.
- Remove both
limit_sourceandgroup_name. Showing only the group name can be misleading because it implies the request was blocked by the group budget, which isn't necessarily true.
I'd prefer the first option, but up to you.
There was a problem hiding this comment.
Showing only the group name can be misleading because it implies the request was blocked by the group budget, which isn't necessarily true.
I don't quite agree with this. A budget is always associated with a group, either via a group budget or a user override, so a request is always blocked by a group (the only difference is how it is configured).
IMO, I think we can keep the group, but I would prefer removing the user_override label. For me, the sentence This limit is set specifically for your account. is confusing and don't adds value.
There was a problem hiding this comment.
A budget is always associated with a group, either via a group budget or a user override, so a request is always blocked by a group (the only difference is how it is configured).
That was my initial approach, but Agent raised this concern: #27346 (comment), and I agreed with it. "Effective group" felt out of place in this notification template.
This limit is set specifically for your account. is confusing and doesn't add value.
I think it still provides value. Imagine the Engineering group has a limit of $2,000, and everyone (users and admins) is aware of it. A user suddenly gets blocked at 1000$ but doesn't know why. This message provides a useful hint that they're likely hitting an individual limit rather than the group limit.
That said, I'm also okay with removing both limit_source and group_name.
There was a problem hiding this comment.
This message provides a useful hint that they're likely hitting an individual limit rather than the group limit.
But this is not relevant for the regular user, and the user doesn't need to know whether the budget is set via the group or via the user override (it doesn't even need to know about this difference). For the user, it sees that it is for group X, notifies the admin, and the admin goes to group X and sees the user information (which shows if it is via group budget or user override).
I say, keep the group_name and remove limit_source.
There was a problem hiding this comment.
I still agree with Agent's comment:
Effective group: {{.Labels.group_name}}leaks internal resolver jargon and is actively misleading when the limit comes from a per-user override.
That said, I'll change it.
There was a problem hiding this comment.
@ssncferreira I reverted to previous version.
| $$You have used more than {{.Labels.threshold}}% of your {{.Labels.period}} AI budget ({{.Labels.limit}}). | ||
| {{- if eq .Labels.limit_source "group"}} | ||
|
|
||
| This limit is set by your group **{{.Labels.group_name}}**. |
There was a problem hiding this comment.
Maybe?
| This limit is set by your group **{{.Labels.group_name}}**. | |
| This limit is set by your AI Governance Effective Group **{{.Labels.group_name}}**. |
There was a problem hiding this comment.
- To be honest, I don't like "Effective". It feels like an internal term. Do we expose it anywhere in the UI?
- "AI Governance" is better, but it still implies there are AI Governance-specific groups, which isn't true.
There was a problem hiding this comment.
@ssncferreira I reverted the template to its previous version, so this comment is no longer relevant unless you want to change the template text.
| {{- else if eq .Labels.limit_source "user_override"}} | ||
|
|
||
| This limit is set specifically for your account. | ||
| {{- end}}$$, |
There was a problem hiding this comment.
I thought about that too, but it could be a bit tricky given the different budget periods.
Do you want to hardcode monthly for this?
| // pre = 850_500 - 1000 = 849_500 (< 850_000) | ||
| // post = 850_500 (>= 850_000) -> crosses. |
There was a problem hiding this comment.
nit: for clarity, I would suggest using integer values and the associated USD cost in the comments.
There was a problem hiding this comment.
I don't think "associated USD cost" would be clearer. For example:
// pre = 849_500 ($0.8495, < $0.85)
// post = 850_500 ($0.8505, >= $0.85) -> crossesI can scale it to something like:
// pre = 84_950_000 ($84.95, < $85)
// post = 85_050_000 ($85.05, >= $85) -> crossesBut I'm not sure it's worth it.
There was a problem hiding this comment.
Actually, I changed my mind—I refactored the test, and I think it looks cleaner now. Let me know what you think.
| // TestRecordTokenUsageBudgetLimitReachedNotification verifies that crossing the | ||
| // 100% limit enqueues the limit-reached notification, and that a single | ||
| // interception crossing both the warning and limit thresholds enqueues both. | ||
| func TestRecordTokenUsageBudgetLimitReachedNotification(t *testing.T) { |
There was a problem hiding this comment.
I think this can probably be merged with the previous test: TestRecordTokenUsageBudgetWarningNotification
For cleaner tests and to avoid bigger tests
There was a problem hiding this comment.
make sense, done
|
|
||
| // TestRecordTokenUsageBudgetNotificationUserOverride verifies that when the | ||
| // limit comes from a per-user override, the notification reflects that. | ||
| func TestRecordTokenUsageBudgetNotificationUserOverride(t *testing.T) { |
There was a problem hiding this comment.
This one can also probably be merged with the previous TestRecordTokenUsageBudgetWarningNotification and TestRecordTokenUsageBudgetLimitReachedNotification. This test file is getting quite big...
There was a problem hiding this comment.
This test is no longer relevant, so I removed it.
| return nil, xerrors.Errorf("compute AI budget period: %w", err) | ||
| } | ||
|
|
||
| spend, err := tx.GetUserAISpendSince(ctx, database.GetUserAISpendSinceParams{ |
There was a problem hiding this comment.
I would prefer to do this on the caller side, just to have all database operations in the same place. Instead of passing tx we would just pass the spend.
There was a problem hiding this comment.
Good idea, but my concern is that it doesn't fit well with our best-effort semantics.
One option would be to move this logic into the transaction:
period, err := budget.CurrentPeriod(createdAt, s.budgetPeriod)
if err != nil {
s.logger.Error(ctx, "compute AI budget period for threshold detection", slog.Error(err))
return nil
}
spend, err := tx.GetUserAISpendSince(ctx, database.GetUserAISpendSinceParams{
UserID: intc.InitiatorID,
EffectiveGroupID: cost.effectiveGroupID.UUID,
PeriodStart: period.Start,
})
if err != nil {
s.logger.Error(
ctx,
"get user AI spend for threshold detection",
slog.F("initiator_id", intc.InitiatorID),
slog.Error(err),
)
return nil
}However, these early returns would make it harder to extend the transaction with additional operations in the future.
Alternatively, I could log the errors without returning early, but that could leave period and spend undefined or invalid later in the scope, which also doesn't seem ideal.
I can move:
period, err := budget.CurrentPeriod(createdAt, s.budgetPeriod)outside the transaction, but the issue remains.
detectBudgetThresholdCrossings nicely encapsulates these concerns for us.
| // Threshold detection is best-effort: a failed read must not roll back | ||
| // the committed spend, so the error is logged rather than propagated. |
There was a problem hiding this comment.
I don't think we are testing this 🤔 this would be nice to have in the tests
There was a problem hiding this comment.
I wasn't sure this needed a test, but I added one anyway.
It still uses mockdb, but I think that's sufficient.
9e4e3a5 to
0e7d916
Compare
There was a problem hiding this comment.
Obligatory reminder to check migration number before merge!
There was a problem hiding this comment.
I just synced with main
| switch p { | ||
| case "day": | ||
| return "daily" | ||
| case "week": | ||
| return "weekly" | ||
| case AIBudgetPeriodMonth: | ||
| return "monthly" | ||
| case "year": | ||
| return "yearly" | ||
| default: | ||
| return string(p) |
There was a problem hiding this comment.
This would be simple enough to generalize and move to a util function:
if p[len(p)-1] == 'y' {
p[len(p)-1] = 'i'
}
return p + 'ly'
I looked in github.com/dustin/go-humanize and it has some English-specific functions, but not what we ware looking for.
There was a problem hiding this comment.
To be honest, I don't like this approach:
if p[len(p)-1] == 'y' {
p[len(p)-1] = 'i'
}
return p + "ly"It feels a bit hard to read.
I prefer the current approach. If you'd like, I can simplify it to something like this instead:
// Adjective renders the period as the adjective used in user-facing text (e.g. "monthly").
func (p AIBudgetPeriod) Adjective() string {
switch p {
case "day":
return "daily"
case "week", AIBudgetPeriodMonth, "year":
return string(p) + "ly"
default:
return string(p)
}
}Co-authored-by: Cian Johnston <cian@coder.com>
|
/coder-agents-review |
There was a problem hiding this comment.
Round 2. Thanks for the thorough pass: the PR description now explains the design, the clock-source arm of CRF-2 is fixed (period derived from the interception createdAt), CRF-3 moved to AsNotifier, CRF-4 templates the period via Adjective(), CRF-5 reworked the group phrasing, and CRF-8/10/11/12/13 are addressed. CRF-6 and CRF-1 are acknowledged with real reasoning, and CRF-7 is deferred to AIGOV-448. That is genuine engagement and most of it looks right on a read-through.
The panel did not run this round, and this is not an approval. The re-review gate is blocked because five findings received a content-free reply ("I think it's fine", "it's okay", "yeah, it's fine") with no reasoning and no code change. A bare acceptance is not a resolution: without a stated reason the panel cannot tell an intentional decision from an overlooked one, and neither the bot nor the author can accept a known gap as permanent on the reader's behalf. Each of these needs one line of reasoning, or a change, to unblock the panel:
- CRF-9 (Nit, migration up.sql) - warning body says "more than {threshold}%" but the crossing fires at
>=, so it can read "more than 85%" at exactly 85%. Either reword to "at least"/"reached", or say why the boundary wording is acceptable. - CRF-14 (Note, notifications.go) -
budgetNotificationsCreatedBy = "aigateway"is a third spelling for this subsystem (packageaibridgedserver, path AI Bridge) and is persisted with every notification. Confirm the label is deliberate, or align it. - CRF-16 (Note, notifications.go) - the finding only asked you to confirm the 85%/100% thresholds are intentionally non-tunable. One sentence confirming that closes it; "yeah, it's okay" without the reason does not.
- CRF-17 (Note, notifications.go) - a single interception crossing both thresholds sends "approaching" and "reached" in the same instant. Confirm that double-send is intended (and why), or suppress the warning when the limit-reached fires together.
- CRF-18 (Note, notifications.go) -
group_namerenders inside**...**, so a group named*prod*ora_b_cshows garbled emphasis (cosmetic, not injection). State that this is acceptable, or escape it.
Once these have a substantive response or a change, the next round runs the full panel to verify the round-2 fixes (all currently recorded as claimed-but-unverified) and to weigh the acknowledged best-effort decision on CRF-1 and the contested CRF-15. Note also that CRF-5's group-naming wording is still under active discussion with the human reviewers; whatever lands there is what the panel will assess.
This review was generated by Coder Agents.
🤖 This review was automatically generated with Coder Agents.

Implements: https://linear.app/codercom/issue/AIGOV-289/notify-users-and-admins-on-budget-warning-and-limit-reached
Notify users when their AI spend crosses a budget threshold for their effective group. Two thresholds are covered: a warning at 85%, and a limit-reached notification at 100%.
Detection runs on the post-response path, right after the interception's cost is added to the user's daily spend. It reads the user's AI spend on the same transaction where token usage is recorded and AI daily spend is incremented, and derives the pre-interception total by subtracting this interception's cost. In case of
oldSpend < threshold && newSpend >= threshold- notification is sent. A single interception that crosses both thresholds enqueues both notifications.Detection and delivery are best-effort: a failure is logged and never fails usage recording. The payload uses only stable values (the threshold percentage and the spend limit, not the exact spend), so duplicate enqueues are deduplicated by the notification system.
The two templates are added via migration and appear in each user's notification settings under the "AI Budget" group.
Admin notifications (owners and user admins) are a follow-up: #27415.
Screenshots: