feat: add CSV export for AI spend data - #27491
Conversation
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
|
/coder-agents-review |
|
Chat: Review posted | View chat Review history
deep-review v0.9.0 | Round 1 | Last posted: Round 1, 17 findings (1 P1, 4 P2, 5 P3, 3 Nit, 4 Note), REQUEST_CHANGES. Review Finding inventoryFinding inventory, PR #27491Findings
Round logRound 1Panel (18 reviewers + Netero first pass + 2 wildcards). Netero first pass: P3/Note only, mechanical floor clean, panel proceeded. Findings: 1 P1, 4 P2, 6 P3, 3 Nit, 4 Note. Reviewed against 0f1eafa..2cb0bba. CI green (23 passed, 9 skipped). CSV formula injection (CRF-1) converged across Hisoka, Kurapika, Ryosuke (P2) and Meruem, Chopper, Razor (P3); higher wins, model/provider verified unconstrained (stored verbatim from intercepted request, never rejected). Missing index (CRF-3) rated P1 by Killua, P2 by Knuckle; kept P1 per the drop/downgrade gate: the consequence (full scan of the highest-volume table, unbounded by anything but multi-month retention, DB-connection pinned, triggerable by any in-org member via the post-filter) is set by consequence not by the experiment gating (probability). CRF-3 and CRF-4 compound into a low-privilege DB resource-exhaustion vector. Null-group drop (CRF-2) kept P2 over the lower Note ratings: consistency-with-rollup is not correctness for a money artifact, and the swagger promises org spend without a caveat. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
Clean, well-scoped feature: one authorized read endpoint, its SDK client, and dense tests (62.9% test density, no drive-by refactors). The per-row authorization is the right call and is genuinely proven: TestExportOrganizationAISpendRoleAccess shows admins see every row and a plain member sees only their own via the fetchWithPostFilter owner-scoped RBAC object, and CustomPeriodHalfOpen pins the [start, end) boundary with rows seeded at both edges. The panel also confirmed the raw-table source is forced (the daily rollup has no model/provider/token columns) and that the spend column stays consistent with the rollup because both read the same effective_group_id/cost_micros snapshot.
Requesting changes on the storage and correctness surface, not the design. Severity count: 1 P1, 4 P2, 6 P3, 3 Nit, 4 Note.
The two that compound and are worth fixing together: this export walks away from the index the rollup path relies on (CRF-3) and drops the per-call cap every sibling AI-spend endpoint enforces (CRF-4). Because authorization is a post-filter, any in-org member's request runs the full-org aggregation, full-scans the highest-volume table, and materializes the whole result in memory before returning their single row. Knuckle put it well: "This is the loan... Interest rate: every export gets slower as the table grows, linearly, with no ceiling short of retention."
The rest: CSV formula injection from user-controlled model/provider (CRF-1), a silent money undercount from the inner group join (CRF-2), a month-boundary test flake (CRF-9), the retention guard misfiring on the default window and keying off the wrong timestamp (CRF-5, CRF-6), and CSV/contract polish (CRF-7, CRF-8, CRF-10).
Process: the PR description still tells consumers to use start/end, but commit d216b5f renamed the params to period_start/period_end in the handler, SDK, swagger, and docs (CRF-11, P3). The description also omits the retention-window constraint the same commit added. The generated docs are correct; only the human-facing description is stale. Please update it, since it is exactly what a reviewer reads to understand the contract.
No agent-accepted permanence: CRF-2 (which spend the export omits) and the CRF-3/CRF-4 scale posture are product/operational decisions. If any is accepted as-is rather than fixed, that needs a human decision (surface the omission or document the caveat in swagger, and/or file a ticket), not a silent defer.
🤖 This review was automatically generated with Coder Agents.
| COALESCE(SUM(tu.cost_micros), 0)::BIGINT AS cost_micros | ||
| FROM aibridge_token_usages tu | ||
| JOIN aibridge_interceptions ai ON ai.id = tu.interception_id | ||
| JOIN groups ON groups.id = tu.effective_group_id |
There was a problem hiding this comment.
P2 [CRF-2] The inner JOIN groups ON groups.id = tu.effective_group_id silently drops real spend, so this "organization AI spend" CSV can undercount money with no signal. (Mafuuu P2, Knuckle P3)
The raw table is the superset that contains spend the rollup omits... an operator exporting to reconcile or charge back AI cost gets a total that is systematically short by the entire spend of users who resolve to no effective group.
Two classes vanish: token usage with effective_group_id IS NULL (a real, tested state carrying non-zero cost, per ExcludesNullEffectiveGroup), and usage referencing a hard-deleted group (no FK; DeleteGroupByID deletes the row, orphaning historical effective_group_id). "Consistent with the rollup" is the defense and it holds for consistency, but the PR chose the raw table because it is the superset, then filtered it back to the rollup's blind spot. The swagger @description promises org spend with no caveat. Whether ungrouped/orphaned spend belongs here is a product decision: surface it (a sentinel/ungrouped group value) or document the exclusion in swagger. Knov also notes the export keeps cost_micros <= 0 rows the rollup skips, a second divergence from the "consistent" claim. Intentional-but-flagged, proven by test; not a silent accept.
🤖
There was a problem hiding this comment.
Ungrouped spend only happens for users with no org membership, and the daily table can't represent it either since effective_group_id is NOT NULL there. Regarding cost_micros, the only advantage I see is that the CSV could distinguish a priced model that resulted in zero cost from an unpriced one. But if the price is updated during the budget period (currently only via a release), usage recorded before and after it would aggregate into the same row, so we would lose that distinction anyway. Let me know what you think.
|
|
||
| // Build the full CSV in memory so the response is sent once with a | ||
| // Content-Length, and so a write error surfaces as a 500 before any | ||
| // status is written. The row count is bounded by the 31-day period cap. |
There was a problem hiding this comment.
P2 [CRF-4] The full aggregation is buffered in memory with no row cap, the comment justifies it with a bound that does not exist, and the post-filter runs the full-org query for every caller. (Knov P2, Meruem P2)
The comment claims "The row count is bounded by the 31-day period cap." The 31-day cap bounds the time window, not cardinality. Result rows are one per distinct (initiator_id, effective_group_id, provider, model) tuple... There is no LIMIT, unlike the sibling endpoints in this same file which explicitly cap output (maxOrganizationGroupsAISpendGroupIDs = 100).
Separately, fetchWithPostFilter runs ExportOrganizationAISpend unscoped and materializes every org row into Go (a second copy in rbac.Filter, then the bytes.Buffer), so a plain member exporting "their own" spend still forces the whole-org aggregation and buffer. This compounds with CRF-3 (the full scan) into a low-privilege resource-exhaustion vector. Nearly the whole panel flagged the false comment (Hisoka, Pariston, Leorio, Ryosuke, Razor, Chopper, Kurapika). Fix: bound the row count (cap distinct users like the siblings cap IDs, or paginate/stream), and state the real bound in the comment.
🤖
There was a problem hiding this comment.
Members can't hit this endpoint anymore. We still build the whole CSV in memory, which is what the RFC describes (link), and it's also what lets us fail with a clean 500 rather than a truncated file. fetchWithPostFilter already loads every row before the handler runs, so streaming wouldn't avoid that allocation, but we can add it later if this proves problematic. Also worth noting the endpoint is capped at 31 days, which helps bound the result size.
cc4d080 to
f39e6d9
Compare
|
|
||
| // The export aggregates the whole organization, so require organization-wide | ||
| // read rather than letting the per-row filter narrow it to the caller. | ||
| if !api.Authorize(r, policy.ActionRead, rbac.ResourceGroupMember.InOrg(org.ID)) { |
There was a problem hiding this comment.
Scoping this endpoint only to organization-level admin permissions as per the RFC.
This is slightly inconsistent with the group members spend endpoint, which for regular members, returns just their spend. But this is a different scope, which seems admin-related.
f39e6d9 to
4eed9b8
Compare
Description
Adds
GET /api/v2/organizations/{organization}/ai/spend/export, returningtext/csvwith per-user, per-group, per-model, per-provider aggregated AI spend. The data is built from the raw AI Gateway token usage tables rather than theai_user_daily_spendrollup, but stays consistent with it: spend is attributed through the token usage's effective group and bucketed by the token usagecreated_at, the same values the daily rollup derives from.The period defaults to the current UTC month, narrowed to the configured AI Gateway retention window when the month begins before retained data does. Explicit
period_start/period_endparams must be provided together, are interpreted as UTC, and may span at most 31 days. Unlike the default period, an explicit period that begins before the retention window is rejected rather than narrowed. Every row echoes the applied bounds, so a narrowed window is visible in the export.The endpoint requires organization-level admin permissions.
Changes
ExportOrganizationAISpendquery aggregatingaibridge_token_usagesjoined toaibridge_interceptions, scoped to the organization via the effective group, resolving the username, group name, and organization name alongside their IDs.exportOrganizationAISpendhandler and route, gated by theaigateway-cost-controlexperiment and theAIBridgefeature, returning the CSV in a single response.ExportOrganizationAISpendcodersdk client method.ResourceGroupMemberread, since the export aggregates every user in the organization. The per-row filter stays indbauthzas defence in depth.aibridge_token_usages (effective_group_id, created_at), which the period and group predicates otherwise cannot use.Closes https://linear.app/codercom/issue/AIGOV-293/add-csv-export-for-ai-spend-data
Note
Generated by Coder Agents on behalf of @ssncferreira