[None][fix] Update Scaffolding MCP transport and KV cache truncation - #19055

Summary

Fixes #19329.
Fixes #19281.
Fixes #19319.

This PR consolidates three separately tracked Scaffolding fixes into one review/CI target, following the contributor's discussion with WeiHaocheng. It supersedes #19014 and incorporates the code from #19282.

1. MCP Streamable HTTP migration (#19329)

• Add a shared core helper for opening and initializing Streamable HTTP MCP sessions.
• Migrate core MCPWorker, per-scope ApiaryMCPWorker, and the contrib MCP client.
• Migrate Coder/Apiary and Weather, E2B, Google Search, Google Scholar, Tavily, Fetch Webpage, and WordLlama example servers to the official SDK Streamable HTTP APIs.
• Preserve bearer authentication, explicit client_id sandbox identity, image selection, attach/detach behavior, and application/session cleanup.
• Update example/default URLs to /mcp, including matching the Coder server's default port, and require mcp>=1.23.0,<2.0.0.
• Remove legacy sse_client, SseServerTransport, and legacy route references from Scaffolding examples and implementation.

The MCP specification replaces the legacy HTTP+SSE transport with Streamable HTTP. SSE framing itself remains supported within Streamable HTTP. Existing deployments using legacy-only MCP servers must migrate/configure compatible endpoints.

2. KV truncate endpoint (#19281)

• Send message-based cache hints to the registered /_control/kv_cache/truncate endpoint instead of kv_cache_hints.
• Normalize trailing slashes and strip the optional trailing /v1.
• Preserve request payloads, Authorization headers, and response handling.
• Leave server compatibility routes and the token-level truncate API unchanged.

3. Retain only the system prefix (#19319)

The agreed agent retention policy is to retain the reusable system prompt, not the task-specific user/assistant/tool context.

• DropKVCacheTask now retains only consecutive leading system messages, stopping at the first non-system message. It does not collect later system messages across gaps.
• Preserve the target conversation; only the retained-message selection changes.
• Audit the principal ChatTask-based workflows: Coder, multi-round chat, ODR brief/planning/research/compression, IterResearch, ToT Research, and contrib MCP already use system roles for their leading system instructions.
• The audit found IterResearch's intermediate/final system templates embedded the per-task question/date (and intermediate tool summary). Move these input sections into their corresponding user templates, keeping the instructions, input content, and system prompt IDs.
• Leave ODR's final-report request as a user turn: it is a stage-transition instruction appended to the existing conversation, not a reusable leading system prefix. Raw GenerationTask/completion workflows are not converted to chat roles.

This does not change the engine's block-granular truncate implementation or claim exact byte/token-boundary eviction, GPU-memory release, or a measured performance improvement. Prompt-role changes in IterResearch should receive behavioral review/CI validation.

Validation of the combined branch

• 21 focused tests passed in an isolated local harness: 20 KV endpoint/retention/prompt tests plus the Coder missing-client-ID test. The harness imports the actual repository task/worker modules, stubs unrelated native TensorRT/CUDA imports, and bypasses the global unittest conftest. This is not a full supported-runtime test run.
• Tests cover root and /v1 base URLs with/without trailing slashes and API keys; preservation of target messages and headers; multiple user turns; tool turns; missing/multiple/late system messages; worker dispatch; and task-independent IterResearch system templates with preserved task inputs.
• Local Streamable HTTP MCPWorker/server smoke test passed: initialization, tool discovery, two tool calls, and session teardown.
• All applicable repository pre-commit checks passed for the complete PR file set.
• Syntax parsing passed for all 29 changed Python files.
• Legacy SSE import/route scan under Scaffolding implementation/examples returned no matches.
• Normal pytest collection was attempted but remains blocked locally by ModuleNotFoundError: No module named 'cuda' during package initialization. No GPU-backed KV eviction or model-quality benchmark was run.

Suggested CI targets:

pytest tests/unittest/scaffolding/test_kv_cache_hint.py tests/unittest/scaffolding/test_mcp_worker.py tests/unittest/scaffolding/test_coder_mcp.py

The existing unittest/scaffolding entries in l0_cpu.yml and l0_h100.yml already include these tests. Latest upstream main (b941c5c15) was merged without conflicts; existing review commits/history are preserved.

Dev Engineer Review

• Migrates Scaffolding MCP clients, workers, servers, examples, and documentation from SSE to Streamable HTTP.
• Adds open_mcp_session with session and transport cleanup.
• Updates Coder authentication, Apiary session handling, attach/detach behavior, health checks, and lifecycle management.
• Updates endpoints to /mcp and constrains MCP dependencies to >=1.23.0,<2.0.0.
• Fixes the KV-cache truncate endpoint and retains only reusable leading system-message prefixes.
• Verify external clients, dependency resolution, host binding, and lifecycle behavior.
• Two L0 merge-request pipelines failed. Failure resolution and CI re-trigger remain pending.
• Focused pytest execution was blocked because PyTorch was unavailable.

QA Engineer Review

• tests/unittest/scaffolding/test_mcp_worker.py covers Streamable HTTP setup, health checks, worker lifecycle, tool dispatch, errors, and shutdown.
• tests/unittest/scaffolding/test_coder_mcp.py verifies HTTP 400 JSON rejection when client_id is missing.
• tests/unittest/scaffolding/test_kv_cache_hint.py covers endpoint normalization, truncate payloads, system-prefix retention, handler forwarding, and iter-research prompt isolation.
• The unittest/scaffolding suite is listed in tests/integration/test_lists/test-db/l0_cpu.yml and l0_h100.yml.
• No manual QA list changes are shown.
• Coverage is needs follow-up because focused tests did not run locally and L0 pipelines failed.

Per-File QA Perspective

• Benchmark, Coder, and research runner files: Verify documented and generated URLs use reachable /mcp endpoints.
• MCP README files: Verify startup, authentication, session, and endpoint guidance matches runtime behavior.
• coder_mcp.py: Verify authentication, Apiary sessions, attach/detach, reaping, shutdown, health, and JSON error paths.
• MCP example servers: Verify Streamable HTTP startup, host and port handling, and removal of SSE routes.
• Example pyproject.toml files and requirements.txt: Verify package metadata, dependencies, and MCP version constraints resolve consistently.
• mcptest.py and the trace-replay README: Verify prompts and local endpoint instructions use /mcp.
• Scaffolding contribution files: Verify /mcp configuration, documentation, and Streamable HTTP client initialization.
• mcp_client.py: Verify open_mcp_session closes the session and transport on normal and error exits.
• worker.py: Verify queueing, dispatch, shutdown, sandbox selection, and error behavior.
• task.py and prompts.py: Verify system-prefix retention and prompt-context isolation.
• test_mcp_worker.py, test_coder_mcp.py, and test_kv_cache_hint.py: Verify the listed behavior. The unittest/scaffolding suite is included in the CPU and H100 CI test lists.