[TRTLLM-16220][feat] Enable KVCacheManagerV2 by default for Nemotron multimodal models by eopXD · Pull Request #19140 · NVIDIA/TensorRT-LLM

eopXD · 2026-09-14T07:54:33Z

Description

Fix from #18885 now allows KVCacheManagerV2 to successfully deal with multimodal inputs. Nemotron multimodal models should safely allow v2_kvcm as default now.

Use KVCacheManagerV2 for Nemotron-H Nano VL — and its NemotronH_Nano_Omni_Reasoning_V3 / NemotronH_Omni_Reasoning_V3 siblings, which share the class.

Test Coverage

Question a reviewer will ask	Test
Do all three registered architectures resolve `"auto"` to V2?	`test_registered_models_prefer_v2` — `tests/unittest/llmapi/test_llm_args.py:740`
Does the preference survive the outer VL config, which has no `hybrid_override_pattern`?	`test_nemotron_nano_v2_vl_prefers_v2_on_the_outer_config` — `tests/unittest/_torch/modeling/test_modeling_nemotron_nano_v2_vl.py:675`
Does it stay V2 on the disagg NIXL route instead of demoting?	`test_registered_models_keep_v2_on_nixl` — `tests/unittest/llmapi/test_llm_args.py:776`
Does it resolve V2 and PYTHON together, like its hybrid siblings?	`test_hybrid_models_prefer_v2_and_python_transceiver` — `tests/unittest/_torch/executor/kv_cache/test_mamba_cache_manager.py:767`
Is block reuse still opt-in now that V2 is the default?	`test_nemotron_nano_v2_vl_defaults_keep_block_reuse_opt_in` — `tests/unittest/_torch/modeling/test_modeling_nemotron_nano_v2_vl.py:651`

Existing coverage that guards the shared paths this PR touches:

TestNemotron_Nano_12B_V2_VL::test_auto_dtype[forced_chunked_prefill] (tests/integration/test_lists/test-db/l0_h100.yml:413) — does not pin the flag, so it now exercises this family on MambaHybridCacheManagerV2.
tests/unittest/_torch/executor/kv_cache/test_kv_cache_v2_vocab_size.py — the llm_config nesting fallback for resolve_vocab_size under V2.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Dev Engineer Review

NemotronH_Nano_VL_V2 now selects KV cache manager V2 and the Python transceiver by default. The change applies to the shared NemotronH Nano VL and Omni reasoning model classes. Existing V1 selection and block reuse behavior remain opt-in.

Documentation lists the affected multimodal models as V2 hybrid Mamba models. No unrelated API or configuration changes are introduced.

QA Engineer Review

Tests cover automatic V2 resolution, outer VL configuration handling, NIXL behavior, Python transceiver selection, and block reuse defaults. Registered-model coverage includes the NemotronH Nano and Omni reasoning architectures. Coverage verdict: sufficient.

Per-File QA Perspective

docs/source/features/kvcache.md: Documents the affected NemotronH multimodal models. QA should verify that the model list matches runtime support.
tensorrt_llm/_torch/models/modeling_nemotron_nano.py: Adds wrapper-level preferences for V2 and the Python transceiver. QA should verify automatic, explicit V1, and NIXL paths.
tests/unittest/_torch/executor/kv_cache/test_mamba_cache_manager.py: Covers V2 and Python-transceiver routing. This unit test has no integration test-list requirement.
tests/unittest/_torch/modeling/test_modeling_nemotron_nano_v2_vl.py: Covers outer VL configuration resolution. This unit test has no integration test-list requirement.
tests/unittest/llmapi/test_llm_args.py: Covers V2 preferences for NemotronH Nano and Omni architectures, including the NIXL/Python path. This unit test has no integration test-list requirement.

eopXD · 2026-09-14T07:54:46Z

/bot run

eopXD · 2026-09-14T08:00:26Z

/bot run

eopXD · 2026-09-14T08:00:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-09-14T08:01:36Z

PR_Github #73232 [ run ] triggered by Bot. Commit: e861485 Link to invocation

coderabbitai · 2026-09-14T08:03:03Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 406d13ca-55d1-411a-9635-46b9e0ae5fb6

📥 Commits

Reviewing files that changed from the base of the PR and between 6e6a1ac and e861485.

📒 Files selected for processing (1)

docs/source/features/kvcache.md

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

NemotronH_Nano_VL_V2 now exposes V2 KV-cache manager and Python transceiver preferences through its outer wrapper. Tests cover model routing and configuration resolution. The KV-cache documentation includes NemotronH multimodal models.

Changes

Nemotron KV-cache preferences

Layer / File(s)	Summary
Wrapper runtime preferences `tensorrt_llm/_torch/models/modeling_nemotron_nano.py`	`NemotronH_Nano_VL_V2` delegates KV-cache manager selection and selects the Python transceiver runtime.
Routing coverage and documentation `tests/unittest/_torch/executor/kv_cache/test_mamba_cache_manager.py`, `tests/unittest/_torch/modeling/test_modeling_nemotron_nano_v2_vl.py`, `tests/unittest/llmapi/test_llm_args.py`, `docs/source/features/kvcache.md`	Tests cover V2 and Python routing for the updated architectures and outer configuration. The documentation lists NemotronH multimodal models as V2 models.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Feature

Suggested reviewers: bowenfu

Merge Risk: ⚪ Minimal · up to e8614

The change consistently routes the affected NemotronH multimodal models to the intended V2 KV-cache manager and Python transceiver, with matching coverage and documentation.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 4 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes the main change: enabling KVCacheManagerV2 by default for Nemotron multimodal models. It follows the required ticket and type format.
Description check	✅ Passed	The description includes the required Description, Test Coverage, and PR Checklist sections. It explains the motivation, affected models, implementation scope, and relevant tests. The checklist is mar…

Full details: Docstring Coverage

Explanation

Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 4 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-09-14T08:06:27Z

PR_Github #73234 [ run ] triggered by Bot. Commit: e861485 Link to invocation

tensorrt-cicd · 2026-09-14T08:07:28Z

PR_Github #73235 [ run ] triggered by Bot. Commit: e861485 Link to invocation

tensorrt-cicd · 2026-09-14T08:07:30Z

PR_Github #73234 [ run ] completed with state ABORTED. Commit: e861485

Link to invocation

tensorrt-cicd · 2026-09-14T08:08:08Z

PR_Github #73232 [ run ] completed with state ABORTED. Commit: e861485

Link to invocation

tensorrt-cicd · 2026-09-14T11:14:04Z

PR_Github #73235 [ run ] completed with state SUCCESS. Commit: e861485
/LLM/main/L0_MergeRequest_PR pipeline #60169 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions · 2026-09-14T18:30:43Z

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

…al models The NemotronH_Nano_VL_V2 class serves three registered architectures -- Nemotron Nano VL v2, Nemotron Nano Omni V3, and Nemotron 3.5 Super VL -- all hybrid Mamba, and all resolved use_kv_cache_manager_v2="auto" to V1. The preference is read off the registered architecture class, and this class composes NemotronHForCausalLM rather than inheriting it, so the backbone's V2 and Python-transceiver preferences never reached the resolver. Both hooks now delegate to the backbone, alongside the get_model_defaults delegation already there for the same reason. Neither inspects pretrained_config: at resolution time that is still the outer multimodal config, whose hybrid_override_pattern lives one level down under llm_config. Signed-off-by: Yueh-Ting Chen <yueh.ting.chen@gmail.com>

eopXD · 2026-09-15T07:06:30Z

/bot skip --comment "Ran CI. The sole failure in single-GPU pipeline is not related to the change of this MR. Multi-GPU pipeline should be skippable. I have tested multi-GPU runs with the Nemotron Super 3.5 VL,with dis-agg, plus MTP, plus reuse enabled and the runtime works fine. Skipping to land the MR."

eopXD · 2026-09-15T07:48:23Z

/bot skip --comment "Re-skipping. Whole justification mentioned above."

eopXD · 2026-09-15T16:23:26Z

/bot skip --comment "Ran CI. The sole failure in single-GPU pipeline is not related to the change of this MR. Multi-GPU pipeline should be skippable. I have tested multi-GPU runs with the Nemotron Super 3.5 VL,with dis-agg, plus MTP, plus reuse enabled and the runtime works fine. Skipping to land the MR."

tensorrt-cicd · 2026-09-15T16:30:09Z

PR_Github #73627 [ skip ] triggered by Bot. Commit: 3d65425 Link to invocation

tensorrt-cicd · 2026-09-15T16:42:42Z

PR_Github #73627 [ skip ] completed with state SUCCESS. Commit: 3d65425
Skipping testing for commit 3d65425

Link to invocation

eopXD requested review from a team as code owners September 14, 2026 07:54

eopXD requested review from laikhtewari, lori-ren, moraxu, nvpohanh, pranav-nvidia, yizhang-nv, zhaoyuanh-nvidia and zhenhuaw-me September 14, 2026 07:54

github-actions Bot assigned eopXD Sep 14, 2026

eopXD changed the title ~~[None][feat] Enable KVCacheManagerV2 by default for Nemotron-H Nano VL~~ [None][feat] Enable v2_kvcm as the default on Nemotron multi-modal Sep 14, 2026

eopXD changed the title ~~[None][feat] Enable v2_kvcm as the default on Nemotron multi-modal~~ [None][feat] Enable KVCacheManagerV2 by default for Nemotron multimodal models Sep 14, 2026

eopXD changed the title ~~[None][feat] Enable KVCacheManagerV2 by default for Nemotron multimodal models~~ [TRTLLM-16220][feat] Enable KVCacheManagerV2 by default for Nemotron multimodal models Sep 14, 2026

eopXD force-pushed the user/yuehtingc/enable-kvcm-v2-nemotron-nano-vl branch from 6e6a1ac to e861485 Compare September 14, 2026 08:00

Wanli-Jiang approved these changes Sep 14, 2026

View reviewed changes

yizhang-nv approved these changes Sep 14, 2026

View reviewed changes

lori-ren reviewed Sep 14, 2026

View reviewed changes

Comment thread tests/unittest/_torch/modeling/test_modeling_nemotron_nano_v2_vl.py Outdated

lori-ren approved these changes Sep 14, 2026

View reviewed changes

2ez4bz approved these changes Sep 14, 2026

View reviewed changes

arysef approved these changes Sep 14, 2026

View reviewed changes

trtllm-agent added the ci: full pre-merge approved label Sep 14, 2026

eopXD force-pushed the user/yuehtingc/enable-kvcm-v2-nemotron-nano-vl branch from e861485 to 3d65425 Compare September 15, 2026 07:04

eopXD enabled auto-merge (squash) September 15, 2026 07:06

eopXD merged commit 000d3e9 into NVIDIA:main Sep 15, 2026
8 checks passed

eopXD deleted the user/yuehtingc/enable-kvcm-v2-nemotron-nano-vl branch October 20, 2026 03:11

Conversation

eopXD commented Sep 14, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Uh oh!

eopXD commented Sep 14, 2026

Uh oh!

eopXD commented Sep 14, 2026

Uh oh!

eopXD commented Sep 14, 2026

Uh oh!

tensorrt-cicd commented Sep 14, 2026

Uh oh!

coderabbitai Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

tensorrt-cicd commented Sep 14, 2026

Uh oh!

tensorrt-cicd commented Sep 14, 2026

Uh oh!

tensorrt-cicd commented Sep 14, 2026

Uh oh!

tensorrt-cicd commented Sep 14, 2026

Uh oh!

Uh oh!

tensorrt-cicd commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 14, 2026

Uh oh!

eopXD commented Sep 15, 2026

Uh oh!

eopXD commented Sep 15, 2026

Uh oh!

eopXD commented Sep 15, 2026

Uh oh!

tensorrt-cicd commented Sep 15, 2026

Uh oh!

tensorrt-cicd commented Sep 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

eopXD commented Sep 14, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Sep 14, 2026 •

edited

Loading