TensorRT-LLM 1.4 Release Candidate Changes KV Defaults

TensorRT-LLM 1.4.0 release candidate needs a staged upgrade

NVIDIA published the open-source TensorRT-LLM `v1.4.0rc1` release candidate on October 22 UTC. The release notes cover model support, serving APIs, KV-cache management, decoding paths and hardware-specific optimizations. A release candidate is a useful integration target for teams that need one of those changes, but it is not the same as a broadly proven production build. The notes themselves flag known issues. One concerns cancelled streaming requests with disaggregated serving and the default Python KV-cache transceiver on A100; another says a JSON-schema serving example can return unbounded output with Qwen3 models. Those caveats should be part of any evaluation plan, not buried behind the feature list.

Cache defaults are a compatibility change

The notes mark two API changes as breaking: KV cache manager V2 and the Python KV-cache transceiver become defaults, and unused `LlmRequest` fields are removed. A service upgraded from an earlier version may therefore change behavior even if its own configuration file does not. Teams using prefill and decode separation, cancellation or an external cache connector should test those paths explicitly. NVIDIA adds a Mooncake connector for KV-cache manager V2, a lending API for transfer backends, native KV event streaming and routing-load exposure. These are useful primitives for operators who need to place work and move state across workers, but the release page does not prove a throughput gain for every topology.

The specific cancellation issue illustrates the risk of a hidden state transition. A client can close a stream after receiving enough output; the server still has cache state and transfer work to reconcile. If a transceiver fails after that cancellation, later generation requests may be affected. A controlled load test should mix normal completions with deliberate client disconnects, inspect error rates and cache cleanup, and repeat on the same hardware and serving mode planned for deployment. A simple one-request latency benchmark would miss this failure mode. The issue is reported as observed on A100, so it should not be stated as universal across all GPUs.

Model and precision support expands

The release adds support for DeepSeek-V4.1-Flash and NVFP4 GLM-5.3-Flash checkpoints, plus an experimental Modeling V2 target for Qwen3.8-Flash-Next NVFP4. It also mentions DFlash draft checkpoints for LiLiCorr and changes the fallback behavior for Nemotron-H NVFP4 on Hopper. Each entry is a compatibility claim for a particular model format and execution path. It is not a statement that model outputs or performance are identical across precisions. Teams should verify a checkpoint's license, tokenizer, chat template, numerical tolerance and representative output quality before replacing a known-good deployment. Quantization can reduce memory or improve throughput while changing edge-case behavior, especially in long or structured outputs.

NVIDIA also lists options for RoPE precision in NVFP4 cold-page compression, a progressively filled host cache tier, and rebalancing parameters for cache V2 pools. These knobs expose resource tradeoffs that a busy serving cluster faces: GPU memory, host memory, transfer time and response latency. A useful measurement reports all four, with p50 and tail latency under the same prompt-length distribution. The release notes mention many hardware paths, including Rubin support, Blackwell CuTe DSL work and attention variants. They should be treated as architecture-specific developments until tested on the intended machines. A headline about one GPU family should not be copied into a budget for another.

Structured output and determinism need their own tests

The JSON-schema example caveat for Qwen3 is important for agent systems that expect bounded machine-readable responses. A schema feature is not operationally useful if a client can wait indefinitely or accept a truncated object without noticing. The release also adds deterministic FP32 GVR V2 top-K decoding and honors per-request seeds in certain batched sampling paths. Deterministic behavior is scoped to those stated paths, not a blanket promise that every distributed run will reproduce byte for byte. Teams should set response caps, validate parsed output, log the model and decoding settings, and compare repeated runs under the exact batching plan used in production.

A decision for October 25 readers

If the current stack is stable and none of the new model or cache features is needed, the release candidate can remain an evaluation target. If a needed checkpoint or connector requires it, stage the upgrade on an isolated service and collect a before-and-after profile covering cancellation, prefill/decode transfer, structured output, quality and resource use. Keep a tested downgrade path for config and cache behavior. That is an editorial recommendation from the published change and issue list, not a claim that the candidate is unsafe everywhere. The October 25 digest presents the October 22 original release with its limitations intact; no final 1.4.0 availability is implied.