BitDecoding-FP4: Native FP4 KV Cache Decoding

Background and Context

In long-context LLM serving, the KV cache is one of the largest consumers of GPU resources. It fills memory capacity, and every decoding step reads the whole cache again, so it also eats memory bandwidth. A longer context or a higher number of concurrent requests makes both costs worse.

Compressing the cache to 4 bits is the obvious answer. Microscaling formats such as NVFP4 and MXFP4 shrink the cache. They also let attention run directly on FP4 Tensor Cores, which removes a dequantization detour. The BitDecoding-FP4 paper argues that research on algorithms and work on systems have not met in this area. Current FP4 KV systems use naive, vanilla quantization. They gain speed but lose a lot of accuracy. Accuracy-focused 4-bit algorithms rely on vector codebooks, and the table lookups they need slow down decoding throughput. Neither side delivers accuracy and efficiency together.

BitDecoding-FP4 is proposed as an algorithm-system co-design to close that gap. Its goal is accurate and efficient native FP4 KV decoding. The authors first separate the sources of error. They then treat the query, the attention probabilities and the keys one by one, and finally remove the softmax bottleneck at the system level. The method was validated on dense and MoE models, with both GQA and MLA attention, and the implementation is open source on GitHub.

Deep Analysis

The first algorithm piece is Dual-Residual Query Quantization. Native FP4 computation requires the query to be quantized as well, and a single FP4 pass leaves a large error in the query logits. The authors split the query into two FP4 tensors. The second tensor compensates for the residual that the first one leaves behind, so the two low-precision parts together come close to the original value. According to the paper's abstract, this cuts query logit error from 60% to 1%. The second piece is Underflow-Aware Probability Scaling. In long contexts, each softmax probability is very small. The E4M3 scale used when quantizing probabilities can then round to zero, and a whole block of probabilities collapses to zero. The later multiply-accumulate with the values loses that information. The technique adjusts the probability scaling to prevent this underflow. The third piece is Outlier-Aware Key Quantization. Key tensors contain outliers. Under token-wise scaling, one outlier stretches the quantization range of its whole token and damages the precision of the other channels. The authors combine static smoothing, Hadamard rotation and scale search. Smoothing and rotation spread the outlier energy across channels. Scale search then picks a better scale for each group of values.

How does BitDecoding-FP4 remove the softmax bottleneck? It does so with three system techniques. A transposed softmax is paired with fused probability quantization, so quantizing the probabilities no longer needs its own round of memory reads and writes. For paged KV caches, a stacked-query MMA keeps the matrix-multiply units better occupied. The reported accuracy results are measured against BF16 KV. With NVFP4, accuracy stays within 0.5 points of BF16 KV. With MXFP4, it stays within 1 point. The slightly larger MXFP4 gap suggests that the scale representation of the format affects error. The abstract does not name the benchmarks, so this article repeats only the gaps. The method also keeps pace with BF16 KV in RL training rollouts. That is a sensitive test, because a large distribution shift from KV quantization could disturb the training signal.

Industry Impact

On performance, the authors ran end-to-end serving tests on NVIDIA B300. The models were Qwen3-4B, a small dense model, and Qwen3-30B-A3B, an MoE model. Peak serving throughput is 2.4x to 3.0x higher than with BF16 KV. For operators, a smaller cache means one card can hold a longer context or more concurrent requests. A higher peak throughput should lower the cost per served token on the same hardware.

The result also changes the framing of the problem. Earlier routes either accepted lower accuracy for simple vanilla quantization, or accepted lower throughput for codebook accuracy. This work shows that once error sources are separated and the algorithm is fitted to the hardware data path, both goals can be met. The paper targets the FP4 Tensor Cores of the Blackwell-generation B300. It can therefore serve as a reference for low-precision attention work on later hardware.

The open-source release matters for engine developers. They can study the transposed softmax, the fused probability quantization and the stacked-query MMA directly. Researchers can build more aggressive quantization schemes on top of the same code. The match with BF16 KV in RL rollouts also removes one worry for teams that want to share a low-precision cache between training and inference.

Outlook

The paper's own evidence has limits. The abstract reports serving results for only two Qwen3 models. It gives no per-technique ablation numbers, so the separate contribution of each algorithm piece cannot be quantified here. Behavior on more models and on longer contexts still needs follow-up work.

Three points deserve attention next. First, whether the 0.5-point NVFP4 gap and the 1-point MXFP4 gap hold on other model families and on the full range of context lengths. Second, how much of the 2.4x to 3.0x gain comes from the smaller cache and how much from the system techniques. Third, whether the stacked-query MMA and fused probability quantization are adopted by mainstream inference engines.

For now, BitDecoding-FP4 is a concrete data point: native FP4 KV decoding can stay close to BF16 accuracy while raising peak throughput on B300.