NVIDIA Vera Rubin NVL72 Excels in MLPerf Inference v6.1

Background and Context

In the latest MLPerf Inference v6.1 benchmark release, NVIDIA’s next-generation Vera Rubin architecture made its debut through the NVL72 system, immediately capturing leading performance across multiple mainstream models. The results, disclosed on NVIDIA’s official blog, highlight a decisive throughput advantage in large language model (LLM) inference scenarios. MLPerf, widely regarded as the industry’s most authoritative AI performance yardstick, spans tasks from image classification and object detection to medical imaging, speech recognition, and generative AI. Vera Rubin NVL72’s first submission not only topped several categories but also signaled a broader industry pivot from training-centric metrics to inference economics.

The standout feature of the NVL72 system is not merely its single-node performance but its efficient multi-node scaling—throughput grows nearly linearly as GPUs are added. This near-linear scalability directly translates into lower cost per token generated, a critical metric for cloud inference services that must handle massive concurrent requests. By demonstrating that raw system throughput can be proportionally amplified with hardware, NVIDIA underscores that infrastructure scaling efficiency is now as vital as peak chip performance in determining the total cost of ownership for AI deployments.

Deep Analysis

Vera Rubin NVL72’s performance stems from a three-layer co-evolution. At the architectural level, the Vera Rubin GPU is expected to leverage a more advanced manufacturing process and incorporate larger-capacity HBM high-bandwidth memory. It continues and upgrades the Transformer Engine, supporting dynamic mixed-precision computation down to FP8 and even lower precisions. This directly accelerates matrix operations in large models while improving energy efficiency, reducing latency per token without sacrificing accuracy.

The system interconnect represents a qualitative leap. NVL72 uses NVLink to fuse 72 GPUs into a single giant shared-memory node, with cross-GPU bandwidth reaching multiple terabytes per second. This virtually eliminates communication bottlenecks for distributed inference strategies like tensor parallelism and pipeline parallelism, which is the hardware foundation for near-linear scaling. Such a tightly coupled design ensures that adding GPUs yields proportional throughput gains, a feat that eludes loosely connected clusters.

On the software side, NVIDIA’s TensorRT-LLM inference framework has been deeply optimized for Vera Rubin. Techniques such as operator fusion, automatic kernel tuning, and advanced memory management compile model graphs into highly parallel instruction sequences, squeezing out additional hardware potential. Together, these three layers allow the NVL72 to process hundred-billion-parameter models with extremely low single-token latency while scaling system throughput seamlessly with GPU count—a combination essential for production-grade cloud inference.

Industry Impact

For cloud service providers, inference cost is the linchpin of AI application profitability. Vera Rubin’s high throughput and linear scaling mean that the same capital expenditure can serve more users and generate more tokens, directly improving unit economics. This will likely accelerate the migration from Hopper or Blackwell architectures to Vera Rubin and could trigger a new wave of infrastructure investment as providers seek to maximize revenue per rack.

The competitive pressure on AMD and Intel intensifies. AMD’s MI300 series has narrowed the gap in training but still lags in inference ecosystem maturity and system-level scaling efficiency. Intel’s Gaudi line remains constrained by a less mature software stack and slower customer adoption. Vera Rubin’s MLPerf debut effectively raises the bar: contenders must now deliver not only strong single-chip performance but also efficient large-cluster scaling with a production-ready software suite. Custom chip efforts like Google’s TPU and AWS Trainium excel in specific internal workloads but lack the generality and ecosystem breadth of NVIDIA’s CUDA, further cementing NVIDIA’s de facto standard position in generative AI inference and squeezing the addressable market for second-tier accelerators.

Outlook

Several signals merit close attention. First, Vera Rubin’s official production timeline and initial customer deliveries will shape the AI compute supply landscape in the second half of 2026; a smooth capacity ramp could prompt cloud providers to revise capital expenditure upward. Second, MLPerf inference benchmarks are progressively adding more generative AI workloads, including multimodal models and retrieval-augmented generation scenarios, and Vera Rubin’s performance on these emerging tasks will influence its long-term competitiveness.

NVIDIA’s software strategy may tilt further toward inference, potentially offering vertical-specific inference containers or microservices that convert hardware advantages into platform lock-in. Meanwhile, competitor responses—such as AMD’s next-generation CDNA 4 architecture achieving breakthroughs in system scalability, or major customers like OpenAI and Microsoft increasing investment in custom inference chips—could disrupt the market. Finally, the focus on inference economics may spawn new business models, such as token-throughput-based compute leasing or differentiated pricing tied to precision and latency tiers, reshaping value distribution across the AI supply chain. Vera Rubin NVL72’s MLPerf debut is thus not just a performance demonstration but a clear statement of NVIDIA’s strategic direction for the inference era.