NVIDIA Measures 42.5% More GSM8K Decode Throughput on DGX Spark

NVIDIA's October 27 technical post tests three ways to speed Ling-3.0-flash inference on one DGX Spark at batch size one. The example model is a mixture of experts with 124 billion total parameters and 5.1 billion active parameters per token. Profiling found decode limited mainly by reading weights, especially the routed expert work and the output projection. The article includes actual measured comparisons, so it would be wrong to say there is no benchmark. It would also be wrong to turn those vendor measurements into an independent result or a guarantee for unrelated models and workloads.

The first change swaps the baseline FlashInfer MXFP4 mixture-of-experts backend for Humming kernels specialized to the model's shapes. On NVIDIA's synthetic workload of 32,000 input tokens and 2,000 output tokens, time per output token moved from 27.71 milliseconds to 25.35 milliseconds, and decode throughput rose from 36.09 to 39.45 tokens per second, a reported 9.3% gain. The second change applies online FP8 quantization only to the language-model head. That layer's weights are otherwise read in BF16 for every output token, so the FP8 form halves its weight-read volume. Combined with Humming, the reported synthetic throughput becomes 43.94 tokens per second, or 21.7% over the baseline. These figures use the stated baseline and test configuration; they are not a before-and-after result for all serving systems.

The third change uses the model's NEXTN layer for multi-token prediction speculative decoding. Draft tokens are checked together, reducing the number of serial decode steps when a prefix is accepted. In the synthetic test, the average accepted length was 3.90 tokens per round and the three changes together reached 102.88 tokens per second, 185.1% above the baseline. NVIDIA explains an integration trap: the draft must share the whole FP8 language-model-head module with the target. Sharing only the weight tensor can silently produce an interpretation mismatch, collapse accepted length to one and make the combined configuration slower than no optimization. Operators therefore need to verify that the intended backends and sharing path are active, not merely set flags.

The post then uses all 1,319 GSM8K test questions for a less repetitive end-to-end comparison. With five-shot prompting and the same sampling controls on both sides, decode throughput changes from 37.81 to 53.89 tokens per second, a 42.5% gain. Mean latency moves from 15.21 to 11.49 seconds, a 24.5% reduction. The measured accuracy moves from 96.89% to 96.82%, a 0.08 percentage-point difference that NVIDIA describes as not statistically detectable. Time to first token does not improve in this table: 0.207 versus 0.214 seconds. The synthetic 185.1% number cannot replace the GSM8K 42.5% number, because the prompts, output lengths and draft acceptance behavior differ.

The engineering conclusion is narrower than “MoE inference is three times faster.” Humming and FP8 reduce work per step, while speculative decoding reduces the number of steps, and their combination needs correct integration. This particular model on this particular device shows a useful gain under the reported tests. A deployment should compare its own prompt distribution, sampling, acceptance length, correctness and end-to-end latency against a frozen baseline. NVIDIA's article supplies methods and a cookbook for that exercise, but it does not establish a universal speedup.