Arm Engineers Profile Nemotron 3.5 Streaming Speech Recognition on a Pixel 11 with ExecuTorch

Arm engineers profile streaming speech recognition on a phone with ExecuTorch

On October 26, the PyTorch blog published a case study by two Arm engineers. They took the multilingual streaming speech model Nemotron 3.5, exported it from PyTorch to ExecuTorch, and profiled it on a Google Pixel 11 phone. The authors work for Arm, which sells the instruction-set extension under test, so every number below is a vendor-side measurement. The study asks how numeric precision and audio chunk size change accuracy, how much processing time the audio needs on one thread, and which kernels actually run. It starts from a warning: a model with the best accuracy on whole recordings can do much worse when it must emit text as people speak.

Accuracy depends on chunk size

Streaming recognition processes audio in chunks. The model, as the post describes it, turns audio into one feature vector per 10 ms, and its encoder reduces that to one position per 80 ms. Chunk sizes of 80, 320, 560 and 1,120 ms were tested. The authors measured word error rate (WER), the sum of substitutions, deletions and insertions divided by the total number of reference words, on a host computer with ExecuTorch's XNNPACK backend. They used 14,019 utterances from four public corpora. From 80 ms to 1,120 ms, WER with 16-bit floating point (FP16) falls from 15.2% to 11.5%, and with 8-bit integer weights (called QC8W) from 15.5% to 11.7%. FP16 differs from 32-bit floats by less than 0.03 percentage points at every size, while QC8W adds 0.21 to 0.31 points. Longer chunks help accuracy but make the first text appear later.

What the phone measurements show

For speed, the team processed one 24.9-second recording on a Pixel 11 with 1,120 ms chunks and a single inference thread pinned to a mid-power CPU core. They built ExecuTorch 1.3.1 twice, with and without Arm's SME2 matrix instructions, and each setting had two warm-up runs and one measured trace. With SME2, FP16 processing fell from 9.79 to 4.61 seconds, a 2.12 times speedup, and QC8W fell from 5.20 to 3.24 seconds, 1.61 times. The authors express this as a real-time factor, processing time divided by audio length: 0.19 for FP16 and 0.13 for QC8W. They stress that this headroom does not remove the wait for audio to arrive. Matrix multiplication accounted for most of the gain, dropping from 8.2 to 3.1 seconds for FP16, while data movement took about the same time in both builds and became a larger share of the total.

Which kernels ran

Operator timings do not say which low-level routines executed, so the authors also sampled native call paths with an Arm profiler. They report that matrix hardware was allocated 38.4% of the capture for FP16 and 53.7% for QC8W. They state that this is not a share of peak arithmetic throughput. Samples confirmed that kernels written for SME2 ran in both cases, next to other kernels: some FP16 operations still used the older NEON vector path, and some QC8W operations still ran in 32-bit floats. The post says the two tools used here, ETDump and Streamline, do not directly connect model layers to the kernels that ran them; the authors list layer-level mapping as future work.

How to use the results

The study shows a method more than a verdict: choose chunk size and precision together, and measure both accuracy and latency on the target device. Its limits are plain. The accuracy tests ran on a host, not the phone. The phone timing used one recording, one thread and one measured run per setting. Results were obtained on a single device with Arm's own optimizations. A team planning a product should repeat the tests with its own audio, languages and thermal limits.