Omnilingual-MsASR: Long Multi-Speaker ASR Benchmark

Background and Context

Why do long, multi-speaker videos still break omni-modal large language models? Omni-modal models have improved fast at audio and video understanding. But a clip that runs for tens of minutes and holds many voices is a different test. The model must follow the whole recording. It must also keep track of who said what, and when. Most existing evaluations stop at short audio or a single speaker, so they say little about the true upper limit of this skill.

Omnilingual-MsASR was built to fill that gap. It is a multilingual, multimodal benchmark for long-form multi-speaker automatic speech recognition, shortened to MsASR. It covers six languages. A single recording can last up to 30 minutes, and a scene can contain up to 20 speakers. The benchmark joins three hard conditions in one test bed: long duration, many speakers and several languages.

The paper makes three main contributions. First, it gives researchers one unified platform that combines the long, multi-person and multilingual difficulties. Second, it supplies parallel audio and video inputs for the same content, so a team can compare how a model behaves under each modality. Third, it runs a broad evaluation of leading omni-modal models and shows where their limits lie. The code and data are open-sourced under the QwenLM organization on GitHub, which lets the community reproduce the results and build on them.

Deep Analysis

The paper's focus is not a new model. Its focus is a clear definition of the task. Classic speech recognition asks only whether the transcribed words are correct. A multi-speaker setting asks for three outputs at once: the content, the speaker identity and the time position. If one of the three is missing, the transcript is not usable for minutes, subtitles or interviews. For this reason the benchmark uses tcpWER as its core metric. This metric starts from word error rate and adds speaker attribution and time constraints to the calculation. A word counts as correct only when it is right, assigned to the right person and placed in a reasonable time window. This is much stricter than a plain word error rate. It is also closer to real needs such as meeting records, interview captions and multi-person video.

The second key design choice is the parallel audio and video input. Audio carries voice features and tone. Video carries lip movement, the people on screen and the scene. By comparing results under the two inputs, researchers can test cross-modal consistency, which means whether the output stays stable when the same content arrives in a different form. They can also test whether a model uses visual cues to resolve audio ambiguity, or uses sound to fill a gap in the picture. Six languages, 30-minute recordings and up to 20 speakers push context length, speaker count and language variety to a stress-test level together. What does a tcpWER above 100% mean in practice? The evaluation of leading omni-modal models gave a direct answer: clear bottlenecks, with tcpWER often above 100%. A value over 100% means the error count is larger than the number of words in the reference text. This can happen when a model drops long stretches of speech and also inserts wrong or repeated content. It can also happen when speaker labels or time positions are wrong, so many correct words are still scored as errors. These models may look strong on short clips. At half an hour and a dozen voices, the output is hard to use as it stands. From this result the authors name three key challenges. The first is fine-grained temporal grounding: models struggle to say exactly when a sentence was spoken. The second is speaker discrimination: with many people and overlapping turns, models confuse who is talking. The third is cross-modal synergy: models do not truly merge the complementary information in audio and video, and results often differ between the two inputs. One limit applies to this analysis. The source summary gives no per-model scores and no ablation details, so the comparison of model gaps rests only on the paper's overall conclusions.

Industry Impact

For industry and the open-source community, the first value of this benchmark is to calibrate expectations. Meeting transcription, long-video captioning, call-center quality checks, and classroom or interview records all depend on long multi-speaker recognition. A team that looks only at short-audio leaderboards may overrate how ready omni-modal models are. Omnilingual-MsASR offers a ruler that sits closer to deployment.

The second value is direction. The benchmark points to specific work: finer timestamp modeling, long-range memory of speaker identity, and real use of lip movement and on-screen people to support what the model hears. These parts are easy to overlook in current omni-modal foundation model training, and the three named challenges give teams a concrete checklist.

The third value is fairness across languages. The benchmark is open and covers six languages. This helps compare models in a multilingual setting and reduces the risk that a skill is checked only in English or other high-resource languages. A shared public test also lowers the cost for smaller labs, because they can measure against the same data without building their own long-form set.

Outlook

A benchmark can expose a problem. It cannot solve it. Closing the gap will need matching training data, long-context modeling and training methods that are aware of speakers. The paper itself frames fine-grained temporal grounding, speaker discrimination and cross-modal synergy as the three problems that future omni-modal foundation models must overcome.

What should readers watch next? First, whether new models report tcpWER on Omnilingual-MsASR and bring the number well under 100%. Second, whether the audio-versus-video comparison shows that video cues give a real gain, or only add noise. Third, whether the six-language coverage reveals large gaps between languages once more results are published. Because the code and data sit under QwenLM on GitHub, these follow-up results can be checked by anyone.

In a field that lacked a shared long-form test, this work gives the community a common reference point. Its tcpWER figures, often above 100%, are a plain reminder that hearing a whole 30-minute meeting and tagging every speaker correctly remains an open problem for omni-modal models.