MoTE: A Mixture-of-Task-Experts Decoder for Multi-Task Video Understanding

Background and Context

Procedural video-language models face a demanding requirement: they must perform heterogeneous downstream tasks—action recognition, action prediction, and process prediction—simultaneously from the same set of visual evidence. This forces a tension the field has struggled to resolve. A model needs to share a single underlying visual representation across tasks, yet still reason differently for each one. The authors frame this as a decoding problem, arguing that how the decoder is structured determines whether tasks stay cleanly separated or bleed into one another.

The paper identifies a concrete weakness in the dominant approach. Dense transformer decoders share one feedforward network across all tasks. Because every task draws on the same weights, the behavior of different tasks becomes entangled, making it hard to isolate individual capabilities. This entanglement also makes controlled extension of specific abilities difficult, since expanding one task inevitably disturbs the shared parameters. The authors argue this is a structural limitation, not merely a scaling issue.

Sparse Mixture-of-Experts decoders offer conditional computation to save compute, but their standard token-level learned routing does not naturally align with task-level objectives. The routing decisions emerge from learned weights that are difficult to map onto clear task semantics. The result is a decoder that is efficient yet opaque, unable to guarantee that a given computation actually serves the intended task. This mismatch motivates the central proposal of the work.

Deep Analysis

The authors propose MoTE, the Mixture of Task Experts decoder, which restructures the feedforward network of a large language model into a set of task-specific experts while keeping the multimodal backbone shared across all tasks. Each sample routes along a single task to activate its corresponding experts. This design preserves the efficiency of conditional computation while making the routing outcome map directly onto a concrete task, striking a balance between interpretability and cost.

The key technical move is sample-level task routing rather than token-level learned routing. Because each input follows one explicit task path, the amount of activated computation is decoupled from the total number of stored task experts. This property matters for controlling inference cost: the expense does not grow linearly with the number of tasks, which the authors identify as essential for systems that must continually add new task capabilities. The routing behavior is made explicit and observable, rather than buried inside learnable sparse weights.

Training leverages the combination of a shared backbone with task-specific decoding experts, allowing visual representations and task-reasoning capabilities to be optimized separately within a unified framework. This preserves cross-task sharing of visual understanding while enabling task-specific expression. The structure extends the mature large language model decoder paradigm while reworking it at the semantic level through task-level routing, so that conditional computation genuinely serves the objectives of procedural video understanding.

Industry Impact

The authors evaluate VideoLLM-MoTE across five COIN benchmarks, driving the decoder with explicit task routing. The five-expert model activates approximately two billion large language model parameters per sample, yet achieves average top-1 accuracy that surpasses recent VideoLLM baseline methods. These results position the approach competitively against contemporary systems despite activating a controlled fraction of parameters.

Ablation and controlled comparisons prove decisive. Under the same expert topology, MoTE outperforms both the dense full-expert activation scheme and a learned sparse-routing control group. This comparison demonstrates that task-structured routing holds an advantage over purely learned routing in both efficiency and performance, rather than merely matching it. The evidence suggests the interpretability gains are not purchased at the cost of accuracy.

For industrial deployment, the decoupling of activated compute from stored experts means inference cost stays controllable even as the task portfolio expands. This matters for scenarios requiring continuously added task capabilities. Explicit routing also makes model behavior more traceable, supporting trust in high-reliability domains such as video understanding and process analysis. For the open-source community, combining the large language model decoder paradigm with task-level expert mechanisms provides a reusable architectural reference for future work.

Outlook

The work establishes task-structured routing as an interpretable and computationally efficient decoding scheme for multi-task video-language learning. By showing that task-specific experts with sample-level routing can beat both dense and learned-sparse alternatives under identical topology, it offers a concrete alternative to the opaque routing that currently dominates sparse expert systems.

The most promising direction is scaling to systems that must accumulate many task capabilities over time. Because activated compute does not grow with the number of stored experts, the architecture is well suited to products that repeatedly add new functions without proportional cost increases. The explicit routing further supports deployment in regulated or safety-sensitive settings where traceability is required.

Broader, the approach may extend beyond video to any cross-task conditional computation where shared representations must coexist with distinct reasoning paths. Whether the gains hold under larger backbones and more diverse task mixes remains to be tested, but the current results provide a credible foundation for treating task structure as a first-class design principle in decoder architecture.