RP-OPSD: Reasoning Pivot-Guided Online Policy Self-Distillation for Multilingual Reasoning Transfer
This paper addresses the challenges of multilingual reasoning transfer in large language models by proposing RP-OPSD. While existing Online Policy Self-Distillation (OPSD) provides dense token-level supervision, it fails to explicitly prioritize reasoning signals crucial for cross-lingual transfer. The authors identify that target language reasoning consists of surface text generation and reasoning pivots (key decisions guiding the reasoning process). RP-OPSD leverages the distribution shift between teacher views with and without English reference solutions as an operational proxy to guide prioritized distillation and reference anchoring for reasoning pivots. On mathematical reasoning benchmarks covering 17 languages and multiple difficulty levels, RP-OPSD significantly outperforms strong multilingual reasoning baselines and OPSD variants. Further analysis shows that RP-OPSD concentrates prioritized distillation on reasoning control and state update tokens while reducing weights on surface implementation tokens, effectively enhancing multilingual reasoning capabilities.
Background and Context
The transfer of complex reasoning capabilities from high-resource languages, such as English, to low-resource or other high-resource languages remains a critical challenge in the development of multilingual large language models. While On-Policy Self-Distillation (OPSD) has emerged as a promising technique by providing dense token-level supervision through student-generated rollouts, it suffers from a fundamental limitation: the lack of explicit prioritization for reasoning signals that are crucial for cross-lingual transfer. Traditional distillation objectives often optimize all prediction probabilities uniformly, failing to distinguish between the structural "skeleton" of logical deduction and the superficial "flesh" of linguistic realization. This uniform approach leads models to memorize surface-level language patterns while losing the core logical framework necessary for accurate reasoning in target languages.
To address this gap, researchers have introduced the concept of Reasoning Pivots, defined as the key decision points that guide and redirect the reasoning process within a target language. The reasoning process is thus decomposed into surface text generation and these critical reasoning pivots. Existing mechanisms do not sufficiently prioritize these pivots, resulting in poor logical migration across languages. The proposed RP-OPSD method aims to rectify this by focusing distillation efforts specifically on these reasoning pivots, thereby enabling more precise and efficient transfer of multilingual reasoning capabilities. This approach fills a significant void in how reasoning signal priority is allocated during the training of large language models.
Deep Analysis
The technical innovation of RP-OPSD lies in its construction of a distribution-shift-based proxy mechanism to precisely locate and reinforce reasoning pivots. The method operates by comparing two distinct teacher views: one that includes English reference solutions and another that excludes them. The underlying hypothesis is that the introduction of English reference solutions causes a significant shift in the model's distribution when generating reasoning pivots, reflecting a dependency on correct logical paths. RP-OPSD leverages this distribution shift as an operational proxy to guide prioritized distillation and reference anchoring for these pivots. This allows the model to identify which tokens are critical for logical progression versus those that are merely linguistic fillers.
In terms of training strategy, RP-OPSD dynamically adjusts distillation weights rather than treating all tokens equally. Tokens identified as reasoning-control or problem-conditioned state-update receive higher distillation pressure, forcing the student model to mimic the teacher's decision distribution at critical logical nodes. Conversely, tokens serving surface realization, such as common conjunctions or modal particles, are assigned lower weights. This differentiated weighting mechanism ensures that optimization resources are concentrated on the "hard problems" of logical reasoning, enhancing accuracy and robustness while maintaining linguistic fluency. By using English as a high-resource reference, the method extracts universal reasoning structures that can be effectively transferred to other languages.
Industry Impact
The implications of RP-OPSD extend beyond academic metrics, offering a new, efficient technical pathway for enhancing the reasoning capabilities of multilingual large language models. By reducing reliance on surface language patterns and strengthening the capture of universal reasoning structures, this method is particularly significant for improving the intelligence level of low-resource language models. It contributes to narrowing the AI capability gap between different languages, fostering a more equitable distribution of advanced AI functionalities globally. The approach provides a novel perspective on extracting transferable reasoning signals from high-resource languages, potentially inspiring future research into methods based on logical structure rather than surface form.
For the open-source community, the publication of code at https://github.com/NJUNLP/RP-OPSD facilitates the reproduction and improvement of related technologies, accelerating progress in multilingual reasoning research. In industrial applications, models with stronger multilingual reasoning capabilities can better serve complex global tasks, such as multilingual legal analysis and cross-lingual code debugging. This enhances the practicality and reliability of AI systems in real-world scenarios. The method not only advances the theoretical understanding of multilingual reasoning transfer but also provides robust technical support for building truly global and intelligent large language models, impacting multiple sub-directions within natural language processing.
Outlook
Experimental validation of RP-OPSD was conducted on mathematical reasoning benchmarks covering 17 languages and multiple difficulty levels. The results demonstrate that RP-OPSD significantly outperforms strong multilingual reasoning baselines and standard OPSD variants. The advantage is particularly pronounced in handling high-difficulty mathematical problems, proving its superiority in complex logical reasoning transfer. Further analysis reveals that the method successfully concentrates prioritized distillation on tokens that control the reasoning flow and update problem states, while reducing weights on surface implementation tokens. Ablation studies confirm that removing the reasoning pivot guidance mechanism or altering the distribution shift calculation leads to significant performance drops, validating the necessity of the proposed approach.
Looking forward, the success of RP-OPSD suggests that future research should continue to explore mechanisms that decouple logical structure from linguistic surface. The ability to explicitly prioritize reasoning signals offers a scalable solution for improving multilingual intelligence without requiring massive amounts of labeled data in every target language. As the field moves towards more autonomous and reasoning-capable AI systems, techniques like RP-OPSD will likely become foundational for ensuring that logical consistency is preserved across linguistic boundaries. This paves the way for more reliable and universally applicable AI assistants capable of nuanced reasoning in any language.