Sequential Beats Joint: On the Interaction Mechanism of Policy Distillation and RLVR

Background and Context

In the post-training phase of large language models, enhancing complex reasoning capabilities remains a central challenge in artificial intelligence research. Two dominant technical pathways have emerged: Online Policy Distillation (OPD) and Reinforcement Learning with Verifiable Rewards (RLVR). Existing studies frequently attempt to fuse these methods into a single-step joint optimization process. This approach typically employs weighted combinations or teacher-modulated scaling to leverage OPD’s dense token-level supervision as a remedy for the sparse rewards inherent in RLVR. The prevailing assumption is that such signal fusion yields synergistic effects, effectively combining the density of distillation with the exploration of reinforcement learning.

However, this research challenges the fundamental assumption that joint optimization is superior to sequential processing. The study posits that merging these signals in a single step may introduce significant interference, thereby degrading performance rather than enhancing it. Instead, the authors propose and validate a deceptively simple yet disruptive alternative: a two-stage serial approach where OPD is executed first, followed strictly by RLVR. This sequential strategy is tested against pure OPD, pure RLVR, and all existing joint baselines across various logic and mathematics reasoning benchmarks. The core contribution lies not only in the empirical superiority of this sequential method but also in the theoretical revelation of how these two mechanisms interact, challenging the entrenched belief that signal fusion always outperforms staged processing.

Deep Analysis

The technical methodology relies on rigorous control variable experiments and an in-depth analysis of learning dynamics to deconstruct the internal mechanisms of OPD and RLVR. The study identifies that OPD’s primary function is to rapidly expand the student model’s coverage of the correct solution space through the dense supervision of a teacher model. This process acts as a "broad exploration," allowing the model to acquire a wide range of teacher-supported solution paths. In contrast, RLVR operates as a "deep optimization" mechanism. Once the solution space is sufficiently covered, RLVR uses verifiable reward signals to refine and sharpen the policy within that established scope, improving the confidence and consistency of the generated solutions.

When joint optimization is employed, the dense gradient updates from OPD and the sparse reward signals from RLVR interfere with each other. This conflict prevents the model from converging to an optimal policy, as the signals compete rather than complement. The two-stage approach resolves this by decoupling the signals temporally. First, OPD establishes a high-quality policy foundation. Subsequently, RLVR is introduced to filter and reinforce these paths without the noise of concurrent dense supervision. Furthermore, the research demonstrates that OPD serves as a significantly better cold start for RL than traditional Supervised Fine-Tuning (SFT). This is because the policy distribution generated by OPD is more closely aligned with the exploratory distribution required for reasoning tasks, providing a more optimal initial state for the reinforcement learning phase.

Industry Impact

The experimental results, derived from multiple mainstream mathematics and logic reasoning benchmark datasets, confirm the robustness of the OPD-then-RL scheme. The model not only achieves higher final accuracy but also exhibits greater stability in its learning curves compared to baselines. Ablation studies highlight the critical importance of the timing for switching stages. By monitoring the OPD verification score, the researchers identified a specific threshold that indicates sufficient coverage of the solution space. Switching to RLVR at this precise moment yields maximum performance gains, whereas switching too early or too late results in diminished returns. Analysis of pass@k behavior further confirms that OPD significantly increases the probability of generating valid solutions, while RLVR enhances the quality and reliability of those solutions.

For the industrial sector, the OPD-then-RL framework offers a simple, efficient, and easily implementable recipe for training reasoning models. It eliminates the need for complex joint optimizer designs, thereby reducing engineering complexity while significantly boosting model performance. For the open-source community, these findings prompt a re-evaluation of existing post-training methods. The evidence of signal interference in joint optimization may trigger a broader reflection on current practices, leading to improved methodologies. The study provides a clear, practical guide that prioritizes simplicity and efficacy over complex architectural modifications, making it highly accessible for immediate adoption.

Outlook

This research provides a self-consistent explanatory framework that underscores the advantages of staged processing in signal decoupling. The identified mechanism of "coverage followed by sharpening" offers theoretical guidance for exploring other multi-signal fusion strategies. Future research can build upon this foundation to investigate finer-grained stage divisions or dynamic switching strategies, potentially further unlocking the potential of reasoning large language models. The mechanism of using verification scores to determine stage transition timing also opens new avenues for designing automated training pipelines.

Ultimately, this work does more than provide a powerful training scheme; it deepens the understanding of interaction mechanisms in multi-signal learning. By proving that sequential processing can outperform joint optimization, it shifts the paradigm for how researchers approach post-training in complex reasoning tasks. The findings suggest that simplicity in architecture and sequence may often yield superior results compared to complex, simultaneous signal integration. This insight will likely influence the design of future models, encouraging a focus on clear, decoupled stages of learning rather than monolithic optimization steps.