Beyond Teacher Likelihood: Group-Calibrated Online Strategy Distillation for Long-Context Reasoning
This paper addresses the failure of Online Policy Distillation (OPD) in long-context reasoning tasks. The authors argue that traditional OPD relies on stronger teacher models to provide dense guidance at every token, but in long-context tasks evidence is often scattered across the entire input, so token-level guidance tends to favor locally plausible responses that miss global evidence or violate task constraints. Meanwhile, task-specific validators give graded rewards at the response level, reflecting partial success, creating a systematic mismatch. The authors diagnosed this mismatch on two representative long-context evidence-aggregation tasks, finding that as inputs lengthen, trajectory-level OPD scores increasingly diverge from validator rewards. They propose Group-Calibrated Online Policy Distillation (GC-OPD), which normalizes validator rewards and OPD scores separately within each rollout group, takes their difference as a signed teacher-validator inconsistency residual, and distributes the residual across tokens by relative OPD advantage via Relative Advantage Credit Allocation (RACA) while preserving the original OPD signal. On five long-context benchmarks, GC-OPD raises the average scores of Qwen3-4B from 29.08 to 40.47 and Qwen3-8B from 35.12 to 44.65, significantly outperforming naive OPD.
Background and Context
Online policy distillation (OPD) is a leading approach for model compression, letting a lightweight student learn a stronger teacher by absorbing dense, token-level guidance while generating its own responses on its own distribution. The new paper argues that this mechanism fails systematically in long-context reasoning tasks, where the goal is not local plausibility but global evidence aggregation. The work does not stop at diagnosing the problem; it proposes Group-Calibrated On-Policy Distillation (GC-OPD) and validates it across five long-context benchmarks, reporting concrete gains for Qwen3-4B and Qwen3-8B.
The root of the problem lies in a mismatch between two kinds of signal. OPD's teacher guidance is token-level, judging whether each token looks reasonable within its immediate local context. But long-context tasks scatter the evidence needed for a correct answer across many positions, requiring the model to integrate information over the whole input and satisfy global task constraints. A response that reads coherently token by token can still omit key evidence or violate constraints. Task-specific validators, by contrast, grade at the response level and return graded rewards reflecting partial success. This gap in scale and granularity is what undermines OPD in long contexts.
To convert this intuition into measurable evidence, the authors ran diagnostic experiments on two representative long-context evidence-aggregation tasks, comparing teacher guidance against validator rewards on fixed responses. Their key finding: as input length increases, trajectory-level OPD scores increasingly diverge from validator rewards. In other words, the teacher's dense per-token judgments stop aligning with what the task actually rewards, implying that relying on them alone trains students toward capabilities inconsistent with real task completion.
Deep Analysis
GC-OPD addresses the mismatch by explicitly modeling and correcting the disagreement between teacher and validator signals. Within each rollout group, it normalizes validator rewards and trajectory-level OPD scores separately, then subtracts them to produce a signed teacher-validator inconsistency residual. A positive residual means the validator favors the trajectory more than the teacher; a negative one means the teacher overestimated it. Crucially, GC-OPD does not discard the original dense token-level OPD signal; it adds this residual as an extra calibration term on top of it.
To distribute the residual across tokens, the method introduces Relative-Advantage-based Credit Allocation (RACA). RACA assigns the trajectory-level residual to individual tokens according to each token's relative OPD advantage, so the validator signal enters the training process without destroying OPD's existing dense guidance structure. This design balances global task-level signals with local token-level signals, avoiding the information loss that comes from naively replacing teacher guidance with validator rewards.
Ablation experiments isolate the contribution of each component. The signed residual term outperforms simply stacking an OPD-derived term, and it also beats directly adding group-normalized validator rewards, showing that explicitly modeling the teacher-validator disagreement while preserving its sign matters. RACA further improves over uniform token allocation, confirming that differentiating residual distribution by relative advantage is both reasonable and necessary. Together these results support the core conclusion: group-relative residual calibration folds validator outcomes into distillation without abandoning dense token-level guidance.
Industry Impact
The results are substantial. Under identical post-training settings, GC-OPD raised the average score of the official Qwen3-4B checkpoint across the five benchmarks from 29.08 to 40.47, and lifted Qwen3-8B from 35.12 to 44.65. The naive OPD baseline reached only 39.31 and 43.56 respectively in the same setup, a clear gap that attributes a real contribution to the group-calibration residual. These numbers demonstrate that correcting teacher-validator misalignment yields meaningful long-context reasoning gains for compact models.
Beyond the method itself, the paper offers a broader lens: when teacher signals and task objectives diverge in granularity and scale, blindly trusting dense guidance can harm final performance. This insight extends to long-context modeling, RL-style training, and any learning system that depends on external validation signals. For industry deployment, GC-OPD provides a task-aligned path for transferring large-model capability to smaller models under limited compute, especially in evidence-aggregation and long-document question-answering scenarios that require cross-passage integration.
The authors have released their code, enabling the open-source community to reuse and extend the framework directly. This lowers the barrier for follow-up work and encourages replication across additional long-context settings where dense teacher signals may misalign with graded validation outcomes.
Outlook
The GC-OPD framework opens several directions for future research. One is modeling the teacher-validator disagreement more finely, moving beyond a single signed residual toward structured representations of where and why the two signals diverge. Another is generalizing the calibration to more varied long-context task types, testing whether the group-relative residual approach transfers across domains beyond evidence aggregation.
A natural next step is scaling the method to larger students and longer inputs, examining whether the alignment gap widens further and whether RACA continues to allocate credit effectively. Integrating multiple validators or hierarchical reward signals could also refine how partial success is credited across tokens.
Ultimately, the work positions online policy distillation as a practical tool for complex reasoning under compute constraints. By reconciling dense local guidance with graded global objectives, GC-OPD points toward student models that learn not just to sound coherent, but to genuinely complete long-context tasks.