TASPO: Bridging the Gap Between Process Supervision and Outcome Credit in Agent Policy Optimization

Background and Context

In the domain of large language model-based agents, enhancing performance in complex tasks through reinforcement learning remains a central challenge. While outcome-based reinforcement learning provides validated feedback signals, it suffers from a fundamental limitation: it uniformly distributes trajectory-level advantages to every decision step within a sequence. This coarse-grained credit assignment is particularly detrimental in long-horizon interactions, as it fails to distinguish which specific actions were critical to success versus those that were neutral or negative. Concurrently, policy-based self-distillation methods attempt to introduce finer-grained supervision by leveraging privileged information (PI) available only during training to re-evaluate sampled behaviors. However, this fine-grained supervision does not equate to fine-grained credit assignment. The likelihood changes induced by privileged information merely describe how additional information alters policy preferences, without directly determining how an executable action should inherit the final task outcome. This disconnect between supervision signals and credit assignment is identified as the "supervision-credit gap."

The core contribution of this research is the introduction of the TASPO framework, designed to bridge this gap by converting fine-grained supervision based on privileged information into outcome-based action credit. This approach enables agents to learn more precisely from successful experiences. The problem arises because existing methods create a mismatch where the granularity of the supervisory signal does not align with the granularity of the credit assigned to actions. TASPO addresses this by ensuring that the credit assignment mechanism retains the stability of result-oriented learning while integrating the precision of process supervision. By resolving this mismatch, the framework allows for a more accurate mapping of success to specific actions, thereby improving the overall efficiency and reliability of agent training in complex environments.

Deep Analysis

TASPO’s technical implementation centers on reconstructing the credit assignment mechanism to balance result-oriented stability with process supervision precision. The framework first extracts privileged information from validated successful experiences and constructs a PI representation suitable for the current decision stage. Crucially, the algorithm aggregates the policy likelihood changes induced by this privileged information at the granularity of executable actions. This step maps signals that might otherwise remain at the token level or be irrelevant to the current state onto the actual operations performed by the agent. By operating at the action level, TASPO ensures that the credit assignment is directly tied to the decisions that influenced the outcome, rather than abstract token probabilities.

Subsequently, TASPO transforms these relative action supports into positive, bounded weights that preserve the original mean, which are then applied to the original trajectory advantages. This design ensures that the validated final result dictates the direction and average scale of policy updates, while the privileged information’s role is limited to redistributing credit weights among different actions. This separation prevents privileged signals from causing misleading updates due to granularity mismatches or lack of result semantics. Mathematically, this guarantees the rationality and stability of credit assignment, allowing the policy optimization process to benefit simultaneously from macro-level result feedback and micro-level process guidance. The method effectively decouples the magnitude of the update from the fine-grained distribution, ensuring that the agent learns from the outcome without being distracted by irrelevant supervisory noise.

Industry Impact

To validate the effectiveness of TASPO, the research team conducted extensive experiments across three mainstream agent benchmarks, comparing its performance against the state-of-the-art GRPO (Group Relative Policy Optimization) method. The results demonstrate that TASPO outperforms GRPO by an average of 10.6% across these benchmarks, significantly surpassing the baseline models. More importantly, TASPO exhibits superior generalization capabilities on unseen tasks, indicating that the learned strategies are robust rather than merely overfitting to specific training scenarios. This improvement is not just a marginal gain but represents a substantial leap in agent performance, highlighting the framework's ability to handle complex, variable environments more effectively than previous methods.

Further ablation studies and in-depth analysis reveal that TASPO’s success is primarily attributed to its effective reduction of supervision mismatch. By assigning credit at the action level, the algorithm avoids erroneously attributing irrelevant PI signals to critical decisions, thereby stabilizing the convergence process of policy optimization. These experimental results not only prove TASPO’s performance advantages but also validate its theoretical correctness in solving long-horizon credit assignment problems. For the industry, this means that agents trained with TASPO are likely to exhibit greater reliability and adaptability in real-world deployments, where tasks often vary and require robust generalization rather than rote memorization of training paths.

Outlook

From the perspective of industry significance and potential impact, the introduction of TASPO provides new insights for the open-source community and industrial practitioners in agent training. As large model agents become increasingly applied in complex scenarios such as code generation and automated workflows, efficiently and stably optimizing their policies has become a key bottleneck. TASPO offers an optimization paradigm that balances stability and precision by decoupling result orientation from process supervision. This approach helps reduce resource consumption during agent training while enhancing final outcomes, making it a valuable tool for scaling agent applications.

For industrial implementation, models that generalize better to unseen tasks imply stronger adaptability and reliability in actual deployment. Furthermore, the identification of the supervision-credit gap may inspire subsequent research to explore more types of privileged information utilization and ways to more naturally integrate process supervision with result feedback. This not only enriches the toolkit for applying reinforcement learning in the agent domain but also lays a crucial methodological foundation for building smarter, more reliable AI agents. Future work may focus on extending TASPO to even longer horizons or integrating it with other advanced training techniques to further push the boundaries of agent intelligence and autonomy.