What is the TRACE algorithm and what problem does it solve?

TRACE (Turn-level Reward Assignment via Credit Estimation) is a dense credit assignment mechanism for agent reinforcement learning. It reformulates trajectories into state transitions bounded by tool-calling events and derives turn-level rewards from a frozen reference model's log probabilities, solving the sparse-reward problem in long-horizon tasks.

How does TRACE improve on traditional reinforcement learning approaches?

Unlike traditional RL that relies on outcome-only rewards causing high variance and misleading credit, TRACE requires no additional critic model training or process-level annotations. It automatically collapses redundant tool calls and achieves fine-grained reward allocation through offline log-probability computation, significantly improving learning efficiency.

What are the next steps or future directions for TRACE?

TRACE's framework generalizes well to code generation, automated workflows, and other long-horizon decision tasks. Experiments show successful transfer from closed to open-network benchmarks, offering a low-cost new paradigm for agent training without expensive data labeling or cold-start fine-tuning.

TRACE：基於信用估計的長程智能體回合級獎勵分配方法

針對多輪智能體在長程任務中面臨的獎勵稀疏與信用分配難題，本文提出TRACE方法。傳統結局獎勵在長軌跡中方差高且易誤導，TRACE通過將軌跡轉化為工具調用邊界的狀態轉移序列，利用凍結參考模型的日誌概率構建對數比率狀態值估計，並據此推導基於時序差分變化的逐動作獎勵。該方法無需額外的批評家模型訓練，也不依賴過程標籤，能自動折疊冗餘調用步驟。在BrowseComp-Plus等基準測試上，該方法使Qwen3系列模型的搜尋能力從7.2大幅提升至35.6，無需冷啟動微調或在線數據訓練，展現了高效的強化學習優化潛力。

Sources

arXiv