STARE is a reinforcement learning stabilization method for large language models that prevents policy entropy collapse. By analyzing first-order token-level gradients, it identifies critical tokens via batch surprise quantiles and selectively reweights their advantages, while using a target-entropy gate to keep entropy stable over thousands of training steps.

Why does STARE matter for LLM training?

Policy entropy collapse has long limited LLM exploration and caused training instability. Across models from 1.5B to 32B parameters, STARE maintains stable entropy and improves AIME24/AIME25 accuracy by 4%-8% over DAPO. Its open methodology enriches the RLHF toolkit while offering industry a way to reduce training crash risks and compute waste.

What should we watch next?

As LLMs tackle increasingly complex reasoning, maintaining diversity without sacrificing stability is crucial. The exploration-exploitation balance in STARE directly benefits creative tasks like open-domain QA and code generation. Future research can adapt its surprise-guided reweighting for credit assignment, enabling agents to learn continuously in open-ended environments.

STARE：基於驚奇度引導的Token級優勢重加權以穩定策略熵

針對大語言模型在基於可驗證獎勵的強化學習（如GRPO）訓練中普遍出現的策略熵崩潰問題，本文提出了一種名為STARE的新方法。透過一階梯度分析，作者揭示了Token級信用分配的不匹配現象，發現熵變由軌跡級優勢與熵敏感度函數乘積構成，呈現出優勢-驚奇度的四象限結構及近臨界特性。STARE利用批次內部的驚奇度分位數識別關鍵Token子集，選擇性重加權其有效優勢，並引入目標熵閉環門控機制以穩定熵調節。在1.5B至32B模型規模及短/長思維鏈、多輪工具使用等任務上，STARE在數千步訓練中維持了穩定的策略熵。在AIME24和AIME25基準上，其平均準確率較DAPO等基線提升4%-8%，且反思Token與響應長度同步增長，證實了探索與利用的平衡，為釋放強化學習的訓練潛力開闢了新途徑。

Sources

arXiv