SafeEvolve: A Synergistic Evolution Framework for Safety Harness and Policy Based on Agent Experience

Background and Context

Large language model agents operating in complex, dynamic environments face a dual-layered security challenge that traditional alignment mechanisms struggle to address. The risks are twofold: the generation of harmful final responses and the emergence of unsafe behaviors within multi-step execution trajectories. Existing safety alignment strategies typically adopt a siloed approach, relying either on static external updates to the execution harness or on isolated internal optimizations of the policy model. This fragmentation creates a significant gap between runtime control mechanisms and the model's intrinsic safety capabilities. When an agent encounters novel or adversarial scenarios, the lack of synergy between the external environment configuration and the internal decision-making logic often leads to vulnerabilities that static rules cannot mitigate.

The core issue lies in the static nature of current safety interventions. Harnesses, which encompass prompt structures, tool-calling rules, and skill libraries, are often updated manually or through rigid protocols that do not adapt to the specific failure modes observed during agent operation. Conversely, policy optimization frequently ignores the structural context provided by the harness, treating safety as a purely internal property of the model weights. This disconnect prevents agents from developing a holistic understanding of safety, where the environment and the agent evolve together to handle emerging threats. Consequently, agents remain brittle, failing to balance operational utility with robust security in real-time interactions.

To bridge this divide, the SafeEvolve framework introduces an experience-driven self-evolving safety alignment mechanism. Rather than treating safety as a one-time configuration step, SafeEvolve leverages the safety experiences accumulated from completed policy trajectories to drive the continuous, synergistic evolution of both the harness and the policy. This approach transforms safety from a static barrier into a dynamic, adaptive asset. By mining historical interaction data, the framework identifies specific points of failure and updates both the external harness components and the internal policy parameters in a coordinated manner. This ensures that the agent not only avoids known pitfalls but also learns to navigate new risks by actively utilizing evolved safety structures.

Deep Analysis

SafeEvolve’s technical architecture is defined by a tightly coupled co-evolution mechanism that operates across two distinct but interconnected dimensions: the harness side and the policy side. On the harness side, the framework moves beyond simple prompt concatenation by transforming trajectory-level safety evidence into auditable, reversible, and bounded component-level updates. The system analyzes interaction logs to pinpoint exact steps where safety violations occurred, then modifies specific elements of the safety prompts or the hierarchical skill library. This granular approach ensures that harness evolution is transparent and controllable, avoiding the opacity of black-box rule accumulation while maintaining the ability to revert changes if they negatively impact performance.

On the policy side, SafeEvolve employs a two-stage Supervised Fine-Tuning and Reinforcement Learning (SFT-RL) paradigm designed to internalize safety behaviors. In the first stage, the policy model undergoes supervised fine-tuning using data generated by the evolved harness. This step is critical as it guides the model to actively recognize and utilize the updated safety assets, fostering a sensitivity to the safe environment. The second stage introduces harness-enhanced reinforcement learning, where a verifier-decomposed reward mechanism evaluates the safety of each action during multi-step exploration. This fine-grained reward shaping encourages the agent to make autonomous safety decisions at every step, rather than relying solely on external constraints, effectively shifting the paradigm from passive defense to proactive safety reasoning.

The synergy between these two components is validated through ablation studies, which demonstrate that neither harness updates nor policy optimization alone can achieve the comprehensive performance gains observed in the full framework. The harness provides the structured safety context, while the policy learns to navigate within that context. This mutual reinforcement ensures that the agent develops a robust internal safety schema that is aligned with the external runtime environment. The result is a system where the agent learns not just what not to do, but how to execute tasks more safely, integrating safety into the very fabric of its decision-making process.

Industry Impact

Empirical evaluations of SafeEvolve on benchmark datasets, particularly AgentDojo, highlight its superior ability to balance security and utility. In tests involving the Qwen3.5-4B model, SafeEvolve reduced the attack success rate by a factor of three, demonstrating exceptional robustness against adversarial inputs. This significant reduction in vulnerability is achieved without compromising the agent's ability to perform legitimate tasks. In fact, the utility score for benign tasks increased from 59.79% to 61.86%, indicating that the safety mechanisms enhance rather than hinder operational efficiency. This outcome challenges the prevailing notion that stricter safety controls inevitably lead to reduced agent capability, proving that co-evolution can yield a net positive in both security and performance metrics.

For the open-source community, SafeEvolve offers a reusable and auditable framework for safety evolution. Developers can leverage historical interaction data to continuously refine agent safety configurations without the prohibitive costs associated with retraining large foundation models. This democratizes access to advanced safety mechanisms, allowing smaller teams to implement enterprise-grade security protocols. The framework’s emphasis on reversible and auditable updates also aligns with the growing demand for transparency in AI systems, providing a clear trail of how safety rules have evolved over time.

In industrial applications, particularly in high-stakes sectors like finance and healthcare, SafeEvolve provides a compliant and transparent method for managing agent risks. The ability to trace safety updates back to specific trajectory evidence satisfies regulatory requirements for explainability and accountability. By ensuring that agent behavior remains within defined safety boundaries while adapting to new operational contexts, SafeEvolve enables the deployment of autonomous agents in environments where errors are not an option. This capability is crucial for scaling AI agents from experimental prototypes to critical infrastructure components.

Outlook

The introduction of SafeEvolve marks a paradigm shift in AI safety governance, moving from static rule-based defenses to dynamic, experience-driven evolution. This shift implies that future research must focus less on isolated model weight adjustments and more on the adaptive optimization of the dynamic interaction environment. The framework suggests that safety is not a fixed state but a continuous process of adaptation, where the agent and its environment co-evolve to meet emerging challenges. This perspective opens new avenues for research into self-improving safety systems that can learn from their own failures in real-time.

Furthermore, the success of SafeEvolve in enhancing benign task utility suggests that safety and capability are not mutually exclusive but can be mutually reinforcing. As AI agents become more autonomous and integrated into complex workflows, the ability to maintain high utility while ensuring robust safety will be a key differentiator. SafeEvolve provides a blueprint for achieving this balance, demonstrating that safety can be an enabler of performance rather than a constraint. This insight is likely to influence the design of next-generation AI systems, where safety is embedded into the learning loop from the outset.

Looking ahead, the principles underlying SafeEvolve could be extended to other domains requiring high-reliability autonomous systems. The mechanisms for trajectory-level evidence analysis and component-level harness updates may prove valuable in robotics, autonomous driving, and other fields where multi-step decision-making under uncertainty is critical. By establishing a standardized approach to co-evolutionary safety, SafeEvolve lays the groundwork for a new generation of trustworthy and resilient AI agents capable of operating safely in the unpredictable real world.