CAFE: Self-Improving Search Agents Require Co-Evolving Feedback

Background and Context

Result-supervised search agents face a structural flaw in their training signal. These agents learn when and how to retrieve evidence, yet terminal rewards can only score the final answer. They cannot localize errors in intermediate retrieval steps, nor can they correct an ongoing trajectory before those errors accumulate into a wrong result. This leaves a large window of silent failure during reasoning where no corrective signal ever arrives.

The paper reframes corrective feedback as a learned, within-trajectory intervention that couples two roles: the agent and the critic. The agent must decide when to request and use feedback, while the critic must infer useful corrections from trajectories whose outcomes are confounded and whose failure modes shift as the agent improves. As the agent gets better, the kinds of mistakes it makes change, so a static critic quickly becomes misaligned with the agent's actual needs.

To address this, the authors propose CAFE, short for Coupled Agent-Feedback Evolution. The framework uses a single shared-parameter model that alternates between the search-agent and critic roles, forcing feedback and policy to co-evolve rather than being optimized independently. The core claim is that a self-improving search agent requires feedback that evolves in lockstep with the strategy it guides.

Deep Analysis

CAFE initializes feedback-conditioned recovery from trajectories built around the base agent's own failures, so the feedback signal is aimed directly at the failure modes the agent actually exhibits. Online and offline optimization are then coupled into a single loop: online exploration generates trajectories, and offline processing extracts preferences from them.

In the online reinforcement-learning phase, the authors introduce a comparative feedback estimate. A prompt-level call-skip success gap shapes the reward for requesting feedback by comparing the success rate when feedback is requested against the success rate when it is skipped. A separate feedback-advantage reweighting adjusts token-level advantages both before and after feedback is applied, distributing credit more precisely.

In the offline phase, feedback is learned from matched success and failure trajectories derived from rollouts using preference optimization. The model learns to distinguish effective corrections by contrasting positive and negative examples side by side. This closed loop avoids the sparse-reward problem of terminal-only supervision and lets corrections intervene early.

Industry Impact

CAFE was evaluated across seven agentic search benchmarks and compared against previously evaluated reinforcement-learning search agents. On average it outperformed them, and it maintained its gains on six out-of-domain benchmarks, indicating the improvement is not overfit to a single task distribution but generalizes across domains. It also reduced answer-level hallucination, suggesting tighter consistency between retrieval and reasoning.

Ablations with a single side reveal why the coupling matters. Improving only the agent or only the critic tends to stall, whereas alternately updating both yields continuous improvement. This directly supports the paper's central thesis that feedback and policy must co-evolve, and that any single-point optimization eventually hits a ceiling.

For open-source work on self-improving agents, the framework reframes feedback as something that should evolve with the policy rather than remain a static external signal. The ideas of feedback-conditioned recovery and comparative feedback estimation can also transfer to other reinforcement-learning settings that require online decision-making.

Outlook

The finding that single-sided improvement stalls while alternating updates keep improving points researchers toward modeling the dynamic relationship between agent and critic instead of optimizing components in isolation. This provides a methodological foundation for building agents that genuinely self-improve over time.

As retrieval-and-reasoning agents expand into question answering, research, and coding, the ability to self-improve becomes a central engineering challenge. CAFE offers a reusable template for sustaining that improvement through shared parameters and coupled optimization.

The stability the framework shows on out-of-domain benchmarks, combined with reduced hallucination, strengthens the case for deploying such systems in real-world settings where reliability matters. The next step is testing whether the co-evolution loop holds up at larger scale and across more demanding agentic tasks.