Reliable search requires more than acquiring external evidence. An agent must also recognize and recover from errors as its trajectory unfolds. In-trajectory feedback provides a mechanism for such recovery by diagnosing where the search has drifted and redirecting subsequent reasoning steps. This is particularly important in long-horizon search, where an early directional error may receive no immediate corrective signal and can compound across later steps. Making such feedback learnable, however, creates a coupled problem: the agent must learn when to request and use feedback, while the critic must learn corrections from outcome-confounded rollouts as the agent's failure patterns evolve. We introduce CAFE (Coupled Agent--Feedback Evolution), a framework in which a shared-parameter model alternates between search-agent and critic roles. CAFE initializes feedback-conditioned recovery from trajectories built around the base agent's own failures, then couples online and offline optimization. During online RL, a comparative feedback estimate uses a prompt-level call--skip success gap to shape request returns, while feedback-aware advantage shaping reweights token advantages before and after feedback. Offline, rollout-derived preference optimization learns feedback from matched successful and unsuccessful trajectories. On seven agentic search benchmarks, CAFE outperforms the evaluated RL-based search agents on average, retains its gains across all six out-of-domain benchmarks, and reduces answer-level hallucinations. One-sided ablations show that improving only the agent or only the critic eventually plateaus, whereas alternating the two updates continues to improve performance. These findings suggest that a self-improving search agent needs feedback that co-evolves with the policy it guides.
Figures & tables
Figure 1: Overview of CAFE. One shared model serves as agent and critic. CFE shapes request returns from the call–skip gap; advantage shaping weights tokens before and after feedback. RDPO updates the critic from prefix-matched pairs mined from recent rollouts.
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
Method
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Closed-source Models
GPT-5-Mini ( 2025 )
67.4
76.0
59.0
72.0
27.2
36.5
34.6
41.1
73.0
81.2
62.4
73.2
31.2
42.6
50.7
60.4
Gemini-2.5-Flash ( 2025 )
63.8
73.6
54.0
67.6
24.4
35.7
40.2
49.2
73.4
81.5
57.6
65.9
39.8
53.0
50.5
60.9
Claude-4.5-haiku ( 2025 )
61.2
68.0
51.0
63.0
17.0
22.2
36.8
43.3
66.8
73.6
52.8
62.4
32.8
42.9
45.5
53.6
Large Open-source Models
Table 1: Main results on seven agentic SearchQA benchmarks. Best and second-best results are bolded and underlined . † Released checkpoint evaluated under our protocol. ∗ Results reported in the original paper. ‡ Our reproduction initialized from our SFT checkpoint.
Online RL
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
CFE
Adv. Shaping
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
✗
✗
80.6
86.6
50.8
61.0
27.2
36.6
44.6
49.0
60.4
67.9
45.6
56.2
38.4
48.6
49.7
58.0
✓
✗
83.2
88.0
52.4
63.0
28.4
36.5
45.0
48.9
60.8
68.5
47.2
56.6
38.7
47.4
50.8
58.4
✗
✓
83.2
88.2
52.4
63.2
30.6
38.5
44.6
48.9
61.8
69.6
47.2
58.9
38.6
48.2
51.2
59.4
✓
✓
83.4
88.3
53.0
63.9
31.8
40.5
46.2
50.8
61.2
69.2
48.0
60.5
40.0
48.8
51.9
60.3
Table 2: Online optimization ablation across seven benchmarks. Best and second-best results are bolded and underlined . CFE augments the task reward with the prompt-level call–skip success gap and a repeated-request cost; Adv. Shaping reweights pre- and post-feedback token advantages.
Online RL
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
CFE
Adv. Shaping
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Rollout-Derived SFT (RSFT)
✗
✗
80.0
85.1
54.2
64.0
26.8
35.9
46.0
51.8
60.8
68.3
50.4
59.7
36.6
45.9
50.7
58.7
✓
✗
80.6
85.2
51.4
62.0
26.4
35.4
43.2
47.5
61.0
67.8
44.8
56.3
36.8
47.3
49.2
57.4
✗
✓
79.8
85.7
51.0
62.2
26.4
35.0
44.2
49.0
62.6
69.9
49.6
58.8
40.6
49.3
50.6
58.6
✓
✓
82.4
87.2
52.4
63.1
27.2
36.2
46.0
49.8
63.2
70.0
48.0
56.2
39.0
49.1
51.2
58.8
Table 3: Offline optimization ablation under a matched five-iteration schedule with 100 online RL steps per iteration. Best and second-best results are bolded and underlined . Row groups specify the offline objective (RSFT or RDPO), while the columns indicate the online RL components used to train the starting checkpoint.
Figure 2: Analysis of co-evolution and feedback-aware credit assignment. In panel (a), agent-only optimization pairs the evolving agent with the frozen SFT feedback model, whereas feedback-only optimization pairs the evolving feedback model with the frozen SFT agent. For CAFE, open markers denote intermediate online-RL checkpoints and filled markers denote the RDPO checkpoints.
Figure 3: Agent–critic cross-play on 2Wiki. Rows and columns denote agent and critic training iterations, respectively. Iteration 0 is the shared SFT initialization. Cells report EM or token-level F1, and red boxes mark same-iteration pairs.
Figure 4: Analysis of answer grounding and feedback evolution. (a) Hallucination rates across seven benchmarks, where lower is better. (b) Dominant feedback terms during early, middle, and late training.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Method
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Existing 3B Search Agents
Search-R1-3B ∗ ( 2025a )
58.8
68.1
46.2
57.8
24.4
32.9
37.0
43.5
56.6
63.2
41.6
53.9
34.4
44.1
42.7
51.9
R-Search-3B ∗ ( 2026 )
65.0
72.6
43.4
54.4
25.8
34.8
37.0
44.9
56.0
64.0
37.6
49.8
35.2
46.0
42.9
52.4
CAFE (Qwen2.5-3B-Instruct)
Qwen2.5-3B-Instruct
11.4
27.6
14.6
25.8
3.8
9.2
11.6
18.2
18.8
32.1
12.8
22.9
3.8
12.8
11.0
21.2
Appendix
Table 4: Results for 3B-scale models. Best and second-best results are bolded and underlined, respectively.
Method
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
Base
36.60
32.80
35.27
24.20
22.29
18.40
39.60
29.88
GRPO
11.00
19.40
26.40
10.82
14.00
16.00
25.80
17.63
CAFE
10.00
16.00
17.00
8.00
9.40
12.80
15.00
12.60
Appendix
Table 5: Hallucination rates (%, lower is better) across seven benchmarks. Best and second-best results are bolded and underlined, respectively.
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
Schedule
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
50×10
81.4
85.5
51.6
62.8
27.6
37.4
46.0
50.9
58.6
67.1
48.8
58.2
34.0
43.8
49.7
58.0
100×5
84.0
89.2
53.4
64.8
30.2
39.1
46.4
51.6
62.6
69.9
50.4
61.4
40.8
49.2
52.5
60.7
250×2
81.6
86.5
53.4
63.1
28.2
37.4
44.0
47.6
59.4
66.3
45.6
57.3
36.8
46.4
49.9
57.8
Appendix
Table 6: Iteration-schedule ablation under 500 online RL steps. A schedule H×K performs H online RL steps followed by one RDPO update and repeats this cycle K times. Best and second-best results are bolded and underlined, respectively.
Figure 5: Online training dynamics through 500 RL steps. (a) Test accuracy on 2Wiki. (b) Prompt-level empirical entropy of feedback policy.
Recent LLM search agents use reinforcement learning with verifiable rewards (RLVR) to learn search-augmented reasoning from outcome rewards. On hard problems, these agents rarely sample end-to-end successful rollouts, leaving outcome-only RLVR with few positive-reward trajectories. We argue that improving learning on such problems requires additional guidance during training, and RLVR already contains verifier-side information that can provide it. This information can identify errors or omissions in the agent's submitted answer and guide revision within the rollout. We propose a training-time mechanism called Credit-Attenuated Privileged Feedback (CAPF), which makes this verifier-side information available through a Privileged Feedback call during training. CAPF lets the policy revise zero-reward attempts into positive-reward repair trajectories and attenuates credit for the feedback call and earlier actions to accommodate deployment without this call. Empirical research demonstrates that CAPF improves Qwen3-4B's average exact-match score from 44.7% under outcome-only RLVR to 48.5% on seven open-domain QA benchmarks.
On-policy self-distillation (OPSD) offers a promising approach for training large language models without relying on a separate teacher model. However, its effectiveness on complex agentic tasks remains largely unexplored. In this work, we instantiate Feedback-Augmented Self-Distillation (FA-SD), a self-distillation algorithm for agentic search that leverages successful demonstrations as privileged information. We identify that models can rely on recurring reasoning-and-search output templates, producing trajectories that appear diverse but are largely agnostic to the input question, making the KL-based self-distillation signal uninformative. We term this phenomenon decoding collapse, a failure mode that can be missed by existing evaluation metrics. To understand its underlying cause, we show that although the self-teacher achieves stronger performance, learning remains inherently unstable due to inconsistent supervision signals. We further decompose this inconsistency into model inconsistency and prompt inconsistency, and show that the latter can significantly degrade the quality of the supervision signal, limiting the effectiveness of self-teacher learning. To mitigate this inconsistency, we introduce an exponential moving average (EMA) teacher to stabilize the self-teacher and provide more consistent supervision signals. Although the EMA teacher requires a warm-up phase during which performance may temporarily regress, it ultimately improves model performance by providing more stable supervision.
Fan Yang, Rui Meng, Yuxin Wen
Chapman University, Orange, CA, USA · Lawrence Berkeley National Laboratory, Berkeley, CA, USA
Agentic search systems iteratively interact with retrieval models to answer complex queries. Despite substantial progress, optimizing retrievers for agentic search remains challenging, often requiring heavy co-training or gold-standard annotations that limit real-world applicability. We propose Critic-R, a framework that explicitly closes the feedback loop between the reasoning agent and the retrieval model during both inference and training. Critic-R introduces a critic model that evaluates the agent's introspective reasoning trace after consuming retrieved evidence to determine whether the retrieved context sufficiently supports the next reasoning step. Critic-R has two complementary mechanisms: Critic-R-Zero, an inference-time query refinement loop that iteratively rewrites queries and retrieval instructions, and Critic-Embed, an optimization approach for retrieval models that leverages successful and failed refinement trajectories as automatic supervision without requiring manual relevance annotation. We evaluate Critic-R on HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle. Results show that Critic-R significantly improves both retrieval quality and downstream answer accuracy.
Md Zarif Ul Alam, Alireza Salemi, Hamed Zamani
Center for Intelligent Information Retrieval University of Massachusetts Amherst United States