Reliable search requires more than acquiring external evidence. An agent must also recognize and recover from errors as its trajectory unfolds. In-trajectory feedback provides a mechanism for such recovery by diagnosing where the search has drifted and redirecting subsequent reasoning steps. This is particularly important in long-horizon search, where an early directional error may receive no immediate corrective signal and can compound across later steps. Making such feedback learnable, however, creates a coupled problem: the agent must learn when to request and use feedback, while the critic must learn corrections from outcome-confounded rollouts as the agent's failure patterns evolve. We introduce CAFE (Coupled Agent--Feedback Evolution), a framework in which a shared-parameter model alternates between search-agent and critic roles. CAFE initializes feedback-conditioned recovery from trajectories built around the base agent's own failures, then couples online and offline optimization. During online RL, a comparative feedback estimate uses a prompt-level call--skip success gap to shape request returns, while feedback-aware advantage shaping reweights token advantages before and after feedback. Offline, rollout-derived preference optimization learns feedback from matched successful and unsuccessful trajectories. On seven agentic search benchmarks, CAFE outperforms the evaluated RL-based search agents on average, retains its gains across all six out-of-domain benchmarks, and reduces answer-level hallucinations. One-sided ablations show that improving only the agent or only the critic eventually plateaus, whereas alternating the two updates continues to improve performance. These findings suggest that a self-improving search agent needs feedback that co-evolves with the policy it guides.
Figures & tables
Figure 1: Overview of CAFE. One shared model serves as agent and critic. CFE shapes request returns from the call–skip gap; advantage shaping weights tokens before and after feedback. RDPO updates the critic from prefix-matched pairs mined from recent rollouts.
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
Method
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Closed-source Models
GPT-5-Mini ( 2025 )
67.4
76.0
59.0
72.0
27.2
36.5
34.6
41.1
73.0
81.2
62.4
73.2
31.2
42.6
50.7
60.4
Gemini-2.5-Flash ( 2025 )
63.8
73.6
54.0
67.6
24.4
35.7
40.2
49.2
73.4
81.5
57.6
65.9
39.8
53.0
50.5
60.9
Claude-4.5-haiku ( 2025 )
61.2
68.0
51.0
63.0
17.0
22.2
36.8
43.3
66.8
73.6
52.8
62.4
32.8
42.9
45.5
53.6
Large Open-source Models
Table 1: Main results on seven agentic SearchQA benchmarks. Best and second-best results are bolded and underlined . † Released checkpoint evaluated under our protocol. ∗ Results reported in the original paper. ‡ Our reproduction initialized from our SFT checkpoint.
Online RL
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
CFE
Adv. Shaping
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
✗
✗
80.6
86.6
50.8
61.0
27.2
36.6
44.6
49.0
60.4
67.9
45.6
56.2
38.4
48.6
49.7
58.0
✓
✗
83.2
88.0
52.4
63.0
28.4
36.5
45.0
48.9
60.8
68.5
47.2
56.6
38.7
47.4
50.8
58.4
✗
✓
83.2
88.2
52.4
63.2
30.6
38.5
44.6
48.9
61.8
69.6
47.2
58.9
38.6
48.2
51.2
59.4
✓
✓
83.4
88.3
53.0
63.9
31.8
40.5
46.2
50.8
61.2
69.2
48.0
60.5
40.0
48.8
51.9
60.3
Table 2: Online optimization ablation across seven benchmarks. Best and second-best results are bolded and underlined . CFE augments the task reward with the prompt-level call–skip success gap and a repeated-request cost; Adv. Shaping reweights pre- and post-feedback token advantages.
Online RL
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
CFE
Adv. Shaping
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Rollout-Derived SFT (RSFT)
✗
✗
80.0
85.1
54.2
64.0
26.8
35.9
46.0
51.8
60.8
68.3
50.4
59.7
36.6
45.9
50.7
58.7
✓
✗
80.6
85.2
51.4
62.0
26.4
35.4
43.2
47.5
61.0
67.8
44.8
56.3
36.8
47.3
49.2
57.4
✗
✓
79.8
85.7
51.0
62.2
26.4
35.0
44.2
49.0
62.6
69.9
49.6
58.8
40.6
49.3
50.6
58.6
✓
✓
82.4
87.2
52.4
63.1
27.2
36.2
46.0
49.8
63.2
70.0
48.0
56.2
39.0
49.1
51.2
58.8
Table 3: Offline optimization ablation under a matched five-iteration schedule with 100 online RL steps per iteration. Best and second-best results are bolded and underlined . Row groups specify the offline objective (RSFT or RDPO), while the columns indicate the online RL components used to train the starting checkpoint.
Figure 2: Analysis of co-evolution and feedback-aware credit assignment. In panel (a), agent-only optimization pairs the evolving agent with the frozen SFT feedback model, whereas feedback-only optimization pairs the evolving feedback model with the frozen SFT agent. For CAFE, open markers denote intermediate online-RL checkpoints and filled markers denote the RDPO checkpoints.
Figure 3: Agent–critic cross-play on 2Wiki. Rows and columns denote agent and critic training iterations, respectively. Iteration 0 is the shared SFT initialization. Cells report EM or token-level F1, and red boxes mark same-iteration pairs.
Figure 4: Analysis of answer grounding and feedback evolution. (a) Hallucination rates across seven benchmarks, where lower is better. (b) Dominant feedback terms during early, middle, and late training.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Method
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Existing 3B Search Agents
Search-R1-3B ∗ ( 2025a )
58.8
68.1
46.2
57.8
24.4
32.9
37.0
43.5
56.6
63.2
41.6
53.9
34.4
44.1
42.7
51.9
R-Search-3B ∗ ( 2026 )
65.0
72.6
43.4
54.4
25.8
34.8
37.0
44.9
56.0
64.0
37.6
49.8
35.2
46.0
42.9
52.4
CAFE (Qwen2.5-3B-Instruct)
Qwen2.5-3B-Instruct
11.4
27.6
14.6
25.8
3.8
9.2
11.6
18.2
18.8
32.1
12.8
22.9
3.8
12.8
11.0
21.2
Appendix
Table 4: Results for 3B-scale models. Best and second-best results are bolded and underlined, respectively.
Method
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
Base
36.60
32.80
35.27
24.20
22.29
18.40
39.60
29.88
GRPO
11.00
19.40
26.40
10.82
14.00
16.00
25.80
17.63
CAFE
10.00
16.00
17.00
8.00
9.40
12.80
15.00
12.60
Appendix
Table 5: Hallucination rates (%, lower is better) across seven benchmarks. Best and second-best results are bolded and underlined, respectively.
2Wiki
HotpotQA
MuSiQue
PopQA
TriviaQA
Bamboogle
NQ
Avg.
Schedule
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
50×10
81.4
85.5
51.6
62.8
27.6
37.4
46.0
50.9
58.6
67.1
48.8
58.2
34.0
43.8
49.7
58.0
100×5
84.0
89.2
53.4
64.8
30.2
39.1
46.4
51.6
62.6
69.9
50.4
61.4
40.8
49.2
52.5
60.7
250×2
81.6
86.5
53.4
63.1
28.2
37.4
44.0
47.6
59.4
66.3
45.6
57.3
36.8
46.4
49.9
57.8
Appendix
Table 6: Iteration-schedule ablation under 500 online RL steps. A schedule H×K performs H online RL steps followed by one RDPO update and repeats this cycle K times. Best and second-best results are bolded and underlined, respectively.
Figure 5: Online training dynamics through 500 RL steps. (a) Test accuracy on 2Wiki. (b) Prompt-level empirical entropy of feedback policy.