Retrieval-augmented agents can query external evidence, yet their reliability in multi-step reasoning remains limited: noisy retrieval may derail multi-hop question answering, while outcome-only reinforcement learning provides credit signals that are too coarse to optimize intermediate steps. We propose EvalAct (Evaluate-as-Action), which converts implicit retrieval quality assessment into an explicit action and enforces a coupled Search-to-Evaluate protocol so that each retrieval is immediately followed by a structured evaluation score, yielding process signals aligned with the interaction trajectory. To leverage these signals, we introduce Process-Calibrated Advantage Rescaling (PCAR), a GRPO-based optimization method that rescales advantages at the segment level according to evaluation scores, emphasizing reliable segments while updating uncertain ones conservatively. Experiments on seven open-domain QA benchmarks show that EvalAct achieves the best average accuracy, with the largest gains on multi-hop tasks, and ablations verify that the explicit evaluation loop drives the primary improvements while PCAR provides consistent additional benefits.
Figures & tables
Figure 1: Overview of EvalAct with PCAR. The agent follows a coupled Search → Evaluate protocol, producing segment-wise self-evaluation scores {zi,k} that PCAR uses to rescale GRPO advantages.
Single-Hop QA
Multi-Hop QA
Avg.
Method
NQ
PopQA
TriviaQA
2Wiki
Bamboogle
HotpotQA
MuSiQue
Backbone: Qwen2.5-3B-Instruct
Direct Generation
10.6
10.8
28.8
24.4
2.4
14.9
2.0
13.4
IRCoT
11.1
20.0
31.2
17.1
24.0
16.4
6.7
18.1
Search-o1
23.8
26.2
47.2
21.8
32.0
22.1
5.4
25.5
Naïve RAG
34.8
38.7
54.4
22.6
8.0
25.5
4.7
27.0
Table 1: Main results (EM, %) on seven open-domain QA benchmarks. Bold and underlined values indicate the best and second-best performance, respectively.
Figure 2: Effect of SFT on Format Alignment. The Base model exhibits high tool parsing failure rates.
Model Variant
SFT
RL Paradigm
Avg. EM
1. Base (Instruct)
✗
-
14.2
2. SFT-Only
✓
-
24.8
3. Base + RL (Vanilla)
✗
Standard
33.1
4. SFT + RL (Vanilla)
✓
Standard
33.5
5. Base EvalAct
✗
EvalAct
17.1
6. EvalAct (Ours)
✓
EvalAct
41.0
Table 2: Ablation of training stages and paradigms on multi-hop benchmarks (Avg. EM, %). Vanilla denotes standard RL without the explicit evaluation loop.
Figure 3: Training curves and ablation overview. (a) Training curves of EvalAct with 3B/7B backbones. (b) Model ablation across training variants. (c) Method ablation on removing the evaluation loop or PCAR. (d) Sensitivity to PCAR rescaling intensity.
Method
2Wiki
Bam.
Hotpot
Mus.
Avg.
EvalAct
50.0
48.0
44.3
21.6
41.0
w/o PCAR
48.2
46.2
44.1
20.8
39.8
w/o Eval Loop
41.4
37.2
38.9
16.5
33.5
Table 3: Method ablation on multi-hop benchmarks (EM, %).
Setting
Params
RIR
Avg.
Δ
EvalAct (Low)
0.05/0.25
1.67
39.8
-0.3
EvalAct (Mid)
0.10/0.50
3.0
40.1
-
EvalAct (High)
0.20/1.00
200
39.4
-0.7
Table 4: Parameter ablation on PCAR intensity. RIR is the ratio between max/min multipliers.