Reinforcement Learning (RL) has shown strong potential for optimizing search agents in complex information retrieval tasks. However, existing approaches predominantly rely on gold supervision, such as ground-truth answers, which is difficult to scale. To address this limitation, we propose Cycle-Consistent Search (CCS), a gold-supervision-free framework for training search agents, inspired by cycle-consistency techniques from unsupervised machine translation and image-to-image translation. Our key hypothesis is that an optimal search trajectory, unlike insufficient or irrelevant ones, serves as a lossless encoding of the question's intent. Consequently, a high-quality trajectory should preserve the information required to accurately reconstruct the original question, thereby inducing a reward signal for policy optimization. However, naive cycle-consistency objectives are vulnerable to information leakage, as reconstruction may rely on superficial lexical cues rather than the underlying search process. To reduce this effect, we apply information bottlenecks, including exclusion of the final response and named entity recognition (NER) masking of search queries. These constraints force reconstruction to rely on retrieved observations together with the structural scaffold, ensuring that the resulting reward signal reflects informational adequacy rather than linguistic redundancy. Experiments on question-answering benchmarks show that CCS achieves performance comparable to supervised baselines while outperforming prior methods that do not rely on gold supervision. These results suggest that CCS provides a scalable training paradigm for training search agents in settings where gold supervision is unavailable.
Figures & tables
Figure 1: Examples of search trajectories. A high-quality trajectory (left) constitutes a lossless encoding of the question’s intent, preserving all necessary information required to reconstruct the original question. In contrast, trajectories that involve insufficient (middle) or irrelevant (right) search steps fail to preserve this information, resulting in incomplete or distorted representations of the question.
Figure 2: Overview of the Cycle-Consistent Search (CCS) framework. The agent πθ generates search trajectories τ to address a question q . To ensure the search process is driven by information gain rather than lexical copying, τ is passed through a strategic information bottleneck ( ψ ) that removes the aT and masks entities within actions (highlighted in grey). A frozen reconstructor Pϕ then attempts to recover the original question from τ~ . The agent is optimized using GRPO based on the semantic similarity between q and q^ , encouraging trajectories that capture sufficient evidence.
Methods
General QA
Multi-Hop QA
Avg.
NQ †
TriviaQA ∗
PopQA ∗
HotpotQA †
2wiki ∗
Musique ∗
Bamboogle ∗
Qwen2.5-7B-Instruct
Model Inference
Direct Inference
0.344
0.111
0.144
0.279
0.248
0.091
0.120
0.191
CoT
0.430
0.139
0.203
0.325
0.318
0.128
0.384
0.275
IRCoT
0.608
0.761
0.515
0.486
0.279
0.129
0.304
0.440
Table 1: Main results. The best and second-best performances are shown in bold and underline, respectively. † / ⋆ denotes in-domain/out-of-domain datasets.
Methods
General QA
Multi-Hop QA
Avg.
NQ †
TriviaQA ∗
PopQA ∗
HotpotQA †
2wiki ∗
Musique ∗
Bamboogle ∗
Actions + Observations + Final Response
0.700
0.852
0.618
0.543
0.476
0.202
0.536
0.561
Actions + Observations
0.679
0.852
0.598
0.526
0.474
0.201
0.488
0.545
Observations
0.691
0.866
0.610
0.589
0.588
0.225
0.520
0.584
Masked Actions + Observations
0.712
0.880
0.639
0.598
0.567
0.239
0.608
0.606
Table 2: Ablation studies on the components of CCS. The best and second-best performances are shown in bold and underline, respectively.
Figure 3: Qualitative analysis of search trajectories. CCS assigns low rewards to trajectories exhibiting (a) Information Void, where the retrieved observations fail to satisfy key entity constraints, and (b) Shallow Depth, where the agent fails to complete the full multi-hop search scaffold. By contrast, high rewards are assigned only to trajectories that preserve both the search structure and the supporting observations needed to faithfully reconstruct the original question. Masked spans are highlighted in gray.
Figure 4: Performance comparison on the ResearchRubrics ( Sharma et al., 2025 ) benchmark for open-ended Deep Research tasks. Scores are evaluated using the fine-grained rubric-based framework of Sharma et al. (2025) , with Gemini 2.5 Pro as the judge, and are reported as domain-wise scores and the overall average across ten domains.
Search agents enable Large Language Models (LLMs) to iteratively retrieve and use information for complex multi-hop questions. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach for post-training such agents, but its reliance on sparse, outcome-based supervision can make credit assignment difficult and limit learning efficiency. In this paper, we systematically investigate how intermediate supervision can improve reinforcement learning for search agents. We study a range of reward-shaping and credit-assignment strategies that provide learning signals from intermediate retrieval steps. Building on these insights, we develop a training framework that combines intermediate signals with final outcome rewards to improve learning from multi-step search trajectories. Experiments across multiple benchmarks under matched training conditions demonstrate improvements in aggregate search-agent performance and show that both the choice of intermediate signal and where its credit is assigned affect training behaviour. These findings show that reward design and credit assignment are important design dimensions for training effective search agents.
Wenyu Huang, Xinyu Hou, Pavlos Vougiouklis +2
University of Edinburgh, UK · Huawei Technologies Research & Development (UK) Limited
Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence. Yet terminal correctness assigns equal credit to direct answers from parametric knowledge, evidence-grounded answers, and answers recalled after redundant search. Process supervision and LLM judges provide richer feedback but can require high-cost annotation or inference; low-cost, non-contrastive policy confidence signals do not directly identify whether retrieved evidence changes an answer. We propose Search-G1, a reward framework that couples retrieval necessity with evidence reliance through two intervention-calibrated hidden-state readouts for selective grounded search. A prompt readout predicts closed-book sufficiency, whose complement proxies policy-relative retrieval necessity; an answer-commit readout predicts evidence reliance as sensitivity to evidence deletion conditional on the realized reasoning prefix. A correctness-first reward gates evidence-reliance credit by retrieval necessity, rewards sufficient direct answers, and penalizes repeated search. After calibration, scoring needs neither process annotations nor LLM-judge inference. As policy-relative targets, trajectory distributions, and representations may drift, Search-G1 periodically regenerates targets and refits its readouts. Across four QA benchmarks and two scales, Search-G1 achieves competitive task accuracy, with stronger counterfactual evidence dependence and fewer searches in the main comparisons, complemented by independent human audits of answer support.
Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang +7
Fudan University · Tencent · Nanjing University +2
Search agents powered by large language models can autonomously decompose queries, retrieve information, and synthesize answers through multi-step reasoning. However, the rapid growth of training methods has outpaced controlled comparison: existing works differ in retrieval corpora, reward designs, and training protocols, making it unclear what actually drives improvements. We present a controlled empirical study that isolates three under-explored dimensions of search agent training. First, we identify a critical data-coverage issue in the widely used Wikipedia 2018 corpus and show that correcting it alone yields larger gains than the differences between training algorithms. Second, we systematically compare outcome-based and process-based reward methods across three base models, finding that the simplest outcome-based approach achieves competitive or superior performance in most settings, and that process-level credit assignment can over-correct agent behavior. Third, we analyze training data diversity, off-policy data utilization, and search budget scaling, distilling practical guidelines for training effective search agents. Our code is available at https://github.com/YiboZhao624/SearchAgentReview.
Yibo Zhao, Zichen Ding, Jiayi Wu +2
School of Data Science and Engineering, East China Normal University · Shanghai AI Laboratory