Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents

Yizhou Liu; Qi Sun; Yulin Chen; Siyue Zhang; Chen Zhao

cs.AIApr 6, 2026

Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents

Authors: Yizhou Liu, Qi Sun, Yulin Chen, Siyue Zhang, Chen Zhao

Organizations: University of Illinois Urbana-Champaign · New York University · Nanyang Technological University

Abstract

Agents equipped with search tools have emerged as effective solutions for knowledge-intensive tasks. While Large Language Models (LLMs) exhibit strong reasoning capabilities, their high computational cost limits practical deployment for search agents. Consequently, recent work has focused on distilling agentic behaviors from LLMs into Small Language Models (SLMs). Through comprehensive evaluation on complex multi-hop reasoning tasks, we find that despite possessing less parametric knowledge, SLMs invoke search tools less frequently and are more prone to hallucinations. To address this issue, we propose \policy, a lightweight fine-tuning approach that explicitly trains SLMs to reliably retrieve and generate answers grounded in retrieved evidence. Compared to agent distillation from LLMs, our approach improves performance by 17.3 scores on Bamboogle and 15.3 scores on HotpotQA, achieving LLM-level results across benchmarks. Our further analysis reveals that adaptive search strategies in SLMs often degrade performance, highlighting the necessity of consistent search behavior for reliable reasoning.

Figures & tables

Figure 1: Left: With Always-Search Policy, the distilled SLM significantly narrows the performance gap with the teacher model. Right: SLMs suffer from adaptive search and ASP is the most effective policy.

Figure 2: Scaling of agentic search performance. Small models trail behind in both performance and retrieval capability on HotpotQA.

Models	ASP	HotpotQA	2Wiki	Bamboogle	MuSiQue	BrowseComp	Frames	LongSeAL
Vanilla-Qwen3-0.6B	✗	19.4	26.0	34.4	8.7	4.6	8.9	7.4
Vanilla-Qwen3-1.7B	✗	42.3	39.8	50.6	22.9	3.2	15.2	9.0
Vanilla-Qwen3-4B	✗	53.4	54.7	69.1	28.7	7.0	28.4	8.2
Vanilla-Qwen3-8B	✗	58.2	58.1	71.5	32.0	16.0	30.2	10.8
Vanilla-Qwen3-32B	✗	60.3	69.9	73.1	32.5	21.0	36.6	13.5
Distilled-Qwen3-1.7B	✗	47.9	44.1	53.2	25.4	10.1	16.7	6.0

Table 1: String-F1 scores across agentic-search benchmarks. Small models distilled with Always-Search Policy have comparable performance with their larger variants. Mixed means start with SFT and then OPD.

Model	P=1% $\downarrow$	P=5% $\downarrow$	P=10% $\downarrow$	P=20% $\downarrow$
Vanilla-Qwen3-32B	$<$ 0.1	0.3	0.6	5.2
SFT-Qwen3-1.7B	1.9	4.8	15.0	22.0
OPD-Qwen3-1.7B	0.8	9.0	14.5	18.0

Table 2: After applying adaptive search with different top-

P

setting on dataset HotpotQA, Vanilla-Qwen3-32B and SFT-Qwen3-1.7B’s performance drop

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture Details
Configuration	Value / Setting
Probe Type	Multi-Layer Perceptron (MLP)
Number of Layers	3
Hidden Dimensions	{512, 256, 128}
Activation Function	ReLU
Input Features	$(h_{L-3},h_{L-2},h_{L-1},h_{L})$

Appendix

Table 3: Hyperparameters and architectural details for the Confidence Probe. The probe utilizes features from the final stages of the backbone model to predict generation confidence.

Model Name	ECE Score	Accuracy
Vanilla-Qwen3-32B	0.041	93.7%
Vanilla-Qwen3-1.7B	0.052	89.6%
SFT-Qwen3-1.7B	0.034	95.1%
OPD-Qwen3-1.7B	0.045	97.3%
Average	$\mathbf{0.043\pm 0.009}$	$\mathbf{93.9\%\pm 3.4\%}$

Appendix

Table 4: Calibration Performance. The Estimated Calibration Error (ECE) for the confidence probes across different LMs. Lower is better. The Accuracy on the validation data across different LMs. Higher the better

Figure 3: Confidence probing results. (a) illustrates the full sample distribution on a log scale; (b) zooms into high-confidence bins (0.5–1.0).

Model	Tool Calls	Avg. Latency
Vanilla-Qwen3-1.7B	1.72	$\sim$ 1.8s
Distilled-Qwen3-1.7B	1.89	$\sim$ 2.7s
SFT-Qwen3-1.7B	2.47	$\sim$ 3.1s
Vanilla-Qwen3-8B	2.84	$\sim$ 5.6s
Vanilla-Qwen3-32B	3.02	$\sim$ 10.3s

Appendix

Table 5: End-to-end inference latency on HotpotQA. Setting: 4

\times

H20, vLLM, FAISS-GPU. ASP-trained SLMs achieve comparable performance to Qwen3-32B at

\sim

\times

lower latency.

Model	MMLU	GSM8K	GPQA
Qwen3-32B	83.61	93.40	49.49
Qwen3-8B	76.89	89.84	44.44
Qwen3-4B	72.99	87.79	36.87
Qwen3-1.7B	62.63	75.44	28.28
Qwen3-0.6B	52.81	59.59	26.77
SFT-Qwen3-1.7B	73.28	85.34	41.26

Appendix

Table 6: Reasoning ability of different models on general reasoning tasks.Data collected from Qwen3 Technical Report

Explore similar work

To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning
Jul 12, 2026Fengji Zhang, Tianyu Fan, Yuxiang Zheng +4AI Agent Reliability LLM Hallucination Mitigation
City University of Hong Kong · Alibaba Group
Plan Before Search: Search Agents Need Plan
May 27, 2026Zhipeng Qian, Zihan Liang, Yufei Ma +7Multi-Hop QA Tool-Augmented Language Model Agents
1Kuaishou Technology · 2Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · 3VCIP, CS, Nankai University
Adaptive Latent Agentic Reasoning
Jun 1, 2026Dongwon Jung, Peng Shi, Yi Zhang +2Agentic Reasoning LLM Agents
University of California, Davis · University of Waterloo · Greenshoe, Inc.

Jul 12, 2026J/K move · Enter open · S save

To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning

Fengji Zhang, Tianyu Fan, Yuxiang Zheng, Xinyao Niu+3

City University of Hong Kong · Alibaba Group

AI Agent Reliability LLM Hallucination Mitigation Question Answering Agentic RAG RL for LLM Agents

Open paper AskPDF arXiv

Recent advances in equipping Large Language Models (LLMs) with search tools and outcome-reward reinforcement learning (RL) have achieved new state-of-the-art results on open-domain QA tasks. However, we argue that current training paradigms harbor a critical vulnerability: they predominantly reward correct answers but fail to penalize fabricated ones when retrieval fails, thereby implicitly exacerbating hallucinations. To address this, we propose Abstention-Aware Reinforcement Learning (AWA-RL), which dynamically shapes the abstention reward utilizing the model's query-specific prior capabilities and continuous on-policy training observations. We also introduce a novel metric, RA-F1, to measure the capability-reliability trade-off. Compared to non-abstaining baselines, AWA-RL boosts absolute precision by up to 10.3% and overall RA-F1 by 2.9%, with only marginal sacrifice in raw accuracy. These results confirm that AWA-RL successfully yields highly capable and reliable search agents. The code, data, and model weights are publicly available at https://github.com/zfj1998/AWA-RL.

Abstract

Explore similar work

To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning

Plan Before Search: Search Agents Need Plan

Adaptive Latent Agentic Reasoning