LexiHorizon: Stabilizing Reinforcement Learning for Long-Horizon Deep Search

Zhiqing Nong; Liang Wen; Chao-Hsuan Liu

cs.AIOct 4, 2026

LexiHorizon: Stabilizing Reinforcement Learning for Long-Horizon Deep Search

Authors: Zhiqing Nong, Liang Wen, Chao-Hsuan Liu

Abstract

Deep search agents tackle complex knowledge tasks through iterative retrieval, multi-hop reasoning, and evidence synthesis across multiple sources. Existing approaches typically assume relatively stable retrieval systems and operate over short-horizon tool interaction. However, when retrieval is sensitive to query formulation, even a semantically appropriate query may fail to surface critical evidence because of mismatched entity names, aliases, or keyword combinations. Recovering from such failures requires repeated query reformulation and longer interaction trajectories. This setting poses a distinct training challenge, as the policy must sustain long-horizon query exploration while managing an expanding volume of retrieved content. We propose LexiHorizon, a framework for training search agents over long horizons that expands the trajectory context budget, manages accumulated retrieval content using a window over recent tool observations while preserving the reasoning history, and introduces an outcome-gated search-effort reward that provides a bounded bonus for tool invocations to trajectories with nonzero answer reward. Experiments on XBench, WebWalkerQA, and BrowseComp-ZH show that the resulting 9B model consistently outperforms both its base model and MiroThinker-1.7-mini, with maximum absolute gains of 8.7 and 23.8 percentage points, respectively. These results suggest that combining an extended context budget with reasoning-preserving context management benefits long-horizon deep search agents.

Figures & tables

Model	XBench	WebWalkerQA	BrowseComp-ZH
Qwen3.5-9B	0.511	0.320	0.364
Qwen3.5-27B	0.550	0.394	0.310
MiroThinker-1.7-mini	0.360	0.313	0.320
LexiHorizon -9B	0.598	0.359	0.385

Table 1: Performance comparison on deep search benchmarks. We report mean accuracy over

n=8

independently sampled trajectories per question, with each trajectory evaluated separately against the reference answer. All models use the same decoding configuration, interaction budget, tool environment, and hardware allocation. The best and second-best results are highlighted in bold and underlined, respectively.

	Reliability		Efficiency
Model	Tool Fail. $\downarrow$	Valid Ans. $\uparrow$	Policy Len. $\downarrow$	Tok./s $\uparrow$
Qwen3.5-9B	12.80	87.60	9,947	106
Qwen3.5-27B	4.00	98.00	7,259	724
MiroThinker-1.7-mini	16.79	59.85	21,969	54
LexiHorizon -9B	2.00	99.88	6,124	1,069

Table 2: Reliability and end-to-end inference efficiency under identical hardware, rollout concurrency, decoding configuration, and tool environment. Tool Fail. and Valid Ans. denote the tool-call failure rate and valid-answer rate, respectively, both reported as percentages. Policy Len. is the average number of policy-generated tokens per trajectory and excludes tokens in tool observations. Tok./s is the generation throughput in policy-generated tokens per second.

Variant	Context	Window	XBench	WebWalker	BrowseComp-ZH	Valid
LexiHorizon -9B	128K	Yes	0.598	0.359	0.385	0.9988
w/o Extended Horizon	64K	Yes	0.228	0.156	0.128	0.6697
w/o Context Window	128K	No	0.496	0.300	0.316	0.9600

Table 3: Component ablation results. All other training and evaluation settings are unchanged.

Panel of Figure 1 — Figure 1: Search behavior under 64K and 128K context budgets.

Panel of Figure 2 — Figure 2: Optimization dynamics under 64K and 128K context budgets.

Tool Calls	Found	Retained	Missed	Retention
6–10	3	3	0	100.00%
11–20	5	5	0	100.00%
21–40	15	15	0	100.00%
$>40$	32	21	11	65.62%
Overall	55	44	11	80.00%

Table 4: Retention of answer-relevant evidence contained in removed tool observations. Candidate evidence is identified through keyword and alias matching and verified by an LLM judge.

Configuration	Avg. Calls	Calls P95	Avg. Ctx. Len.	Ctx. Len. P95
w/o Context Window	29.5	67	51,255.0	120,818.9
w/ Context Window	54.9	97	19,303.0	28,324.5

Table 5: Effect of the context window on test-time trajectory characteristics. Active context length denotes the number of tokens in the final active context, including both policy-generated tokens and retained tool observations.

Panel of Figure 3 — Figure 3: Training dynamics with and without the context window.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Prompt used by the answer correctness judge.

Configuration	XBench	WebWalkerQA	BrowseComp-ZH
w/o Reset	0.598	0.359	0.385
w/ Reset	0.575	0.359	0.404
Difference (pp)	$-2.3$	$0.0$	$+1.9$

Appendix

Table 6: Effect of trajectory reset on benchmark accuracy. We report the mean over eight independently sampled trajectories per question.

Panel of Figure 5 — Figure 5: Test-time response length distributions of policies trained without and with trajectory reset. Reset is disabled during evaluation for both policies. Both panels use identical axes and histogram bins.

Panel of Figure 6 — Figure 6: Active response length quantiles and rollout generation time per token during training.

Explore similar work

SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task
Jul 25, 2026Lang Mei, Xiaohan Yu, Chong Chen +27Synthetic Task Synthetic Data
Huawei Cloud Post-Training Team
Harness-Search: Guiding Long-Horizon Search through Multi-Agent Coordination
Oct 4, 2026Shanyong Wang, Zhenwen Ji, Lei Jin +7Search Agents Long-Horizon Agents
Xiaohongshu Inc. · the Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University · University of Illinois at Urbana-Champaign
Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search
Oct 21, 2025Howard Yen, Yoonsang Lee, Ashwin Paranjape +5Long-Horizon Agents Deep Research
Princeton Language and Intelligence, Princeton University · Work done as an intern at Samaya AI. · Samaya AI

Jul 25, 2026, cs.IRJ/K move · Enter open · S save

SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task

Lang Mei, Xiaohan Yu, Chong Chen, Liyan Liu+26

Huawei Cloud Post-Training Team

Synthetic Task Synthetic Data

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have enabled search agents to autonomously tackle complex tasks across extended search and reasoning horizons. However, training effective search agents remains challenging due to the lack of scalable and long-horizon tasks, and the difficulty of evaluating and correcting intermediate reasoning and tool-use behaviors. We introduce SearchArt, a scalable framework for training long-horizon search agents through verification-driven task synthesis and a multi-stage post-training pipeline. SearchArt constructs large-scale datasets for complex search-, research- and user-oriented tasks by synthesizing diverse information-seeking QA pairs and corresponding search trajectories from web documents and automatically generated evidence graphs. To ensure the reliability of the synthesized data, we design a verification pipeline that jointly evaluates QA consistency, trajectory quality, and the relevance of retrieved evidence. The verified trajectories are subsequently used in a multi-stage training process comprising supervised fine-tuning and reinforcement learning-based policy optimization. Search agents trained with SearchArt exhibit adaptive search planning, iterative evidence aggregation, and complex reasoning over extended interaction horizons. Experimental results demonstrate that, with only (Qwen3.5-) 27B parameters, SearchArt scores 74.39 on BrowseComp-ZH, 70.06 on BrowseComp, and 52.55 on Deepresearch-bench, matching or surpassing frontier closed-source agents on both deepsearch and deepresearch benchmarks.

Abstract

Explore similar work

SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task

Harness-Search: Guiding Long-Horizon Search through Multi-Agent Coordination

Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search