Integrating web search tools has significantly extended the capability of LLMs to address open-world, real-time, and long-tail problems. However, evaluating these Search Agents presents formidable challenges. First, constructing high-quality deep search benchmarks is prohibitively expensive, while unverified synthetic data often suffers from unreliable sources. Second, static benchmarks face dynamic obsolescence: as internet information evolves, complex queries requiring deep research often degrade into simple retrieval tasks due to increased popularity, and ground truths become outdated due to temporal shifts. Third, attribution ambiguity confounds evaluation, as an agent's performance is often dominated by its parametric memory rather than its actual search and reasoning capabilities. Finally, reliance on specific commercial search engines introduces variability that hampers reproducibility. To address these issues, we propose a novel framework, Mind-ParaWorld, for evaluating Search Agents in a Parallel World. Specifically, MPW samples real-world entity names to synthesize future scenarios and questions situated beyond the model's knowledge cutoff. A ParaWorld Law Model then constructs a set of indivisible Atomic Facts and a unique ground-truth for each question. During evaluation, instead of retrieving real-world results, the agent interacts with a ParaWorld Engine Model that dynamically generates SERPs grounded in these inviolable Atomic Facts. We release MPW-Bench, an interactive benchmark spanning 19 domains with 1,608 instances. Experiments across three evaluation settings show that, while search agents are strong at evidence synthesis given complete information, their performance is limited not only by evidence collection and coverage in unfamiliar search environments, but also by unreliable evidence sufficiency judgment and when-to-stop decisions-bottlenecks.
Figures & tables
Figure 1: Overview of Mind-ParaWorld Framework.
Tier
∣F∣ range
#Instances
Percentage%
Easy
1–5
568
36.52
Mid
6–10
619
38.50
Hard
≥ 11
421
26.18
Table 1: Complexity tiers in MPW-Bench, defined post-hoc by the number of atomic facts ∣F∣ in each instance.
Category
#
Category
#
Category
#
Category
#
Category
#
LaLiga Player
100
Serie Player
100
Bundesliga Player
92
Ligue1 Player
88
Premier Player
88
Football Team
96
F1 Player
84
NBA Player
80
Singer
100
Actor
80
Game Company
88
LOL Player
80
Camera Brand
88
Computer Brand
32
Phone Brand
56
Fashion Brand
96
Beauty Brand
92
Car Brand
92
Tech Company
76
Total
1608
Table 2: Distribution of categories in the MPW-Bench.
Model
Overall
Easy
Medium
Hard
MindWatcher 32B [ 3 ]
85.01
88.83
85.23
79.57
Qwen3-30B-A3B [ 28 ]
85.14
87.23
87.16
79.33
Qwen3-32B [ 28 ]
91.04
94.86
91.17
85.75
MiniMax-m2
67.23
66.67
67.58
67.46
MiniMax-m2.1
82.46
84.93
79.94
82.90
Tongyi DeepResearch [ 20 ]
83.40
83.33
83.15
83.85
Table 3: Performance of all models under Setting A (Pass@1 %). E : Easy, M : Medium, H : Hard, O : Overall.
Model
Diff.
Fewshot Prompt
Guidance Prompt
Pass@1
FCR
HitRate
ToolCalls
Pass@1
FCR
HitRate
ToolCalls
E
59.75
56.20
58.00
6.95
59.93
56.40
59.30
6.18
M
47.67
38.80
48.00
10.22
44.14
36.50
45.90
10.43
H
30.88
25.30
40.00
14.15
23.04
21.80
35.40
14.53
MindWatcher 32B
O
47.51
41.40
49.40
10.10
44.15
39.60
47.80
10.01
E
34.04
29.50
41.10
1.98
24.12
20.20
27.00
1.89
Table 4: Performance comparison of all models under Setting B (Fewshot and Guidance prompts) across difficulty levels. E : Easy, M : Medium, H : Hard, O : Overall. Pass@1, FCR, and Hit Rate are reported in percentages (%).
Model
Difficulty
Pass@1
FCR
HitRate
ToolCalls
E
56.69
51.30
55.30
5.78
M
36.03
31.10
40.30
9.19
H
17.81
18.20
30.30
12.00
MindWatcher 32B
O
38.56
34.90
43.00
8.72
E
24.65
19.10
28.50
1.57
M
12.60
6.10
19.00
1.61
Table 5: Performance of locally-deployed models under Setting C (End-to-End ReAct).
Model
Diff.
Pass@1
FCR
HitRate
ToolCalls
E
44.19
30.80
16.60
6.69
M
30.69
16.00
10.30
9.57
H
22.09
8.40
8.30
9.91
GPT-5
O
33.21
19.20
12.00
8.64
E
13.73
11.20
43.30
0.75
M
5.01
4.20
30.70
0.86
Table 6: Performance of commercial API models under Setting C (End-to-End ReAct).
Figure 2: Process-level analysis under Setting C on the relationship between tool-call budget and evidence coverage for three representative search agents (GPT-5, MindWatcher, and MiniMax-m2.1). Upper: mean marginal newly covered atomic facts at the k-th tool call (left axis) and cohort size n(k) (right axis), computed over trajectories with ToolCalls≥kcohort ; regions with n(k)<50 are shown with reduced opacity. Lower: truncated cumulative curves of factual coverage FCR(k) (solid, left axis) and cumulative hit precision HitPrec(k) (dashed, right axis) computed over trajectories with ToolCalls≥ktrunc , revealing diminishing marginal gains and saturation under longer interactions.
Figure 3: Relationship between FCR and Pass@1. The curve shows sample-level correlation from 4,824 samples across Setting B and C. Scatter points represent model-level performance: Setting B Guidance, Setting B Few-shot, and Setting C. The dashed line indicates Setting A performance (FCR=1.0). Shadow band width reflects sample density.
Are LLM-based search agents genuinely searching, or using the web to verify what they already know? We study this question on BrowseComp with three diagnostics. Our analysis reveals Intrinsic Knowledge Dependence (IKD): even with tool access, agents often rely on intrinsic knowledge -- information encoded in the model before retrieval -- rather than on external evidence. Agents answer up to 44.5% of BrowseComp questions without tools, generate more than half of their search queries from internally produced hypotheses rather than retrieved leads, and perform worse than closed-book baselines when answer-supporting evidence is removed. These results suggest that static search benchmarks can reward memory-backed verification rather than evidence-driven discovery, conflating what agents already know with what they can find. We then introduce LiveBrowseComp, a deep-search benchmark designed to evaluate agents beyond intrinsic coverage. It contains 335 human-authored questions whose answers depend on facts published within the 90 days preceding benchmark construction, drawn from six updated sources and filtered to exclude globally salient events. On LiveBrowseComp, all evaluated agents fall below 2% closed-book accuracy, search-augmented scores drop by 25-40 points relative to BrowseComp, and prior model rankings no longer reliably predict performance. LiveBrowseComp is available at https://huggingface.co/datasets/Forival/LiveBrowseComp.
Recent advances in large language models (LLMs) have enabled search agents to autonomously tackle complex tasks across extended search and reasoning horizons. However, training effective search agents remains challenging due to the lack of scalable and long-horizon tasks, and the difficulty of evaluating and correcting intermediate reasoning and tool-use behaviors. We introduce SearchArt, a scalable framework for training long-horizon search agents through verification-driven task synthesis and a multi-stage post-training pipeline. SearchArt constructs large-scale datasets for complex search-, research- and user-oriented tasks by synthesizing diverse information-seeking QA pairs and corresponding search trajectories from web documents and automatically generated evidence graphs. To ensure the reliability of the synthesized data, we design a verification pipeline that jointly evaluates QA consistency, trajectory quality, and the relevance of retrieved evidence. The verified trajectories are subsequently used in a multi-stage training process comprising supervised fine-tuning and reinforcement learning-based policy optimization. Search agents trained with SearchArt exhibit adaptive search planning, iterative evidence aggregation, and complex reasoning over extended interaction horizons. Experimental results demonstrate that, with only (Qwen3.5-) 27B parameters, SearchArt scores 74.39 on BrowseComp-ZH, 70.06 on BrowseComp, and 52.55 on Deepresearch-bench, matching or surpassing frontier closed-source agents on both deepsearch and deepresearch benchmarks.
Search Agents -- large language models augmented with search tools -- have intensified the need for future-proof evaluation benchmarks. Existing benchmarks such as BrowseComp rely on static knowledge, making them vulnerable to test-set contamination and parametric memorization. Consequently, models can achieve high scores through fact recall rather than genuine retrieval, obscuring true browsing competence via reasoning shortcuts. In this paper, we introduce EvoBrowseComp, an evolving benchmark of 400 English and 400 Chinese contamination-free complex questions synthesized via live-web traversal. To collect these questions, we design a three-agent collaborative framework: (1) a QA synthesis agent that retrieves fresh knowledge from the live web to synthesize QA pairs; (2) an information filtering agent that filters retrieved knowledge in terms of credibility and popularity to block parametric shortcuts; and (3) a high-level guidance agent that formalizes questions into reasoning graphs to reduce logical redundancy and shortcuts in synthesized QA pairs. Because the framework supports fully automated synthesis, EvoBrowseComp can be regularly updated to prevent data contamination and maintain temporal freshness. Extensive experiments confirm its great difficulty, requiring broad horizontal search. It establishes a scalable paradigm for auto-updatable, high-difficulty benchmarking that keeps pace with both evolving world knowledge and advancing agent capabilities.
Yunhan Wang, Jiaan Wang, Lianzhe Huang +2
Northeastern University, China · Weixin AI, Tencent Inc, China