LongSearchEval: Evaluating Long-Horizon Web Search Capabilities of Large Language Models via a Trajectory-Aware Agent Framework with Failure-Node Localization
Authors: Jialiang Liu, Shusheng Li, Wenjun Tan
Organizations: College of Information Science and Engineering, Northeastern University, China · Key Laboratory of Intelligent Computing of Medical Images, Ministry of Education, Northeastern University, China. · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Northeastern University, China
Long-horizon web search requires LLM-based agents to integrate information across web pages through extended interactions under task-specific constraints. Evaluating this capability is important not only for measuring final answer quality, but also for assessing whether an agent acquires, verifies, and uses evidence reliably over extended interactions. Existing evaluations, however, mainly rely on simulated trajectories or on post-hoc trajectories reconstructed from final answers, rather than on retained real interaction traces, which limits the fidelity of behavioral assessment and weakens support for fine-grained error localization. To address this gap, we introduce LONGSEARCHEVAL, a trajectory-aware agent evaluation framework that combines a unified agent execution protocol with source constrained outcome verification and retrospective failure-node localization. Stage 1 verifies answer groups against agent-cited webpages without human-authored gold answers; Stage 2 audits normalized rounds and links flagged failure nodes to incorrect answer groups. We evaluate 16 models from eight families on 100 tasks, yielding 1,600 retained runs and 6,545 automatically flagged failure nodes. Trajectory collection and evaluation required 1,037 and 318 hours of cumulative runtime, respectively. Human validation yields 94.50% agreement on supplied answer-group labels and 98.11% feature-correspondence precision for audited round-group links. Together, outcome scores and linked trajectory evidence support controlled model comparison and retrospective inspection of answer-related content.
Figures & tables
Figure 1: Source-grounded answer evaluation, process benchmarks, and retrospective trajectory inspection. Expert labels in (b) are benchmark references, not required localization inputs; (c) returns content related to Stage 1-incorrect answer groups under a shared unified agent execution protocol.
Figure 2: Overview of LongSearchEval . A unified agent interface records each LLM’s observable trajectory and final response. Stage 1 verifies answers using agent-provided URLs. Conditioned on the response and Stage 1 judgments, Stage 2 links related content in normalized rounds to incorrect answer groups without web access or hidden chain-of-thought.
Outcome Quality
Search Behavior
Evidence Use
Execution Robustness
Model
AS ↑
AP (%) ↑
Upper ↑
UGR (%) ↓
Avg. Rounds
Avg. Searches
Avg. Domains
Read kChars
Final URLs
Valid (%) ↑
Prem. (%) ↓
GPT-6.1-Sol
68.80
68.34
75.35
5.32
17.74
9.49
6.59
26.79
10.44
100
0
GPT-6-Luna-Pro
65.88
67.39
73.08
6.12
18.41
14.57
5.38
16.58
8.11
100
0
GPT-5.4
54.23
55.99
58.85
3.56
19.67
11.19
5.37
10.24
12.90
100
0
GPT-5-Mini
47.17
47.17
51.13
4.90
25.07
16.35
5.08
4.78
11.24
92
5
Gemini-3.1-Pro
38.51
38.76
50.15
10.99
24.57
16.64
5.72
24.64
7.99
100
0
Table 1: Outcome quality, search behavior, evidence use, and execution robustness evaluated with LongSearchEval under the controlled agent execution setting. AS, AP, and UGR denote Answer Score, Answer Precision, and Unverifiable Group Rate. AP and UGR are percentages, and Read kChars is the mean character count divided by 1,000. Valid and Prem. use the saved final-answer-validity and premature-answer indicators. Models are grouped by family, with thin horizontal rules separating families to facilitate within- and cross-family comparisons.
Figure 3: Diagnostic type composition of 6,503 automatically flagged and typed candidate nodes. Each model is normalized by its own typed-node count; 42 unassigned nodes remain in localization totals but not in type denominators.
Figure 4: Answer Score versus Answer Precision for 16 LLMs. Colors and lines denote model families.
Process diagnostic
Result
All automatically flagged candidate nodes
6,545
Linked / unlinked candidate nodes
3,664 / 2,881
Typed / unassigned candidate nodes
6,503 / 42
Unlinked incorrect final-answer groups
1,145
Runs with at least one linked candidate node
850
Earliest association by round 5 / 20
46.5% / 88.7%
Table 2: Automatic candidate-node diagnostics under Stage 1-incorrect-group filtering. Earliest-association percentages use the 850 runs with retained links.
Figure 5: Automatic–human outcome-score alignment in the post-hoc validation study: (a) Answer Score and (b) Answer Precision. Both panels use the same 200 response pairs from 100 tasks. Dashed lines indicate exact agreement ( y=x ); marker area is proportional to the number of responses sharing the same score pair.
Score
Pearson r
Spearman ρ
Kendall τ
Answer Score
0.9054
0.9035
0.8609
Answer Precision
0.9132
0.9103
0.8712
Table 3: Score correlations following the 946-group human audit. Each coefficient uses 200 response pairs from 100 tasks.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Unit
Meaning and aggregation
Task / retained run
100 task IDs; 1,600 model–task cells. Retries and resumes do not add samples.
Answer group
7,685 supplied groups. AS, AP, and Upper are run-level scores macro-averaged over 100 tasks per model; UGR pools groups.
Normalized round
42,658 ordered units; distinct from raw controller iterations and individual tool calls.
Flagged node
6,545 consensus-error rounds, including 42 without an assigned type. Counts summarize the automatic diagnostic inventory.
Retained edge
A distinct run–round–incorrect-group triple. 5,798 edges connect 3,664 nodes to 2,229 groups. Each unit has its own denominator.
Earliest association
Minimum linked-round index, conditional on a retained link; defined for 850 runs. The other 750 receive no index.
Appendix
Table A1: Evaluation units and their separate aggregation denominators.
Family
Evaluated models
Trajectories
GPT
GPT-6.1-Sol; GPT-6-Luna-Pro; GPT-5.4; GPT-5-Mini
400
Gemini
Gemini-3.1-Pro; Gemini-2.5-Flash
200
Kimi
Kimi-K2.6; Kimi-K2.5
200
Qwen
Qwen3.7-Plus; Qwen3.6-Flash
200
DeepSeek
DeepSeek-V3.2; DeepSeek-V4-Flash
200
GLM
GLM-5; GLM-4.7-Flash
200
Appendix
Table A2: Main evaluation inventory across 100 tasks. Each of the 16 models contributes one retained trajectory per task, giving 1,600 trajectories across eight families. Model names follow the main paper.
Figure A1: Recorded rounds R11–R14: candidate discovery, metadata-only opening, cached-text exposure, and literal localization. R13–R14 reuse the page identifier from R12; ellipses abbreviate stored fields.
Control
Effective value
Maximum ReAct budget
120 iterations
Early-stopping method
generate
web_search results
Default 10; hard cap 10
read_page characters
Default 3,000; hard cap 5,000
LLM request timeout
600 seconds per attempt
web_search timeout
10-second connect; 20-second read
Appendix
Table A3: Recorded collection controls; historical per-run coverage is specified in the text. Per-request timeouts and the subprocess watchdog are distinct from search-stagnation controls.
Field
Modal coverage
Tied tasks
Median unique-mode count
Distinct values/task
Required count N
1,556/1,600 (97.25%)
1
16/16 (99 tasks)
1–5
Hard-constraint count
755/1,600 (47.19%)
11
7/16 (89 tasks)
1–9
Normalized grouping-rule text
927/1,600 (57.94%)
2
10/16 (98 tasks)
1–16
Appendix
Table A4: Field-level agreement among the 16 stored rubrics for each task. Grouping-rule agreement measures normalized wording; the other rows measure agreement in the required count and number of constraints.
Figure A2: Field-level rubric stability across 16 saved runs per task (100 tasks; 1,600 rubrics). Panels (a) and (c) show the frequency of the unique modal required count and hard-constraint count, excluding 1 and 11 tasks with tied modes, respectively. Panels (b) and (d) retain all 100 tasks and show the number of distinct values per task. The panels measure structural count agreement. Each run retains its original rubric for scoring.
Figure A3: Stage 2 round accounting across 1,600 runs. (a) The 16,279 dual-judged rounds yield 14,734 evaluable records and 1,545 post-judging skips; the latter combine with 26,379 pre-skipped rounds to give 27,924 final skips. The stored error label denotes an automatic candidate flag. (b) Persisted-record composition uses all 42,658 rounds as its denominator: 61.84% pre-skipped summaries, 34.54% detailed evaluable records, and 3.62% post-judge summaries.
Error type
Diagnostic definition and nearest boundary
Wrong entity
The target is identified, substituted, or merged with a different entity, alias, version, institution, product, or location. If the entity is the intended one but fails an eligibility condition, use constraint violation.
Constraint violation
A candidate fails an explicit hard condition such as a feature, threshold, status, or eligibility rule. Date, geographic, and population boundaries belong to temporal or scope error.
Missing required field
An identifiable answer unit lacks a required attribute or source. An existing value assigned to the wrong field is a mapping error; incorrectly formed units are a grouping error.
Insufficient evidence
The available observation is too incomplete to establish a claim, for example a title, metadata, or partial passage lacking the required detail.
Source not supporting
Inspected content can be assessed but is irrelevant, omits the claimed fact, supports another value, or contradicts the interpretation. The issue is support for the claim, rather than source type.
Source quality error
The source is invalid, inaccessible, unsuitable, or explicitly disallowed by an official/primary-source requirement.
Appendix
Table A5: Definitions and category boundaries of all ten diagnostic categories. Every category is defined here; the four selected cases illustrate a subset of the vocabulary. Stored labels remain automatic diagnostic assignments, including when the associated webpage observation itself is correct.
Figure A4: Illustrative source–answer comparisons: source not supporting, constraint violation, source quality error, and wrong entity. The original four-panel layout pairs returned observations with final-answer groups; round units and type descriptions reproduce evaluator interpretations. The accompanying case descriptions identify each model, task, round, and group.
Figure A5: Diagnostic-type composition and assignment coverage. (a) Counts over 6,503 typed candidate nodes; all ten categories are retained. (b) Among 6,545 automatically flagged nodes, 6,503 (99.36%) have a normalized type and 42 (0.64%) remain unassigned. The latter stay in localization and linking totals under a separate unassigned status. Consensus error nodes denote the automatic candidate flags defined in Appendix F .
Figure A6: Extended trajectory and execution diagnostics for 1,600 retained runs. (a) Linked and unlinked candidate flags, including unassigned types. (b) Earliest returned association among 850 trajectories with links. (c) Separately recorded, overlapping execution markers. The marker totals are 25 invalid-final flags, 20 final-only syntheses, 11 premature-answer flags, and 99 resume-marked trajectories.
Model
All rounds
Evaluable rounds
Flagged nodes
Runs with flag
Linked nodes
Runs with link
GPT-6.1-Sol
1,774
762
155
43
56
20
GPT-6-Luna-Pro
1,841
552
130
49
43
21
DeepSeek-V3.2
2,767
1,221
506
86
270
54
GPT-5.4
1,967
838
284
70
133
53
DeepSeek-V4-Flash
3,022
853
304
60
194
47
Kimi-K2.6
3,467
1,348
548
72
270
50
Appendix
Table A6: Raw process counts over 100 trajectories per model. Evaluable rounds have an error or nonerror consensus verdict; final skipped rounds are excluded. Linked nodes and runs use the retained intersection with Stage 1 incorrect groups. Counts summarize observed activity and diagnostic coverage within each model’s retained trajectories.
Model
Search
Open
Read
Find
Extract
Tools
Rounds
GPT-6.1-Sol
9.49
13.41
7.75
4.30
0.85
35.80
17.74
GPT-6-Luna-Pro
14.57
9.29
5.38
4.15
0.39
33.78
18.41
DeepSeek-V3.2
13.75
15.64
10.84
12.98
0.65
53.86
27.67
GPT-5.4
11.19
10.40
4.86
8.68
0.76
35.89
19.67
DeepSeek-V4-Flash
21.99
14.21
12.09
5.27
0.35
53.91
30.22
Kimi-K2.6
21.98
17.42
15.64
8.59
0.32
63.95
34.67
Appendix
Table A7: Mean tool use and normalized length over 100 trajectories per model. The Tools column sums the five tool categories; normalized rounds are counted separately from tool calls.
Automatic → H1 (946 groups)
H1 → H2 (172 groups)
Row label
C
I
U
Total
Row label
C
I
U
Total
C
475
21
2
498
C
103
5
1
109
I
22
373
1
396
I
6
51
0
57
U
1
5
46
52
U
0
0
6
6
Total
498
399
49
946
Total
109
56
7
172
Appendix
Table A8: Supplied-group label matrices. Columns identify the second member of each comparison; C/I/U mean correct/incorrect/unverifiable.
Automatic–H1
H1–H2
Metric
AS
AP
Upper
AS
AP
Upper
Pearson r
0.9054 [0.8496, 0.9519]
0.9132 [0.8595, 0.9588]
0.8798 [0.8107, 0.9372]
0.8961 [0.7335, 0.9825]
0.9022 [0.7468, 0.9834]
0.9215 [0.8003, 0.9879]
Spearman ρ
0.9035 [0.8441, 0.9515]
0.9103 [0.8514, 0.9583]
0.8816 [0.8107, 0.9370]
0.8614 [0.6248, 0.9837]
0.8679 [0.6394, 0.9855]
0.9003 [0.7314, 0.9876]
Kendall τ
0.8609 [0.7931, 0.9204]
0.8712 [0.8054, 0.9299]
0.8326 [0.7557, 0.8992]
0.8087 [0.5839, 0.9578]
0.8136 [0.5899, 0.9623]
0.8561 [0.6881, 0.9675]
MAE
4.8792 [2.8749, 7.1625]
4.4542 [2.4832, 6.6833]
6.2625 [3.7458, 9.1167]
5.0625 [1.5625, 9.3958]
5.0625 [1.5625, 9.3958]
4.5625 [1.3333, 8.7297]
Bias
0.8375 [ −1.4584 , 3.2500]
0.4625 [ −1.7584 , 2.7833]
2.0542 [ −0.7502 , 5.1333]
0.8958 [ −3.3333 , 5.7500]
0.8958 [ −3.3333 , 5.7500]
0.3958 [ −3.5417 , 5.0625]
Appendix
Table A9: Response-score agreement on 200 automatic–H1 responses and the independent 40-response H1–H2 subset. Each cell gives the point estimate above its 95% confidence interval. AS is Answer Score, AP is Answer Precision, and Upper is Upper Score. AS/Upper errors are points on a 0–100 scale; AP errors are percentage points and AP correlations use 0–1 values. Bias is automatic minus H1 in the left block and H1 minus H2 in the right block. H2 labels do not replace H1 in the primary comparison.
Human \ Automatic
Error
Non-error
Skipped
Total
Error
0
0
0
0
Non-error
329
589
353
1,271
Skipped
20
9
1,342
1,371
Total
349
598
1,695
2,642
Appendix
Table A10: Complete process cross-tabulation on 100 trajectories from 50 tasks. The two sets of labels concern different annotation targets.
Phase
Input (M)
Output (M)
USD
Collection
2,782.84
60.50
1,812.30
Stage 1
157.17
118.48
676.66
Stage 2
102.07
44.84
454.03
Budgeted total
3,042.07
223.82
2,942.98
Appendix
Table A11: Token workload and upper-scenario API budget for the retained-run pipeline. M denotes one million tokens. Collection uses estimated workload; evaluation uses recorded usage. All API amounts are reference-price budgets. Totals are computed before rounding.
Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers. Diagnosing such failures is difficult, requiring the manual inspection of extremely long execution traces, which could be beyond human capacity. We therefore introduce SearchAuditBench, a benchmark that evaluates whether LLM auditors can localize, attribute, and repair these failures, thereby reducing the human burden. SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks, each expert-annotated with the critical error step, a search-specific root cause, and a reference repair with grading rubrics. We further propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures through evidence-grounded adjudication. Experimental results show that even the strongest baseline, when powered by a frontier model like GPT-5.5, attains only a 26.6% end-to-end pass rate. In contrast, our SearchAuditor consistently outperforms all baselines across different frontier models, achieving an end-to-end pass rate of 32.3%, and resuming failed runs with its repairs enables agents to better recover from errors.
Zhixiang Liang, Yifei Liu, Yidan Huang +5
University of Illinois Urbana-Champaign · 2Joy Future Academy, JD
Existing web agent benchmarks have largely converged on short, single-site tasks that frontier models are approaching saturation on. However, real world web use consists of long-horizon, multi-site workflows. Common web navigation tasks, such as comparing products across different domains, planning trips across multiple services, or summarizing information from multiple search queries, require sustained context and cross-site reasoning over potentially hours of browsing. To capture and evaluate such behaviors, we introduce Odysseys: a benchmark of 200 long-horizon web tasks derived from real world browsing sessions evaluated on the live Internet. We find that binary pass/fail evaluation is inadequate for long-horizon settings and introduce a rubric-based evaluation, annotating each Odysseys task with an average of 6.1 graded rubrics. We demonstrate that this yields higher agreement with humans and provides a more fine-grained signal than commonly used trajectory-level LLM-as-a-judge evaluation metrics. We tested several leading frontier models and find that the strongest models achieve a success rate of 44.5%, which leaves substantial room for future improvements. Beyond task success, we argue that efficiency is a first-class concern for long-horizon agents. We introduce a Trajectory Efficiency metric (rubric score per step) and find that even frontier agents achieve only 1.15%, marking an evident need for agents that can succeed efficiently and not simply eventually. Odysseys isolates the critical evaluation of long-horizon proficiency in open-web environments, providing a realistic benchmark to measure progress towards computer-use agents that can potentially productively operate for hours. We release our tasks, evaluation scripts, and other results at https://odysseys-website.pages.dev
Lawrence Keunho Jang, Jing Yu Koh, Daniel Fried +1
Recent advances in large language models (LLMs) have enabled search agents to autonomously tackle complex tasks across extended search and reasoning horizons. However, training effective search agents remains challenging due to the lack of scalable and long-horizon tasks, and the difficulty of evaluating and correcting intermediate reasoning and tool-use behaviors. We introduce SearchArt, a scalable framework for training long-horizon search agents through verification-driven task synthesis and a multi-stage post-training pipeline. SearchArt constructs large-scale datasets for complex search-, research- and user-oriented tasks by synthesizing diverse information-seeking QA pairs and corresponding search trajectories from web documents and automatically generated evidence graphs. To ensure the reliability of the synthesized data, we design a verification pipeline that jointly evaluates QA consistency, trajectory quality, and the relevance of retrieved evidence. The verified trajectories are subsequently used in a multi-stage training process comprising supervised fine-tuning and reinforcement learning-based policy optimization. Search agents trained with SearchArt exhibit adaptive search planning, iterative evidence aggregation, and complex reasoning over extended interaction horizons. Experimental results demonstrate that, with only (Qwen3.5-) 27B parameters, SearchArt scores 74.39 on BrowseComp-ZH, 70.06 on BrowseComp, and 52.55 on Deepresearch-bench, matching or surpassing frontier closed-source agents on both deepsearch and deepresearch benchmarks.