cs.AIOct 26, 2026
SaveLongSearchEval: Evaluating Long-Horizon Web Search Capabilities of Large Language Models via a Trajectory-Aware Agent Framework with Failure-Node Localization
Organizations: College of Information Science and Engineering, Northeastern University, China · Key Laboratory of Intelligent Computing of Medical Images, Ministry of Education, Northeastern University, China. · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Northeastern University, China
Abstract
Long-horizon web search requires LLM-based agents to integrate information across web pages through extended interactions under task-specific constraints. Evaluating this capability is important not only for measuring final answer quality, but also for assessing whether an agent acquires, verifies, and uses evidence reliably over extended interactions. Existing evaluations, however, mainly rely on simulated trajectories or on post-hoc trajectories reconstructed from final answers, rather than on retained real interaction traces, which limits the fidelity of behavioral assessment and weakens support for fine-grained error localization. To address this gap, we introduce LONGSEARCHEVAL, a trajectory-aware agent evaluation framework that combines a unified agent execution protocol with source constrained outcome verification and retrospective failure-node localization. Stage 1 verifies answer groups against agent-cited webpages without human-authored gold answers; Stage 2 audits normalized rounds and links flagged failure nodes to incorrect answer groups. We evaluate 16 models from eight families on 100 tasks, yielding 1,600 retained runs and 6,545 automatically flagged failure nodes. Trajectory collection and evaluation required 1,037 and 318 hours of cumulative runtime, respectively. Human validation yields 94.50% agreement on supplied answer-group labels and 98.11% feature-correspondence precision for audited round-group links. Together, outcome scores and linked trajectory evidence support controlled model comparison and retrospective inspection of answer-related content.
Figures & tables
Figure 1: Source-grounded answer evaluation, process benchmarks, and retrospective trajectory inspection. Expert labels in (b) are benchmark references, not required localization inputs; (c) returns content related to Stage 1-incorrect answer groups under a shared unified agent execution protocol.
Figure 2: Overview of LongSearchEval . A unified agent interface records each LLM’s observable trajectory and final response. Stage 1 verifies answers using agent-provided URLs. Conditioned on the response and Stage 1 judgments, Stage 2 links related content in normalized rounds to incorrect answer groups without web access or hidden chain-of-thought.
| Outcome Quality | Search Behavior | Evidence Use | Execution Robustness | ||||||||
| Model | AS | AP (%) | Upper | UGR (%) | Avg. Rounds | Avg. Searches | Avg. Domains | Read kChars | Final URLs | Valid (%) | Prem. (%) |
| GPT-6.1-Sol | 68.80 | 68.34 | 75.35 | 5.32 | 17.74 | 9.49 | 6.59 | 26.79 | 10.44 | 100 | 0 |
| GPT-6-Luna-Pro | 65.88 | 67.39 | 73.08 | 6.12 | 18.41 | 14.57 | 5.38 | 16.58 | 8.11 | 100 | 0 |
| GPT-5.4 | 54.23 | 55.99 | 58.85 | 3.56 | 19.67 | 11.19 | 5.37 | 10.24 | 12.90 | 100 | 0 |
| GPT-5-Mini | 47.17 | 47.17 | 51.13 | 4.90 | 25.07 | 16.35 | 5.08 | 4.78 | 11.24 | 92 | 5 |
| Gemini-3.1-Pro | 38.51 | 38.76 | 50.15 | 10.99 | 24.57 | 16.64 | 5.72 | 24.64 | 7.99 | 100 | 0 |
Table 1: Outcome quality, search behavior, evidence use, and execution robustness evaluated with LongSearchEval under the controlled agent execution setting. AS, AP, and UGR denote Answer Score, Answer Precision, and Unverifiable Group Rate. AP and UGR are percentages, and Read kChars is the mean character count divided by 1,000. Valid and Prem. use the saved final-answer-validity and premature-answer indicators. Models are grouped by family, with thin horizontal rules separating families to facilitate within- and cross-family comparisons.
Figure 3: Diagnostic type composition of 6,503 automatically flagged and typed candidate nodes. Each model is normalized by its own typed-node count; 42 unassigned nodes remain in localization totals but not in type denominators.
Figure 4: Answer Score versus Answer Precision for 16 LLMs. Colors and lines denote model families.
| Process diagnostic | Result |
| All automatically flagged candidate nodes | 6,545 |
| Linked / unlinked candidate nodes | 3,664 / 2,881 |
| Typed / unassigned candidate nodes | 6,503 / 42 |
| Unlinked incorrect final-answer groups | 1,145 |
| Runs with at least one linked candidate node | 850 |
| Earliest association by round 5 / 20 | 46.5% / 88.7% |
Table 2: Automatic candidate-node diagnostics under Stage 1-incorrect-group filtering. Earliest-association percentages use the 850 runs with retained links.
Figure 5: Automatic–human outcome-score alignment in the post-hoc validation study: (a) Answer Score and (b) Answer Precision. Both panels use the same 200 response pairs from 100 tasks. Dashed lines indicate exact agreement ( ); marker area is proportional to the number of responses sharing the same score pair.
| Score | Pearson | Spearman | Kendall |
| Answer Score | 0.9054 | 0.9035 | 0.8609 |
| Answer Precision | 0.9132 | 0.9103 | 0.8712 |
Table 3: Score correlations following the 946-group human audit. Each coefficient uses 200 response pairs from 100 tasks.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Unit | Meaning and aggregation |
| Task / retained run | 100 task IDs; 1,600 model–task cells. Retries and resumes do not add samples. |
| Answer group | 7,685 supplied groups. AS, AP, and Upper are run-level scores macro-averaged over 100 tasks per model; UGR pools groups. |
| Normalized round | 42,658 ordered units; distinct from raw controller iterations and individual tool calls. |
| Flagged node | 6,545 consensus-error rounds, including 42 without an assigned type. Counts summarize the automatic diagnostic inventory. |
| Retained edge | A distinct run–round–incorrect-group triple. 5,798 edges connect 3,664 nodes to 2,229 groups. Each unit has its own denominator. |
| Earliest association | Minimum linked-round index, conditional on a retained link; defined for 850 runs. The other 750 receive no index. |
Table A1: Evaluation units and their separate aggregation denominators.
| Family | Evaluated models | Trajectories |
| GPT | GPT-6.1-Sol; GPT-6-Luna-Pro; GPT-5.4; GPT-5-Mini | 400 |
| Gemini | Gemini-3.1-Pro; Gemini-2.5-Flash | 200 |
| Kimi | Kimi-K2.6; Kimi-K2.5 | 200 |
| Qwen | Qwen3.7-Plus; Qwen3.6-Flash | 200 |
| DeepSeek | DeepSeek-V3.2; DeepSeek-V4-Flash | 200 |
| GLM | GLM-5; GLM-4.7-Flash | 200 |
Table A2: Main evaluation inventory across 100 tasks. Each of the 16 models contributes one retained trajectory per task, giving 1,600 trajectories across eight families. Model names follow the main paper.
Figure A1: Recorded rounds R11–R14: candidate discovery, metadata-only opening, cached-text exposure, and literal localization. R13–R14 reuse the page identifier from R12; ellipses abbreviate stored fields.
| Control | Effective value |
| Maximum ReAct budget | 120 iterations |
| Early-stopping method | generate |
| web_search results | Default 10; hard cap 10 |
| read_page characters | Default 3,000; hard cap 5,000 |
| LLM request timeout | 600 seconds per attempt |
| web_search timeout | 10-second connect; 20-second read |
Table A3: Recorded collection controls; historical per-run coverage is specified in the text. Per-request timeouts and the subprocess watchdog are distinct from search-stagnation controls.
| Field | Modal coverage | Tied tasks | Median unique-mode count | Distinct values/task |
| Required count | 1,556/1,600 (97.25%) | 1 | 16/16 (99 tasks) | 1–5 |
| Hard-constraint count | 755/1,600 (47.19%) | 11 | 7/16 (89 tasks) | 1–9 |
| Normalized grouping-rule text | 927/1,600 (57.94%) | 2 | 10/16 (98 tasks) | 1–16 |
Table A4: Field-level agreement among the 16 stored rubrics for each task. Grouping-rule agreement measures normalized wording; the other rows measure agreement in the required count and number of constraints.
Figure A2: Field-level rubric stability across 16 saved runs per task (100 tasks; 1,600 rubrics). Panels (a) and (c) show the frequency of the unique modal required count and hard-constraint count, excluding 1 and 11 tasks with tied modes, respectively. Panels (b) and (d) retain all 100 tasks and show the number of distinct values per task. The panels measure structural count agreement. Each run retains its original rubric for scoring.
Figure A3: Stage 2 round accounting across 1,600 runs. (a) The 16,279 dual-judged rounds yield 14,734 evaluable records and 1,545 post-judging skips; the latter combine with 26,379 pre-skipped rounds to give 27,924 final skips. The stored error label denotes an automatic candidate flag. (b) Persisted-record composition uses all 42,658 rounds as its denominator: 61.84% pre-skipped summaries, 34.54% detailed evaluable records, and 3.62% post-judge summaries.
| Error type | Diagnostic definition and nearest boundary |
| Wrong entity | The target is identified, substituted, or merged with a different entity, alias, version, institution, product, or location. If the entity is the intended one but fails an eligibility condition, use constraint violation. |
| Constraint violation | A candidate fails an explicit hard condition such as a feature, threshold, status, or eligibility rule. Date, geographic, and population boundaries belong to temporal or scope error. |
| Missing required field | An identifiable answer unit lacks a required attribute or source. An existing value assigned to the wrong field is a mapping error; incorrectly formed units are a grouping error. |
| Insufficient evidence | The available observation is too incomplete to establish a claim, for example a title, metadata, or partial passage lacking the required detail. |
| Source not supporting | Inspected content can be assessed but is irrelevant, omits the claimed fact, supports another value, or contradicts the interpretation. The issue is support for the claim, rather than source type. |
| Source quality error | The source is invalid, inaccessible, unsuitable, or explicitly disallowed by an official/primary-source requirement. |
Table A5: Definitions and category boundaries of all ten diagnostic categories. Every category is defined here; the four selected cases illustrate a subset of the vocabulary. Stored labels remain automatic diagnostic assignments, including when the associated webpage observation itself is correct.
Figure A4: Illustrative source–answer comparisons: source not supporting, constraint violation, source quality error, and wrong entity. The original four-panel layout pairs returned observations with final-answer groups; round units and type descriptions reproduce evaluator interpretations. The accompanying case descriptions identify each model, task, round, and group.
Figure A5: Diagnostic-type composition and assignment coverage. (a) Counts over 6,503 typed candidate nodes; all ten categories are retained. (b) Among 6,545 automatically flagged nodes, 6,503 (99.36%) have a normalized type and 42 (0.64%) remain unassigned. The latter stay in localization and linking totals under a separate unassigned status. Consensus error nodes denote the automatic candidate flags defined in Appendix F .
Figure A6: Extended trajectory and execution diagnostics for 1,600 retained runs. (a) Linked and unlinked candidate flags, including unassigned types. (b) Earliest returned association among 850 trajectories with links. (c) Separately recorded, overlapping execution markers. The marker totals are 25 invalid-final flags, 20 final-only syntheses, 11 premature-answer flags, and 99 resume-marked trajectories.
| Model | All rounds | Evaluable rounds | Flagged nodes | Runs with flag | Linked nodes | Runs with link |
| GPT-6.1-Sol | 1,774 | 762 | 155 | 43 | 56 | 20 |
| GPT-6-Luna-Pro | 1,841 | 552 | 130 | 49 | 43 | 21 |
| DeepSeek-V3.2 | 2,767 | 1,221 | 506 | 86 | 270 | 54 |
| GPT-5.4 | 1,967 | 838 | 284 | 70 | 133 | 53 |
| DeepSeek-V4-Flash | 3,022 | 853 | 304 | 60 | 194 | 47 |
| Kimi-K2.6 | 3,467 | 1,348 | 548 | 72 | 270 | 50 |
Table A6: Raw process counts over 100 trajectories per model. Evaluable rounds have an error or nonerror consensus verdict; final skipped rounds are excluded. Linked nodes and runs use the retained intersection with Stage 1 incorrect groups. Counts summarize observed activity and diagnostic coverage within each model’s retained trajectories.
| Model | Search | Open | Read | Find | Extract | Tools | Rounds |
| GPT-6.1-Sol | 9.49 | 13.41 | 7.75 | 4.30 | 0.85 | 35.80 | 17.74 |
| GPT-6-Luna-Pro | 14.57 | 9.29 | 5.38 | 4.15 | 0.39 | 33.78 | 18.41 |
| DeepSeek-V3.2 | 13.75 | 15.64 | 10.84 | 12.98 | 0.65 | 53.86 | 27.67 |
| GPT-5.4 | 11.19 | 10.40 | 4.86 | 8.68 | 0.76 | 35.89 | 19.67 |
| DeepSeek-V4-Flash | 21.99 | 14.21 | 12.09 | 5.27 | 0.35 | 53.91 | 30.22 |
| Kimi-K2.6 | 21.98 | 17.42 | 15.64 | 8.59 | 0.32 | 63.95 | 34.67 |
Table A7: Mean tool use and normalized length over 100 trajectories per model. The Tools column sums the five tool categories; normalized rounds are counted separately from tool calls.
| Automatic H1 (946 groups) | H1 H2 (172 groups) | ||||||||
| Row label | C | I | U | Total | Row label | C | I | U | Total |
| C | 475 | 21 | 2 | 498 | C | 103 | 5 | 1 | 109 |
| I | 22 | 373 | 1 | 396 | I | 6 | 51 | 0 | 57 |
| U | 1 | 5 | 46 | 52 | U | 0 | 0 | 6 | 6 |
| Total | 498 | 399 | 49 | 946 | Total | 109 | 56 | 7 | 172 |
Table A8: Supplied-group label matrices. Columns identify the second member of each comparison; C/I/U mean correct/incorrect/unverifiable.
| Automatic–H1 | H1–H2 | |||||
| Metric | AS | AP | Upper | AS | AP | Upper |
| Pearson | 0.9054 [0.8496, 0.9519] | 0.9132 [0.8595, 0.9588] | 0.8798 [0.8107, 0.9372] | 0.8961 [0.7335, 0.9825] | 0.9022 [0.7468, 0.9834] | 0.9215 [0.8003, 0.9879] |
| Spearman | 0.9035 [0.8441, 0.9515] | 0.9103 [0.8514, 0.9583] | 0.8816 [0.8107, 0.9370] | 0.8614 [0.6248, 0.9837] | 0.8679 [0.6394, 0.9855] | 0.9003 [0.7314, 0.9876] |
| Kendall | 0.8609 [0.7931, 0.9204] | 0.8712 [0.8054, 0.9299] | 0.8326 [0.7557, 0.8992] | 0.8087 [0.5839, 0.9578] | 0.8136 [0.5899, 0.9623] | 0.8561 [0.6881, 0.9675] |
| MAE | 4.8792 [2.8749, 7.1625] | 4.4542 [2.4832, 6.6833] | 6.2625 [3.7458, 9.1167] | 5.0625 [1.5625, 9.3958] | 5.0625 [1.5625, 9.3958] | 4.5625 [1.3333, 8.7297] |
| Bias | 0.8375 [ , 3.2500] | 0.4625 [ , 2.7833] | 2.0542 [ , 5.1333] | 0.8958 [ , 5.7500] | 0.8958 [ , 5.7500] | 0.3958 [ , 5.0625] |
Table A9: Response-score agreement on 200 automatic–H1 responses and the independent 40-response H1–H2 subset. Each cell gives the point estimate above its 95% confidence interval. AS is Answer Score, AP is Answer Precision, and Upper is Upper Score. AS/Upper errors are points on a 0–100 scale; AP errors are percentage points and AP correlations use 0–1 values. Bias is automatic minus H1 in the left block and H1 minus H2 in the right block. H2 labels do not replace H1 in the primary comparison.
| Human Automatic | Error | Non-error | Skipped | Total |
| Error | 0 | 0 | 0 | 0 |
| Non-error | 329 | 589 | 353 | 1,271 |
| Skipped | 20 | 9 | 1,342 | 1,371 |
| Total | 349 | 598 | 1,695 | 2,642 |
Table A10: Complete process cross-tabulation on 100 trajectories from 50 tasks. The two sets of labels concern different annotation targets.
| Phase | Input (M) | Output (M) | USD |
| Collection | 2,782.84 | 60.50 | 1,812.30 |
| Stage 1 | 157.17 | 118.48 | 676.66 |
| Stage 2 | 102.07 | 44.84 | 454.03 |
| Budgeted total | 3,042.07 | 223.82 | 2,942.98 |
Table A11: Token workload and upper-scenario API budget for the retained-run pipeline. M denotes one million tokens. Collection uses estimated workload; evaluation uses recorded usage. All API amounts are reference-price budgets. Totals are computed before rounding.
More in this topic
Long-Horizon Agent Evaluation
Long-Horizon Agent Evaluation measures whether AI agents can plan, act, remember, and stay reliable across extended tasks and changing environments.
See all 359 papers