Where Scientific Search Agents Fail: Decision-Checkpoint Auditing of Exposure and Inspection Attempts

Hongmin Li; Wanli Zhao

cs.AISep 29, 2026

Where Scientific Search Agents Fail: Decision-Checkpoint Auditing of Exposure and Inspection Attempts

Authors: Hongmin Li, Wanli Zhao

Organizations: School of Life Science and Technology, Institute of Science Tokyo Tokyo, Japan · Department of Computational Biology and Medical Sciences Graduate School of Frontier Sciences, The University of Tokyo Kashiwa, Japan · Department of Information Science Graduate School of Advanced Science and Engineering, Hiroshima University Higashihiroshima, Japan

Abstract

Final-answer accuracy does not reveal whether a scientific-search agent failed to encounter a target paper, attempt to inspect it, or return an accepted answer after inspection. We introduce decision checkpoints that record observations and tool actions without benchmark labels during inference, then join target identities and evaluator labels to assign outcome categories from recorded events. Across five conditions on 540 answerable AutoResearchBench Deep questions in a fixed, target-enriched environment, keyword search achieves 24.6% accuracy, compared with 17.8% for raw search. The keyword condition has fewer incorrect answers with neither target exposure nor inspection, but more incorrect answers after the target is exposed and left uninspected. Compared with keyword search, read-first has 27.4% more recorded evidence-search calls. Target inspection attempts occur on 199 questions under read-first and 191 under keyword search; both conditions achieve 24.6% accuracy. The checkpoint protocol makes these question-level differences explicit, distinguishing target exposure and inspection from aggregate accuracy and total tool use.

Figures & tables

Figure 1: Schematic of decision-checkpoint auditing. (a) Three distinct failure paths share the same terminal score. (b) Inference-time checkpoints contain no ground-truth (GT) labels; target identities and evaluator labels are added afterward. Observable trajectories are classified by correctness, then inspection and exposure. API exceptions without trajectories form a separate unobservable category. Inspection denotes an executed within-paper evidence-search attempt.

Figure 2: Decision-checkpoint audit on all 540 answerable Deep questions (+both backend, request alias gpt-5.6-luna). (a) Correct answers take precedence; remaining questions are classified by inspection and exposure. Raw and keyword conditions differ in neither exposed nor inspected (333

\to

64) and exposed, not inspected (67

\to

277) failures, with Acc@1 of 17.8% and 24.6%. (b) Cumulative fraction of questions whose target first appears within a given rank of an observation; under kw-ft the target is exposed within rank 20 on 336/540 questions.

(a) Outcome categories: counts, with accuracy in parentheses
Condition	Correct $n$ (%)	Neither exposed nor inspected	Exposed, not inspected	Inspected	Unobs.
raw-ft	96 (17.78)	333	67	44	0
kw-ft	133 (24.63)	64	277	66	0
read-first	133 (24.63)	62	273	71	1
decomp-ft	138 (25.56)	67	277	58	0
adaptive-ft	122 (22.59)	68	282	68	0

Table 1: Outcome categories across five conditions on the same 540 questions. Categories are mutually exclusive; all except Correct and Unobservable are incorrect outcomes. Exposure and inspection refer to the target paper. Exposure is also counted separately.

raw $\downarrow$ kw $\rightarrow$	Neither exposed nor inspected	Exposed, not inspected	Inspected	Correct
Neither exposed nor inspected (333)	52	182	35	64
Exposed, not inspected (67)	4	46	8	9
Inspected (44)	6	13	12	13
Correct (96)	2	36	11	47

Table 2: Per-question outcome transitions raw-ft

\to

kw-ft on the same 540 questions. All categories except Correct are incorrect outcomes; exposure and inspection refer to the target paper.

First-exposure rank	1–5	6–20	21–50	51–200
Exposed questions $n$	175	161	79	61
Later target inspection, $n$ (%)	92 (52.6)	60 (37.3)	26 (32.9)	13 (21.3)
Correct answers, $n$ (%)	65 (37.1)	39 (24.2)	21 (26.6)	8 (13.1)

Table 3: Keyword first-exposure rank, subsequent target inspection attempts and final correctness. All counts are numbers of questions. Percentages use the exposed questions in each rank group as denominators. “Later target inspection” requires an executed target call on a strictly later turn.

(a) Correctness on all $N=540$ questions
Comparison	Both correct	Gain	Loss	Both wrong	$\Delta$ Acc. (pp)
raw $\to$ kw	47	86	49	358	+6.85
kw $\to$ read-first	73	60	60	347	0.00
kw $\to$ decomp	76	62	57	345	+0.93
kw $\to$ adaptive	69	53	64	354	$-2.04$
(b) Target inspection attempt: kw $\to$ read-first, $N=539$ jointly observable

Table 4: Paired correctness and questions with a target inspection attempt. Gains and losses refer to the left-to-right policy change.

Condition	Searches	Evidence-search	Tokens	Normal	Context	Turn	API
	/q	calls/q	/q (k)		limit	limit	exception
raw-ft	3.61	8.43	844	386	131	23	0
kw-ft	7.49	7.56	807	306	198	36	0
read-first	7.51	9.63	846	301	172	66	1
decomp-ft	7.38	7.43	762	290	205	45	0
adaptive-ft	7.36	7.85	791	287	203	50	0

Table 5: Recorded inference use and termination. Means use all 540 primary episodes per condition; termination columns are counts. Evidence-search calls include all candidate papers. The second panel separates call origins in the 539 observable read-first trajectories.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Measure	raw-ft	kw-ft
Never-exposed failures (%)	61.7	11.9
Exposed, uninspected failures (%)	12.4	51.3
Inspected, incorrect failures (%)	8.1	12.2
Correct (%)	17.8	24.6
Search-result lists containing the target per question	1.0	3.8
Uninspected-failure stops: normal / turn / context limit	47 / 4 / 16	110 / 24 / 143

Appendix

Table 6: Supplementary exposure, outcome and resource statistics for raw and keyword search on 540 questions per condition.

Explore similar work

Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents
Aug 3, 2026Qi Liu, Jiaxin Mao, Fengbin Zhu +1Search Agents Relevance
Renmin University of China Beijing, China · National University of Singapore Singapore
SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents
Aug 5, 2026Zhixiang Liang, Yifei Liu, Yidan Huang +5Search Agents Model Auditing
University of Illinois Urbana-Champaign · 2Joy Future Academy, JD
Search, Inspect, Fetch: Exploiting Structure-Aware Boolean Retrieval for Deep-Research Agents
Aug 3, 2026Shuai Wang, Haodong Chen, Yu Yin +3Retrievers Fetch
The University of Queensland, Brisbane, Australia · CSIRO, Brisbane, Australia

Aug 3, 2026, cs.AIJ/K move · Enter open · S save

Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents

Qi Liu, Jiaxin Mao, Fengbin Zhu, Tat-Seng Chua

Renmin University of China Beijing, China · National University of Singapore Singapore

Search Agents Relevance Long-Horizon Task Planning Reasoning Gaps Analytics

Open paper AskPDF arXiv

Deep search agents answer difficult information-seeking questions by iteratively issuing search queries to gather supporting evidence, but it remains unclear whether and how greater search effort leads to better answers. We study these questions through a trajectory-level diagnosis of long-horizon search agents. Using human-annotated document-level relevance judgments, we evaluate the evidence retrieved at each search step and separate two stages of agent behavior: what evidence an agent retrieves and how effectively it uses that evidence. This distinction further allows us to decompose failures into retrieval gaps, where the necessary evidence is never found, and utilization gaps, where relevant evidence is retrieved but not used correctly. With the retrieval model and evaluation harness held fixed, we compare six agents on BrowseComp-Plus and further validate our findings on BrowseComp with an open-web search API. Across settings, we find that search effort and answer quality are only weakly aligned. Answer accuracy is better correlated with the quality of retrieved evidence, especially cumulative retrieval recall, than with the number of searches or the amount of context consumed. Useful evidence often appears early in the trajectory, yet agents tend to continue searching, producing a long tail of low-yield retrieval steps. At the query level, exploratory reformulations remain useful, but the best-performing agents issue far fewer redundant queries. Overall, by systematically characterizing the search behavior and failure modes of long-horizon search agents, this work points to practical directions for building better deep research systems, including stronger query formulation, more effective evidence selection and context management, and stopping criteria based on whether sufficient supporting evidence has been retrieved.

Abstract

Explore similar work

Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents

SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

Search, Inspect, Fetch: Exploiting Structure-Aware Boolean Retrieval for Deep-Research Agents