AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering

Chanhee Park; Jeongho Yoon; Sungbin Han; Hyeonseok Moon; Heuiseok Lim

cs.CLSep 28, 2026

AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering

Authors: Chanhee Park, Jeongho Yoon, Sungbin Han, Hyeonseok Moon, Heuiseok Lim

Organizations: Korea University · Sookmyung Women’s University

Abstract

Agentic tasks require a large language model to interact with the world, navigating information and gathering evidence across multiple steps with restricted resources. Due to this complexity, agentic task failures arise from various sources, and pinpointing these failure causes is essential to diagnose and improve agentic systems. Existing benchmarks, however, tend to focus on a single leaderboard score, leaving the underlying failure modes opaque. To fill this gap, we introduce AgentHop, a diagnostic benchmark of 1,011 multiple-choice questions paired with a controlled seven-tool sandbox under fixed token, turn, and tool-call constraints. AgentHop reveals model vulnerabilities by dissecting a single accuracy score along four axes of agent operation: retrieval, synthesis, tool-call, and resource management. Across 19 models, we find that behavior clusters by model family, with tool-call signatures revealing distinct family fingerprints: GPT models commit early, Anthropic and GLM checkpoints verify before committing, DeepSeek and Kimi over-search, and Gemini-3 Pro stays balanced. Decomposed axes further expose within-family structure: Claude Opus 4.6 and Sonnet 4.6 land within one accuracy point yet diverge on retrieval-versus-synthesis emphasis, with Opus retrieving more and Sonnet synthesizing better. We release the full benchmark set and the harness to support diagnostic agent benchmarking.

Figures & tables

Benchmark	N	Search	Synth.	Tool-use	Resource	Diagnostic
HotpotQA [ 16 ]	113K	✓	✓
2WikiMultiHopQA [ 17 ]	167K		✓
OpenScholar [ 7 ]	2,967	✓	✓
WebArena [ 18 ]	812	✓	✓	✓
GAIA [ 19 ]	466	✓	✓	✓
BrowseComp [ 20 ]	1,266	✓	✓	✓

Table 1: Comparison of AgentHop with existing benchmarks along the four ability axes and a diagnostic indicator.

Figure 1: Eight-stage construction pipeline. Four generation stages (top) seed citation chains from $953$ source papers across nine CS venues, draft grounded question–answer pairs, and assign each pair three engineered distractors. Four filtering stages (bottom) retire candidates via an automated 3-model ensemble check, a structural-integrity sanity pass, a 7-auditor human review, and a post-audit recall-and-length pass; $1,011$ items remain.

Model	Search $\uparrow$				Synthesis $\uparrow$		Tool-use	Resource $\downarrow$			Failure Ratio $\downarrow$			Outcome $\uparrow$
	Recall (ST)	Recall (MT)	SecRec (ST)	SecRec (MT)	Conversion (ST)	Conversion (MT)	Calls /Turn	Avg. Tokens	Avg. Turns	Avg. Budgets	Max Tokens	Max Turns	Max Budgets	Accuracy
Closed Frontier
Gemini-3 Pro	0.868	0.752	0.859	0.655	0.957	0.970	1.05	92.4	9.4	19.8	4.4	0.1	0.0	0.891
Gemini-3 Flash	0.724	0.501	0.680	0.386	0.625	0.644	1.01	180.3	14.1	27.9	27.7	2.1	0.2	0.547
GPT-5.3 codex	0.812	0.623	0.775	0.470	0.896	0.902	1.44	69.5	8.5	25.2	0.3	0.0	0.7	0.825
GPT-5.4	0.808	0.643	0.752	0.454	0.878	0.846	1.74	67.5	7.7	26.5	0.3	0.0	4.9	0.790

Table 2: Main AgentHop results across 19 models, grouped by capability tier and ranked by accuracy within each family. Highlighted cells mark tier leaders; bold marks the panel best. Axis and failure definitions in Section 3.3 . Conversion cells are shaded only where a paired same-item test separates the tier leader from the runner-up, as reported in Appendix E.1 .

Model	n	Recover	Stay	Think	Re-nav	Submit
Closed Frontier
Gemini-3 Pro	882	3.2%	33.9%	5.4%	50.2%	7.3%
Gemini-3 Flash	2165	1.2%	39.7%	1.5%	50.2%	7.0%
GPT-5.3 codex	1559	1.9%	39.8%	0.0%	53.4%	4.9%
GPT-5.4	1652	3.1%	39.6%	1.8%	52.1%	3.0%
Claude Opus 4.6	904	1.4%	12.7%	82.3%	3.4%	0.1%

Table 3: Action distribution after a wrong-paper read_section call. Columns: Recover reads a gold paper next, Stay reads another non-gold section, Think invokes think , Re-nav calls any other navigation tool, and Submit commits an answer immediately.

Figure 2: Within-family generation gap on overall accuracy. Each panel pairs older-generation checkpoints (light fill) with newer-generation checkpoints (dark fill) within the same architectural cohort, with arrows annotated by the within-family $\Delta$ in percentage points.

Model	Zero-tool	$\Delta_{\text{tools}}$	No-search	$\Delta_{\text{kw}}$	Full-tool
	(%)	(pp)	(%)	(pp)	(%)
DeepSeek-chat	18.3	$+35.0$	53.3	$+3.1$	56.4
Gemma-4-31B	31.7	$+23.1$	54.8	$+10.0$	64.8
GPT-5.3 codex	48.6	$+30.7$	79.3	$+3.2$	82.5

Table 4: Ablation on three checkpoints, with each delta paired between adjacent settings. Zero-tool : closed-book. No-search : full sandbox minus search_papers . Full-tool : seven-tool sandbox.

\Delta_{\text{tools}}

is the lift from gaining a tool sandbox (zero-tool

\to

no-search);

\Delta_{\text{kw}}

is the additional lift from adding search_papers (no-search

\to

full).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Description	Drops	Cumulative
Generation
1	Seed selection	—	953
2	Citation-chain expansion	—	17,983
3	Q/A generation (BFS + DFS)	—	1,651
4	Distractor generation (3-trap)	—	1,651
Filtering

Appendix

Table 5: Per-stage attrition through the eight-stage construction pipeline. Stages 1–4 grow each sample from a seed paper to a fully populated four-option MCQ; Stages 5–8 then filter the resulting pool.

Theme	$n$	Description
Paper-content extraction broken	12	The gold paper’s HTML or section-extraction pipeline produced empty, mangled, or partially captured sections; the auditor saw that the cited paper content needed to ground the answer was missing.
Weak chain / target paper under-utilised	6	The synthesis relies on only one of the two cited papers, or the bridge $\to$ terminal step does not actually carry the information the question demands.
Trivial or low-quality question	6	The question or its gold answer is too simple to be diagnostic of multi-hop competence (e.g., the answer is a single-word fact derivable without genuine multi-paper reasoning).
Answer-question alignment failure	6	The gold answer is on-topic but does not satisfy what the question explicitly asks ( how vs what , missing tradeoff or comparison, synthesis claim that doesn’t follow from the cited evidence).
Other (sundry, 1 each)	5	Option text misquotes the paper; a numerical detail is incorrect; a non-gold option is also defensible; the answer compares the wrong pair of methods; or the asserted target-paper claim actually originates in the seed paper.
Total retired at Stage 7	35

Appendix

Table 6: Themes among the 35 samples retired at Stage 7 after manual triage of the 116 audit-flagged candidates. The largest cluster is paper-content extraction failures that the automated structural-integrity check at Stage 6 could not catch because they occur deep inside section text rather than at the metadata level.

Figure 3: Gradio human-audit interface used in Stage 7. Each sample is presented with the question, four options (gold and three distractors), the GPT-5.4 audit JSON, and links to underlying papers. Auditors mark three quality axes and submit one of three verdicts. The same interface is released alongside the benchmark, constrained to the seven-tool, 30-point budget environment, for future human-vs-LLM head-to-head studies.

Figure 4: Sample-difficulty distribution across the $1{,}011$ -sample benchmark. Each sample is bucketed by how many of the $19$ evaluated models answered it correctly: low ( $0$ – $6$ , red), mid ( $7$ – $12$ , yellow), high ( $13$ – $19$ , green). (a) Stratified by question type $\times$ depth. (b) Stratified by consensus tier. (c) Per-sample coverage histogram, with bar color matching the low/mid/high coding.

Display name	Identifier	Display name	Identifier
Gemini-3 Pro	gemini-3-pro-preview	MiniMax-M2.7	MiniMaxAI/MiniMax-M2.7
Claude Opus 4.6	claude-opus-4-6	Qwen3.5-27B	Qwen/Qwen3.5-27B
GPT-5.3-codex	gpt-5.3-codex	Qwen3.5-35B-A3B	Qwen/Qwen3.5-35B-A3B
Claude Sonnet 4.6	claude-sonnet-4-6	Qwen3.6-35B-A3B	Qwen/Qwen3.6-35B-A3B
GPT-5.4	gpt-5.4	Qwen3-32B	Qwen/Qwen3-32B
Gemini-3 Flash	gemini-3-flash-preview	Qwen3-30B-A3B	Qwen/Qwen3-30B-A3B-Instruct-2507

Appendix

Table 7: Mapping from each display name used in the paper (and in Table 2 ) to the canonical API model identifier or Hugging Face repository path actually invoked. Qwen3-30B-A3B is the Instruct-2507 snapshot. During our evaluation window (after the 2025-12-01 stable release of DeepSeek-V3.2 [ 29 ] ), both DeepSeek API aliases resolve to the same underlying V3.2 checkpoint: deepseek-chat invokes the non-thinking mode and deepseek-reasoner the thinking mode. Reproducing our numbers therefore requires the V3.2 weights rather than the older V3/R1 weights to which these aliases pointed before the swap.

Model	Full-tool ($)	Zero-tool ($)	$/sample (full)	Input/sample (K)	Output/sample
Gemini 3 Pro	159.27	37.14	0.158	87	5,003
Gemini 3 Flash	77.49	25.17	0.077	169	8,517
Claude Opus 4.6	405.54	79.47	0.401	107	3,890
Claude Sonnet 4.6	205.23	43.86	0.203	92	4,165
GPT-5.3-codex	80.32	14.80	0.079	67	2,208
GPT-5.4	78.99	28.14	0.078	65	2,444

Appendix

Table 8: Per-model API cost and average per-sample token consumption for the 11 API-served models on the 1,011-sample full split, alongside zero-tool ablation cost. All costs use the official per-token pricing for each provider as recorded in the released harness. The 8 open-weight Qwen and Gemma checkpoints (Qwen3-32B, Qwen3-30B-A3B, Qwen3.5-27B, Qwen3.5-35B-A3B, Qwen3.6-35B-A3B, Gemma-3-27B, Gemma-4-26B-A4B, Gemma-4-31B) are served locally on a pair of H100 GPUs and excluded.

	MT-fail					ST (after gold)				MT-both (after both)
Model	$n$	0-Gold	Prem	Explored	Stayed	$n$	Imm	Thnk	Read	$n$	Imm	Thnk	Read
Closed Frontier
Gemini-3 Pro	110	20.9	17.3	40.0	21.8	493	35.7	11.6	48.7	333	39.6	10.2	47.4
Gemini-3 Flash	221	24.0	4.1	56.6	15.4	411	5.1	1.5	89.1	222	13.1	3.6	74.3
GPT-5.3 codex	167	16.2	34.7	31.1	18.0	461	47.3	0.2	47.1	276	66.3	1.1	22.5
GPT-5.4	158	17.1	53.2	27.2	2.5	459	31.4	14.4	45.8	285	38.9	29.5	21.1

Appendix

Table 9: Stopping behavior across MT-fail, ST, and MT-both scopes for the 19-model panel. Each block’s

n

column reports trajectories in scope; subsequent columns report the within-scope share (%) of each category. MT-fail’s four categories sum to 100%; ST and MT-both’s three categories sum to

\leq 100

% (residual is navigation without reading).

Model	st_d1	st_d2	mt_d1	mt_d2	$\Delta_{\text{st\_d1}-\text{mt\_d2}}$
	( $n=90$ )	( $n=478$ )	( $n=396$ )	( $n=47$ )	(pp)
GPT-5.3 codex	$94.4\pm 3.2$	$78.0\pm 3.5$	$85.4\pm 3.1$	$80.9\pm 8.7$	$+13.6$
DeepSeek-chat	$73.3\pm 8.0$	$53.8\pm 4.4$	$56.6\pm 4.8$	$48.9\pm 13.8$	$+24.4$
Gemma-4-31B	$72.2\pm 8.2$	$56.3\pm 4.4$	$72.7\pm 4.2$	$70.2\pm 11.1$	$+2.0$

Appendix

Table 10: Per-cell accuracy on the question-type

\times

depth grid for one representative checkpoint from each tier (point estimate

\pm

Wilson 95% CI half-width).

\Delta

is the drop from the easiest to hardest cell. GPT-5.3 codex holds within

17

pp across all four cells; DeepSeek-chat collapses on multi-target depth-2; Gemma-4-31B is uniformly capped, suggesting that what limits its accuracy is tool-environment overhead rather than synthesis depth.

Group	$n$	Share	Conversion	Zero-tool
Read a direct -labelled section (header match)	4,725	42.7%	74.0%	35.2%
Read the labelled content inside its parent section	4,765	43.0%	78.1%	40.1%
Read no direct -labelled content	1,582	14.3%	49.3%	31.5%

Appendix

Table 11: Evidence-contact decomposition of the

11{,}072

recall-credited trajectories, pooled over the 19-model panel. Conversion is cap-corrected. The zero-tool column reports the same models’ closed-book accuracy on the same items.

	Pair	$n$	Discordant	$p$
MT	Gemini-3 Pro vs Claude Sonnet 4.6	226	20/3	0.0005
MT	Claude Sonnet 4.6 vs GPT-5.4	193	24/16	0.27
MT	GPT-5.4 vs GLM-5.1	227	36/22	0.09
MT	GLM-5.1 vs DeepSeek-reasoner	203	35/21	0.08
MT	DeepSeek-reasoner vs MiniMax-M2.7	168	42/12	0.0001
Closed, ST	Gemini-3 Pro vs Claude Sonnet 4.6	437	25/13	0.073

Appendix

Table 12: Exact McNemar tests on common recalled items (discordant pairs; two-sided

p

). Top: adjacent pairs among the six highest multi-target conversion scores. Bottom: tier conversion leader against tier runner-up, per axis.

Figure 5: Accuracy versus paper recall across the 19-model panel, colored by tier. The diagonal $y=x$ marks accuracy tracking retrieval one-to-one. Models above the diagonal (Gemini-3 Pro, GPT-5.3 codex, Gemma-4-31B) combine retrieval with synthesis lift; models below (Kimi-K2.5, MiniMax-M2.7, the Qwen MoE family, DeepSeek-chat) retrieve correctly but fail to convert.

Bigram	$\Delta$ (pp)	Sig. count	Models
Productive (positive $\Delta$ )
read_section $\to$ think	$+13.6$	$7/19$	$14/19$
think $\to$ submit_answer	$+12.6$	$6/19$	$14/19$
list_sections $\to$ read_section	$+8.4$	$2/19$	$13/19$
get_references $\to$ get_paper_info	$+8.3$	$5/19$	$15/19$
get_paper_info $\to$ get_references	$+8.2$	$4/19$	$14/19$

Appendix

Table 13: Top productive and anti-productive bigrams across the 19-model panel, ranked by panel-mean

|\Delta|

\Delta

is the panel-mean accuracy difference between trajectories containing and not containing the bigram. Sig. count is the number of individual models for which the per-model effect remains significant after Bonferroni correction over the

7\times 7

candidate space; Models is the number with the bigram present in

\geq 1

trajectory.

Model	Top-1 trigram	Concentration
Closed Frontier
Gemini-3 Flash	gpi $\to$ ls $\to$ rs	$0.100$
Gemini-3 Pro	ls $\to$ rs $\to$ rs	$0.091$
Claude Sonnet 4.6	gpi $\to$ gr $\to$ th	$0.090$
GPT-5.4	ls $\to$ rs $\to$ rs	$0.089$
GPT-5.3 codex	gpi $\to$ ls $\to$ rs	$0.089$

Appendix

Table 14: Per-model top-1 trigram and concentration share. Concentration is the fraction of all length-3 sub-patterns occupied by the most-frequent trigram (higher = more strategically locked-in). Models grouped by tier; ranked within tier by concentration.

Figure 6: Per-model tool-call transition matrices for the 19 evaluated models. Each panel is a $7\times 7$ heatmap whose rows index the previous tool action and whose columns index the next; cell color encodes the row-normalised conditional probability $P(\text{next}\mid\text{prev})$ (deeper blue = higher transition probability). Within every panel the tool order along both axes is, top-to-bottom and left-to-right: get_paper_info , get_references , list_sections , search_papers , read_section , think , submit_answer . Panels are ordered to keep within-family checkpoints adjacent.

Model	Acc	NA	MaxT	Bud	Tok	Model	Acc	NA	MaxT	Bud	Tok
Closed Frontier						Open Frontier
Gemini-3 Pro	89.1	4.5	0.0	0.0	4.5	GLM-5.1	69.6	7.9	0.4	3.2	4.4
GPT-5.3 codex	82.5	1.0	0.0	0.7	0.3	DeepSeek-R1	59.6	6.4	0.3	0.0	6.1
GPT-5.4	79.0	5.2	0.0	4.9	0.3	DeepSeek-V3	56.4	14.3	0.3	0.0	14.0
Claude Opus 4.6	77.5	6.7	0.0	0.0	6.7	Kimi-K2.5	44.1	9.9	2.0	2.4	5.5
Claude Sonnet 4.6	76.6	4.4	0.6	0.0	3.8	MiniMax-M2.7	41.3	7.2	1.3	4.8	1.1

Appendix

Table 15: Non-submission terminations across the 19-model panel under the cap-corrected protocol. NA is the fraction of trajectories that fail to produce a parseable answer letter; the three rightmost columns decompose NA by terminating cap (max-turns, budget, token-limit; % of all trajectories). A small residual (

\leq 0.7

pp on Qwen3-32B and Qwen3-30B-A3B) reflects voluntary submit_answer calls with non-letter output. Termination categories are defined in Appendix F.2 .

Model	GPQA-D	HLE	SWE-B Pro	T-Bench 2.0	NL2Repo	BrowseComp	HLE+tools	AgentHop
Closed Frontier
Gemini-3 Pro	91.9	37.5	43.3	56.9	—	59.2	—	89.1
GPT-5.3 codex	92.6	—	56.8	77.3	—	77.3	—	82.5
GPT-5.4	92.8	39.8	57.7	75.1	41.3 ∗	82.7	52.1	79.0
Claude Opus 4.6	91.3	40.0	57.3 ∗	65.4	49.8 ∗	84.0	53.1	77.5
Claude Sonnet 4.6	89.9	33.2	—	59.1	—	74.0	49.0	76.6

Appendix

Table 16: Per-model scores on seven external benchmarks (publicly reported, normalised to percent) alongside AgentHop accuracy. Models grouped by tier and ranked by AgentHop accuracy. ∗ denotes a proxy imputed from a closely related sibling checkpoint;

\dagger

denotes a context-managed variant. deepseek-chat (the non-thinking V3.2 invocation) lacks public benchmark scores at the time of this submission and is reported as —.

Model	Metric	Run 0	Run 1	Run 2	Mean	Std	Range
DeepSeek-chat	Accuracy	0.564	0.553	0.542	0.553	0.009	0.022
	Section recall	0.439	0.452	0.433	0.441	0.008	0.019
	Conversion	0.669	0.652	0.642	0.654	0.011	0.027
	Tokens (k)	156.7	156.1	155.1	156.0	0.7	1.6
Gemma-4-31B	Accuracy	0.648	0.585	0.581	0.605	0.031	0.067
	Section recall	0.275	0.242	0.251	0.256	0.014	0.033

Appendix

Table 17: Run-to-run figures across three runs for the three multi-run checkpoints. The two closed-frontier-style checkpoints (DeepSeek-chat, GPT-5.3 codex) show very low variance (

<0.011

standard deviation on every metric except absolute tokens). Gemma-4-31B is more variable: accuracy std

0.032

, conversion std

0.047

, and notably tokens std

10.2

K (range

21.8

K), a spread comparable to the cross-model gap between adjacent closed-frontier checkpoints. Cross-model rank orderings on accuracy and on each of the four axes are nonetheless preserved across all three runs.

Model	Zero	Full	$\Delta$	Model	Zero	Full	$\Delta$
Closed Frontier				Open Frontier
Gemini-3 Pro	73.4	89.1	$+15.7$	GLM-5.1	37.6	69.6	$+32.0$
GPT-5.3 codex	48.6	82.5	$+33.9$	DeepSeek-reasoner	25.0	59.5	$+34.5$
GPT-5.4	59.3	79.0	$+19.7$	DeepSeek-chat	18.3	56.4	$+38.1$
Claude Opus 4.6	55.2	77.5	$+22.4$	Kimi-K2.5	12.1	44.1	$+32.0$
Claude Sonnet 4.6	48.5	76.6	$+28.1$	MiniMax-M2.7	14.5	41.3	$+26.8$

Appendix

Table 18: Parametric (zero-tool) vs. full-tool accuracy across the 19-model panel, tier-grouped and ranked by full-tool accuracy within each tier. Chance baseline is

25\%

; negative

\Delta

indicates that the seven-tool sandbox reduces accuracy below the closed-book baseline.

Axis	Stratum	$n$	Closed	Open	Comp.-eff.
Citations	Q1 (1–224)	252	57.5	21.7	19.2
	Q2 (226–585)	253	59.2	22.2	17.8
	Q3 (586–1,554)	253	57.1	19.9	16.5
	Q4 (1,554–54,992)	253	54.4	22.2	16.7
Year	2022	72	53.5	18.3	19.6
	2023	355	54.7	19.7	15.9

Appendix

Table 19: Zero-tool accuracy (%) by stratum, with per-tier decomposition. Quartiles are rank-based over the

1{,}011

samples; ranges give citation counts.

Tool	Cost	Input	Returns
get_paper_info	1	paper_id	Title, abstract, year, and venue of the paper.
get_references	1	paper_id	List of cited papers with their IDs, titles, years, and the citation context (the sentence in which they were cited).
list_sections	1	paper_id	Section headers with alphabet aliases and per-section character counts, for example (A) Introduction (5869 chars) . Aliases may be passed to read_section in place of a full header, and the counts let an agent price a 5-point read before paying for it.
search_papers	1	query (str), top_k ( $\leq 10$ , default 5)	Top- $k$ matching papers from the available pool, each with title and abstract.
read_section	5	paper_id , section	Full text of the requested section. The section argument accepts the full header, the alphabet alias from list_sections , or a unique case-insensitive substring.
think	0	thought (str)	Acknowledgement; the thought is appended to the trajectory transcript and remains available for the agent’s later reasoning steps.

Appendix

Table 20: The seven tools exposed to the agent. Costs are per call, deducted from the per-trajectory 30-point budget at invocation. Identical schemas, identical input typings, and identical natural-language descriptions are presented to every model.

Explore similar work

Feb 22, 2026cs.CL

AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAG

With the rapid advancement of agent-based methods in recent years, Agentic RAG has undoubtedly become an important research direction. Multi-hop reasoning, which requires models to engage in deliberate thinking and multi-step interaction, serves as a critical testbed for assessing such capabilities. However, existing benchmarks typically provide only final questions and answers, while lacking the intermediate hop-level questions that gradually connect atomic questions to the final multi-hop query. This limitation prevents researchers from analyzing at which step an agent fails and restricts more fine-grained evaluation of model capabilities. Moreover, most current benchmarks are manually constructed, which is both time-consuming and labor-intensive, while also limiting scalability and generalization. To address these challenges, we introduce AgenticRAGTracer, the first Agentic RAG benchmark that is primarily constructed automatically by large language models and designed to support step-by-step validation. Our benchmark spans multiple domains, contains 1,305 data points, and has no overlap with existing mainstream benchmarks. Extensive experiments demonstrate that even the best large language models perform poorly on our dataset. For instance, GPT-5 attains merely 22.6% EM accuracy on the hardest portion of our dataset. Hop-aware diagnosis reveals that failures are primarily driven by distorted reasoning chains -- either collapsing prematurely or wandering into over-extension. This highlights a critical inability to allocate steps consistent with the task's logical structure, providing a diagnostic dimension missing in traditional evaluations. We believe our work will facilitate research in Agentic RAG and inspire further meaningful progress in this area. Our code and data are available at https://github.com/YqjMartin/AgenticRAGTracer.

Qijie You, Wenkai Yu, Wentao Zhang

University of Science and Technology Beijing · 2Peking University · 3Zhongguancun Academy +1

Agentic Benchmarks Agentic Reasoning Multi-Hop Reasoning Agentic

Ask PDF

Jul 6, 2026cs.CL

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Harsh Soni

UC Berkeley

Large Language Model Agents Large Language Model Tool Use

Ask PDF

May 11, 2026cs.CL

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories. Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls, and runs inside a reproducible Docker container hosting an actual CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to real tools rather than mock services. Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification. Across 19 frontier models, the best, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, while every other model stays below 60%, and switching harness alone shifts a single model by up to 18 points. These results show that long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models. We release the tasks, code, and containerized tooling to support reproducible evaluation.

Shuangrui Ding, Xuanlang Dai, Long Xing +14

1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8

Agentic Benchmarks Agentic Evaluations Openclaw Long-Horizon Task Planning Real-World

Ask PDF