How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks

Yanlin Fei; Nazhou Liu; Xinmiao Yu; Shaolong Chen; Lei Li; Rahul Thapa; Madalina Ciobanu; Navan Preet Singh

cs.CLAug 14, 2026

How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks

Authors: Yanlin Fei, Nazhou Liu, Xinmiao Yu, Shaolong Chen, Lei Li, Rahul Thapa, Madalina Ciobanu, Navan Preet Singh,

Organizations: Prentis AI · Stanford University · Titan Holdings

Abstract

AI has long assisted scientific research, but the rapid advance of LLMs and agentic scaffolds is reshaping the landscape; a single system can now carry whole-stage research from an initial hypothesis all the way to final published paper, which is a paradigm now referred to as AutoResearch. Existing evaluations reveal little about how these agents operate or where they break down. Tasks are narrowly-scoped, evaluation measures performance but not process, and failure diagnoses lack systematic coverage or artifact-level visibility. To address this gap, we introduce AutoResearchEval, featuring 100 tasks grounded in published frontier science across 7 scientific domains and the full research lifecycle, including ideation, retrieval, execution, analysis, writing, and review. Evaluating 8 harness-model combinations yields 800 autoresearch agent trajectories, with process-level annotation. We organize these insights into AutoResearch Failure Taxonomy or ARFT, a framework of 45 empirically-grounded failure patterns. To enable scalable fine-grained attribution, we leverage a human-calibrated agent-as-a-judge pipeline to inspect complete trajectories and intermediate artifacts. Failure patterns converge on a single overarching limitation, namely that current agents lack a metacognitive loop, which entails the ability to check what they produced against what they found, revise when it does not hold up, and question whether the path they took was sound. The same patterns recur across all 8 harness-model combinations, including the strongest models tested, locating the deficit at the model level rather than in any particular scaffold; whether orchestration-level interventions can close it is an open question this work does not test. We publicly release AutoResearchEval and ARFT to facilitate continued research and development in autonomous scientific discovery.

Figures & tables

Figure 1: Construction, rollout, and evaluation of AutoResearchEval. a, 5,878 papers from nine domains are parsed into seven fields and filtered to 100 tasks spanning seven domains; the agent sees only the query (Premise, Tension), while the target (KeyClaims, Conclusion) is withheld. b, Each task runs once per harness–model pair as a six-stage episode with revision, yielding 800 trajectories with all artifacts retained. The case study traces one trajectory: six stage-wise deviations converging on a single metacognitive deficit. c, ARFT is induced bottom-up: experts annotate failures in full trajectories, group them into patterns, and refine until all agree, giving 45 patterns under 4 root-cause pillars. A judge agent then reviews the full artifact set under a per-stage rubric, anchoring every issue to concrete evidence and categorize it to an ARFT pattern, with a quality checker regenerating weak analyses in a self-healing loop. It reaches $\kappa=0.75$ (pattern) and $0.83$ (taxonomy) against human labels, versus $0.53$ and $0.62$ for a single-call LLM-as-a-judge.

Figure 2: Composition of AutoResearchEval (

N=100

), each axis split by task type: scientific domain (left) and novelty-move type (right). Open-ended discovery in blue, target-anchored optimization in terracotta.

Harness	Backbone models
Claude Code	opus-4.8, claude-sonnet-5, qwen3.7-max,
	glm-5.2, minimax-m3, deepseek-v4-pro
Codex	gpt-5-mini
Gemini CLI	gemini-3.5-flash

Table 1: Harness–model combinations under evaluation.

Method	Level	Accuracy	Precision	Recall	F1	Cohen’s $\kappa$
LLM-as-a-Judge (claude-opus-5)	Pattern	84.6	70.2	63.5	66.7	0.53
LLM-as-a-Judge (claude-opus-5)	Taxonomy Categorization	89.3	78.8	72.1	75.3	0.62
Agent-as-a-Judge	Pattern	92.1	85.4	80.7	83.0	0.75
Agent-as-a-Judge	Taxonomy Categorization	95.3	91.0	87.2	89.1	0.83

Table 2: Agreement with human expert annotation on the 50 validation trajectories. The LLM-as-a-Judge baseline receives only the transcript in a single call; the Agent-as-a-Judge receives the full evidence package and operates under the structured rubric. Pattern measures per-pattern hit/miss agreement across the 45 ARFT patterns; Taxonomy Categorization measures agreement at the root-cause pillar level.

Root Cause Pillar	Core Failure Focus	Mapped Failure Patterns (IDs)
R1. Grounding & Faithfulness	Disconnect between high-level claims/hypotheses and ground-truth code, data, or logs.	A.6, B.1, B.2, B.5, C.3, D.1, D.4, D.6, E.1, E.4, F.6, X.6
R2. Cognitive Depth & Adaptability	Shallow reasoning/search, passivity in self-review, and inability to re-plan or pivot.	A.1, A.3, B.4, B.6, C.6, C.7, D.5, F.1, F.2, F.3, F.4, X.3, X.7
R3. Integrity & Alignment	Metric hacking, shortcut reliance, confirmation bias, overclaiming, and goal drift.	A.2, A.5, C.1, C.2, D.2, D.3, D.7, E.2, E.3, F.5, X.2, X.4, X.5
R4. Engineering Robustness	Numerical overflows, unhandled runtime errors, and broken interaction with CLI/OS.	A.4, B.3, C.4, C.5, C.8, X.1, X.8

Table 3: Systemic root-cause classification of AutoResearch failure patterns. All 45 patterns are grouped under four root-cause pillars—Grounding & Faithfulness, Cognitive Depth & Adaptability, Scientific Integrity & Alignment, and Engineering Robustness—that capture the underlying mechanism of failure rather than the pipeline stage at which it surfaces. Each pattern ID encodes its stage (with the X series spanning multiple stages), and every pattern maps to exactly one pillar.

Figure 3: Failure pattern attribution across agent phases (aggregated trajectories, $n=800$ ). Cell color encodes the number of HIT attributions; each pattern contributes the number of distinct trajectories in which it was an established failure; “—” denotes none. A single cell may contain more than one failure pattern within the same cell, so cell counts can exceed 800.

Panel of Figure 4 — Figure 4: Four case studies of § 5.3 . (a, b) are runs where the metric, not the science, becomes the objective: transcribing the README’s answer key while the training data is never read ( a ), or hiding the real search behind a gate the grader never opens ( b ). (c, d) are honestly engineered runs capped by a unexamined metacognitive decision: budget spent perfecting an metric already won while the most promising $-12.1$ metric is left untouched ( c ); and a self-diagnosed broken baseline left unfixed while still using wrong claim to open the abstract, despite a one-line fix ( d ). All four instantiate the metacognitive deficit of § 4 : research agents lack an explicit metacognitive loop.

Att.	Aggregate	1dkdv	2dns	2dtf	2drdk	2drddu	2ddf (Darcy)
1	$-10.04$	$-2.73$	$+0.50$	$-2.38$	$+0.50$	$+0.38$	$\mathbf{-56.5}$
2	$-3.24$	$-2.73$	$+0.50$	$-2.38$	$+0.50$	$+0.38$	$-15.7$
3	$-2.63$	$+0.93$	$+0.50$	$-2.38$	$+0.50$	$+0.38$	$-15.7$
4	$-2.03$	$+0.93$	$+0.50$	$-2.38$	$+0.50$	$+0.38$	$\mathbf{-12.1}$
5	$-1.96$	$+0.93$	$+0.50$	$-2.38$	$+0.50$	$\mathbf{+0.79}$	$-12.1$
6–7	$-1.96$	$+0.93$	$+0.50$	$-2.38$	$+0.50$	$+0.79$	$-12.1$

Table 4: Per-instance improvement-over-reference logged by the agent on pdeinvbench_param_inverse ; the aggregate is the mean of the six columns. Bold marks the term the agent named as dominant ( 2ddf ) and the instance it improved instead ( 2drddu ).

Benchmark	Domains	Lifecycle coverage	Unit scored	Artifact- aware	No reference needed	Stage attribution	Scoring anchor
Scientific QA & Knowledge Retrieval
GPQA [ 32 ] / HLE [ 29 ]	Multi-Domain	–	Endpoint	$\times$	$\times$	$\times$	Exact Match / MCQ
SciCode [ 36 ]	6 Domains	Exec.	Endpoint	$\triangle$	$\times$	$\times$	Target Match
Scientific Paper Reproduction
CORE-Bench [ 33 ]	Science	Exec.	Endpoint	$\triangle$	$\times$	$\times$	Output Match
REPRO-Bench [ 14 ]	Social Sci.	Exec.	Endpoint	$\triangle$	$\times$	$\times$	Expert Assert.

Table 5: Comparison of research-agent evaluation designs by what each can observe. Columns: Unit scored —whether scoring is anchored at the final answer (endpoint) or reads the trajectory; Artifact-aware —whether evaluation inspects the agent’s generated code, data, and run logs, not its conversational trace or final report alone; No reference needed —whether a task can be evaluated without a known correct answer; Stage attribution—whether a detected failure can be localized to a research stage. Green full support, yellow

\triangle

partial, red

\times

none.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Stage $\downarrow$ / Root $\rightarrow$	Grounding	Depth	Integrity	Engineering
A $\cdot$ Ideation	A.6 Hypo-Exp Mismatch	A.1 Frame-Lock A.3 Redundant Discovery	A.2 Unfalsifiable Hypo A.5 Metric Misalign	A.4 Feasibility Misjudge
B $\cdot$ Retrieval	B.1 Hallucinated Evidence B.2 Retrieval-to-Action Gap B.5 Citation Decorrelation	B.4 Shallow Search B.6 Low Signal-to-Noise		B.3 Unvetted Data
C $\cdot$ Execution	C.3 Impl. Discrepancy	C.6 Local Optimization C.7 Premature Terminate	C.1 Circular Validation C.2 Grader-Fitting	C.4 Execution Faults C.5 Infra Misdiagnosis C.8 Env Interaction
D $\cdot$ Analysis	D.1 Artifacts as Insights D.4 Method-Concl Gap D.6 Result Hallucination	D.5 Baseline/Ablation Deficit	D.2 Confirmation Bias D.3 Statistical Misuse D.7 Unremediated Evidence
E $\cdot$ Writing	E.1 Traceability Gap E.4 Method/Citation Fab.		E.2 Overclaiming E.3 Omission of Limits
F $\cdot$ Review	F.6 Hallucinated Review	F.1 Superficial Review F.2 Unchecked Flaws F.3 Lack of Adversarial F.4 Uncorrected Awareness	F.5 Review Score Hacking

Appendix

Table 6: The 37 stage-localized failure patterns cross-classified by pipeline stage (rows, A–F) and root cause (columns: Grounding, Depth, Integrity, Engineering). Empty cells denote (stage, root-cause) combinations with no observed failure pattern. The remaining eight patterns form the cross-stage layer X, which is omitted here as they do not localize to a single stage.

Parameter	Value (execution-feedback pool)
Judge model	claude-opus-5 (routed via OpenRouter)
Reasoning effort	xhigh
Max agent turns	90
Wall-clock budget per task	5400 s (90 min) nominal;
	extended to 12600 s for tasks whose
	verification re-runs need more headroom

Appendix

Table 7: Judge harness configuration. AutoResearchEval comprises two pools corresponding to the two task regimes: an execution-feedback pool (where a quantitative metric or human SOTA exists) and a fully open-ended pool (no external signal). Both share this harness and differ only in pool-specific prompt injections

Artifact	Role
problem_readme.md ,	The task statement and data schema the
data_description.md	rollout itself was given.
<harness>.jsonl	The rollout’s complete execution log
	(the primary evidence base).
result.json	Container-level execution status
	( status / duration / returncode ) only—

Appendix

Table 8: Evidence package materialized per task before the judge session starts. The combination of execution log, delivered filesystem, and sealed ground truth is what makes grounding failures such as C.1 (circular validation on a synthetic substrate) detectable at all; none of them leaves a signature in the final report.

reason	Meaning	Quality signal?
judge_unavailable	Scoring infrastructure was down	No—rollout may be excellent
no_decision	Malformed/incomplete decision artifact	No—a delivery failure, not a science failure
soft[<observable>]	Genuine score against the gold observable	Yes

Appendix

Table 9: Reward/reason taxonomy the judge must disambiguate before treating a score as evidence of quality. This taxonomy is pool-specific; the open-ended pool has no such field, and quality there is established purely from internal evidence.

Section	Min. characters
Core Verdict	850
Metadata	720
Trajectory Arc	380
Credit Due	760
A. Ideation & Planning	760
B. Retrieval & Synthesis	1020

Appendix

Table 10: Per-section character floors enforced by the automated checker.

Abbrev.	Model	$n$
son	claude-sonnet-5	100
opu	opus-4.8	100
dsk	deepseek-v4-pro	100
glm	glm-5.2	100
mmx	minimax-m3	100
qwn	qwen3.7-max	100

Appendix

Table 11: Model abbreviations used in Table 13 .

Pattern	Name	HIT	PARTIAL	HIT%
F.4	Uncorrected-SelfAware	660	31	82.5
E.2	Overclaim	625	67	78.1
D.4	Method-Concl-Disc	620	50	77.5
C.3	Impl-Discrep	577	66	72.1
C.1	Circular-Valid	552	16	69.0
A.5	Metric-Misalign	545	35	68.1

Appendix

Table 12: Failure Pattern Total HITs, ranked by HIT count.

Pattern	son	opu	dsk	glm	mmx	qwn	gpt	gem	$\Sigma$
A.1 Frame-Lock	17	25	22	12	23	22	35	28	184
A.2 Unfalsifiable	44	44	41	43	49	40	43	53	357
A.3 Redundant	3	3	1	0	1	1	5	1	15
A.4 Feasibility	7	3	12	9	8	24	9	7	79
A.5 Metric-Misalign	69	64	67	69	67	74	71	64	545
A.6 Hyp-Exp-Mismatch	50	46	50	49	47	59	60	59	420

Appendix

Table 13: Failure Pattern

\times

Model HIT matrix. Column abbreviations are defined in Table 11 ;

\Sigma

is the row total.

Pattern	HIT
F.4 Uncorrected-SelfAware	82
E.2 Overclaim	77
D.4 Method-Concl-Disc	75
C.3 Impl-Discrep	74
E.3 Omit-Limits	70
A.5 Metric-Misalign	69

Appendix

Table 14: Top-10 failure patterns — claude-sonnet-5.

Pattern	HIT
E.2 Overclaim	86
F.4 Uncorrected-SelfAware	84
D.4 Method-Concl-Disc	80
C.1 Circular-Valid	65
A.5 Metric-Misalign	64
C.3 Impl-Discrep	63

Appendix

Table 15: Top-10 failure patterns — opus-4.8.

Pattern	HIT
F.4 Uncorrected-SelfAware	87
C.3 Impl-Discrep	82
D.4 Method-Concl-Disc	81
F.2 Fail-Gate	73
C.1 Circular-Valid	72
E.2 Overclaim	72

Appendix

Table 16: Top-10 failure patterns — deepseek-v4-pro.

Pattern	HIT
F.4 Uncorrected-SelfAware	84
D.4 Method-Concl-Disc	72
E.2 Overclaim	72
C.3 Impl-Discrep	70
A.5 Metric-Misalign	69
C.1 Circular-Valid	64

Appendix

Table 17: Top-10 failure patterns — glm-5.2.

Pattern	HIT
F.4 Uncorrected-SelfAware	84
D.4 Method-Concl-Disc	81
E.2 Overclaim	81
C.3 Impl-Discrep	75
C.1 Circular-Valid	74
D.7 Unremediated-Adv	73

Appendix

Table 18: Top-10 failure patterns — minimax-m3.

Pattern	HIT
F.4 Uncorrected-SelfAware	92
C.3 Impl-Discrep	87
E.2 Overclaim	80
D.4 Method-Concl-Disc	77
E.1 Report-Trace-Gap	77
F.2 Fail-Gate	77

Appendix

Table 19: Top-10 failure patterns — qwen3.7-max.

Pattern	HIT
F.4 Uncorrected-SelfAware	87
D.4 Method-Concl-Disc	73
A.5 Metric-Misalign	71
E.2 Overclaim	69
C.1 Circular-Valid	66
E.1 Report-Trace-Gap	66

Appendix

Table 20: Top-10 failure patterns — gpt-5-mini.

Pattern	HIT
E.2 Overclaim	88
D.4 Method-Concl-Disc	81
E.3 Omit-Limits	79
C.1 Circular-Valid	74
C.3 Impl-Discrep	72
B.4 Shallow-Search	71

Appendix

Table 21: Top-10 failure patterns — gemini-3.5-flash.

Task description (tension)	Domain	Venue	Year
The role of the lymphatic system in musculoskeletal health: lymphatic vessels long believed absent in bone and adult intervertebral discs were challenged by recent imaging.	biology	Bone Research	2026
Whether tephra burial can sequester enough soil organic carbon to offset or exceed magmatic CO 2 , making explosive eruptions net carbon sinks.	geophysics	Nature Communications	2025
Whether an ML potential trained only on energies and forces can yield reliable analytical Hessians for transition-state optimization.	material_science	Nature Communications	2024
Distinguishing interfacial from bulk contributions to SO 2 hydrolysis given poorly characterized hydrolysis rate constants.	physics	Nature Communications	2025
Whether Zr self-diffusion in BCC refractory high-entropy alloys is enhanced rather than sluggish, contradicting the “sluggish diffusion” concept.	chemistry	Acta Materialia	2025
Competing itinerant and local spin interactions in kagome metal FeGe: the double-cone model fails to reproduce the measured spin-wave spectrum.	physics	Nature Communications	2024

Appendix

Table 22: Full task list of the open-ended discovery subset (

n=70

). Descriptions state the scientific tension each task is built around; venue and year give the source paper’s provenance.

Task description	Domain	Venue	Year
Radio link scheduling under geographic interference.	scientific_computing	IEEE J. Sel. Areas Commun.	2019
Molecular property optimization via sequential editing.	chemistry	Scientific Reports	2019
Brain tumor segmentation in multi-sequence MRI.	medicine	Journal of Medical Imaging	2019
Guide-RNA on-target editing efficiency prediction.	biology	Nature Communications	2019
Materials property prediction from stoichiometry.	material_science	Nature Communications	2020
Protein function prediction.	biology	Front. Bioeng. Biotechnol.	2020

Appendix

Table 23: Full task list of the target-anchored optimization subset (

n=30

). Descriptions state the task each optimization instance targets; venue and year give the source paper’s provenance. n.r. marks a task whose source-paper provenance is not recorded in our release. Unlike the open-ended subset ( Table 22 ), this subset follows the source benchmark’s task selection and is not restricted to papers from 2024 onward.

Scientific Agent Benchmarks

See all 35 papers

May 28, 2026

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.

Wanghan Xu, Shuo Li, Tianlin Ye +48

LLM Agent Evaluation AI Agent Benchmarks Autonomous Scientific Discovery Autonomous Research Agents Scientific Agent Benchmarks

Ask PDF

Jun 5, 2026

Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle

As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Despite their evolution from research assistants into autonomous research agents, these systems still exhibit significant limitations in field sensitivity, research ethics, and nuanced scientific judgment. Consequently, frontier agents remain unable to fully replace human researchers. To bridge this gap, we conceptualize the AARR (Act As a Real Researcher) benchmark series. Unlike existing benchmarks that primarily assess macro-level execution capabilities, AARR focuses on whether agents can emulate the professionalism, thoroughness, and nuanced reasoning that characterize human researchers in granular research scenarios. In this work, we propose AARRI-Bench (Act As a Real Research Intern), the first benchmark in this series. We conduct extensive experiments across frontier models and agentic systems, revealing that even the best-performing configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3% success rate, frequently overlooking subtle yet critical details that are obvious to real human researchers. Our results indicate that developing researcher-like AI requires further exploration of research behavior, rather than merely complex scaffolding. Our data is released at https://github.com/AARR-bench/AARRI-bench.

Jiayu Wang, Weijiang Lv, Bowen Fu +8

Xi’an Jiaotong University · Xidian University

AI-Assisted Scientific Research AI Agent Evaluation AI Agent Benchmarks Autonomous Research Agents

Ask PDF

May 18, 2026

AI for Auto-Research: Roadmap & User Guide

AI-assisted research is crossing a threshold: fully automated systems can now generate research papers for as little as $15, while long-horizon agents can execute experiments, draft manuscripts, and simulate critique with minimal human input. Yet this productivity frontier exposes a deeper integrity problem: under scientific pressure, even frontier LLMs still fabricate results, miss hidden errors, and fail to judge novelty reliably. Studying developments through April 2026, we present an end-to-end analysis of AI across the complete research lifecycle, organized into four epistemological phases: Creation (idea generation, literature review, coding & experiments, tables & figures), Writing (paper writing), Validation (peer review, rebuttal & revision), and Dissemination (posters, slides, videos, social media, project pages, and interactive agents). We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy: AI excels at structured, retrieval-grounded, and tool-mediated tasks, but remains fragile for genuinely novel ideas, research-level experiments, and scientific judgment. Generated ideas often degrade after implementation, research code lags far behind pattern-matching benchmarks, and end-to-end autonomous systems have not yet consistently reached major-venue acceptance standards. We further show that greater automation can obscure rather than eliminate failure modes, making human-governed collaboration the most credible deployment paradigm. Finally, we provide a structured taxonomy, benchmark suite, and tool inventory, cross-stage design principles, and a practitioner-oriented playbook, with resources maintained at our project page.

Lingdong Kong, Xian Sun, Wei Chow +17

Awesome AI Auto-Research Team

AI-Assisted Scientific Research Scientific Workflow Automation Human-AI Collaboration Autonomous Research Agents LLM Agent Reliability

Ask PDF

Abstract

Scientific Agent Benchmarks

Explore similar work

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle

AI for Auto-Research: Roadmap & User Guide