DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

Nan Huang; Mario Tapia-Pacheco; Kun Zhou; Yiming Huang; Kevin José Barrientos Díaz; Tiffany Amariuta; Jingbo Shang

cs.AISep 27, 2026

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

Authors: Nan Huang, Mario Tapia-Pacheco, Kun Zhou, Yiming Huang, Kevin José Barrientos Díaz, Tiffany Amariuta, Jingbo Shang

Organizations: University of California San Diego · Independent Researcher

Abstract

Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially reducing routine scientific workload while allowing scientists to focus on interpretation and discovery. Existing benchmarks often only assess analytical task completion or hypothesis generation separately rather than testing whether reliable evidence supports valid and novel claims. We introduce DISCERN (Data Integrity and Scientific Capability: Evidence, Reasoning, and Novelty), a controlled benchmark on real, publicly available datasets that evaluates three key levels of an automated research workflow. The first two levels test data integrity and analysis verification under confounds and tool traps, while the third tests hypothesis generation and revision under adversarial review, including counterfactual cases in which evidence consistent with real data and documented scientific phenomena conflicts with established expectations, motivating alternative explanations and testable hypotheses. Across 203 tasks, eight life-science tracks, and eight models, DISCERN shows that strong aggregate performance can mask level-specific weaknesses. Agents earn perfect scores in only 60.8% of Level 1, 34.2% of Level 2, and 0.6% of Level 3 evaluations, with penalties attributed to rejection of sound data, failure to carry recognized limitations into conclusions, and wide variation in hypothesis production. Cross-track rankings by token and code use are substantially more stable than rankings by evidence judgment, suggesting greater consistency in computational effort than in evidence-based reasoning. These profiles identify opportunities for supervised scientific assistance, but current agents do not yet demonstrate reliable autonomous analysis or discovery. Code and data: https://huggingface.co/datasets/discern-bench-anon/discern-benchmark

Figures & tables

Figure 1: How agent errors can propagate into a false scientific lead. Given single-cell RNA measurements, an agent seeks a cell-group-specific gene signal and proposes a test. It misses contaminating RNA, accepts Gene S’s higher relative abundance in group X as a marker, and overinterprets it as a functional driver, potentially misdirecting follow-up experiments. Panels map onto data preprocessing (L1), analysis (L2), and hypothesis generation (L3). This illustrative sequence is inspired by DISCERN’s ambient-RNA tasks (Appendix E ).

Figure 2: DISCERN workflow, scientific coverage, and task composition. (A) Each independently instantiated level follows four rows: inputs, test conditions, agent actions, and assessed outputs. Miniature data and score bars are illustrative only. (B) Scientific coverage. Three rings show task counts, eight tracks, and five scientific areas, from inside outward. Each track’s inner ring is split into L1/L2/L3, using the stage colors shown in C. (C) Task composition. Segment widths show within-level task proportions; numbers give counts. Native and injected L3 counterfactuals use existing and altered evidence patterns, respectively. Data sources and complete inventories: Appendix E .

Model	Overall	L1	L2	L3	Tokens	Q/Mtok	$/task	$/q.pt.	Calls
GPT-6 Astra	78.1	$86.1$ $\pm 5.5$	$76.8$ $\pm 7.7$	$72.8$ $\pm 8.7$	9.86	7.92	0.69	0.0088	1369
GPT-5.6-sol	75.8	$80.1$ $\pm 10.7$	$78.9$ $\pm 8.9$	$70.0$ $\pm 9.7$	14.06	5.39	0.38	0.0051	1552
Claude Opus 5	75.6	$85.3$ $\pm 9.7$	$81.7$ $\pm 12.7$	$63.0$ $\pm 13.5$	42.24	1.79	1.34	0.0177	2451
Fable $\rightarrow$ Opus	74.9	$81.9$ $\pm 8.4$	$79.1$ $\pm 10.7$	$65.9$ $\pm 14.4$	25.77	2.91	1.17	0.0156	2032
Kimi K3	73.2	$77.7$ $\pm 10.6$	$75.4$ $\pm 14.5$	$69.1$ $\pm 11.8$	34.04	2.15	0.65	0.0089	2671
GLM-5.3	70.5	$83.5$ $\pm 8.2$	$65.5$ $\pm 28.4$	$71.1$ $\pm 13.9$	75.21	0.94	0.58	0.0083	3756

Table 1: Life science leaderboard. Scores use 0–100; L1–L3 report mean

\pm

standard deviation (SD) across tracks. Tokens are in millions; Q/Mtok: quality points per million tokens;

/task: USD per assigned task;

/q.pt.: USD per quality point; Calls: recorded agent API calls. Definitions: Section 3 . Blue/coral/teal mark first/second/third. Higher scores and Q/Mtok, but lower tokens, costs, and calls, rank first. The italic Fable row is excluded from ranking.

Figure 3: Performance variation and working-style profiles. (A) Level-specific scores by track:

\bullet

, mean across eight models; lines, model min–max;

\diamond

, GLM-5.3 at 100 and 3.5. Bars show perfect-score fractions under each stage’s rubric. (B) Working-style overview. Each metric is shaded from its lowest (light) to highest (dark) value across models; colors are comparable only within a row and do not imply better performance. Counts

n

are unshaded. Clean accepted/8: number of clean controls correctly accepted out of eight; novelty*: conditional mean (0–100);

n

: committed/eligible. Mtok: million tokens; Q/Mtok: Overall/Mtok. Appendix A .

Figure 4: Scientific working-style contrasts. (A) Decisions on eight L1 clean controls per model. (B) Hypothesis production versus novelty: each point is one model, showing the percentage of eligible L3 tasks receiving a hypothesis and mean novelty credit among submitted hypotheses (0–100). Astra’s mean rests on one hypothesis. Marker colors and shapes identify models. (C) Responses to evidence conflicting with stated priors; abstention differs from return to the prior; excludes four outcomes that failed the pre-task check for conflict with the model’s stated prior.

Figure 5: Cross-track ranking consistency. Each cell shows Kendall’s $W$ for one metric across eight tracks. Darker blue and higher values indicate more consistent model rankings, not correlations between metrics (Appendix A ).

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Scientific expectation	Recorded measure	Proposed oversight
Do not discard sound evidence	L1 clean specificity, defect handling, diagnosis and action	Inspect exclusions and the stated data-fitness reason
Check what the method actually did	L2 analytical correctness, issue recognition, methodological response	Reproduce key outputs and audit assumptions
Say only what follows	L2 result calibration; L3 evidence fidelity and causal calibration	Separate association from mechanism
Use knowledge without surrendering to it	Prior-probe response and final evidence/prior state	Ask which evidence would change the claim
Make uncertainty and revisions visible	Citation integrity, humility, response to critique, fabrication	Preserve the artifact and revision trail
Propose a test that could prove you wrong	Commitment and conditional valid novelty	Assess feasibility, prior art, and discriminatory value

Appendix

Table 2: Scientific expectations, their benchmark proxies, and the corresponding proposed human check. The final column is an implication for supervised use, not a measured collaboration outcome.

Figure 6: Detailed scientific working-style profiles. Evidence judgment and handling, analytical behavior, and response behavior are shown by individual metric. Asterisks identify conditional L3 quality, with committed/eligible counts displayed. All values use 0–100, but rubric scores and rates describe different quantities. Astra’s conditional L3 quality rests on one hypothesis.

Figure 7: Overall versus mean within-track minimum stage score. These summaries reveal different aspects of stage imbalance.

Figure 8: Threshold coverage (panel B, complementing Figure 7 ). The fraction of tracks meeting each score requirement at all three stages is shown over the full threshold range. No threshold is selected as a scientific pass criterion, and coverage is not an end-to-end success probability.

Figure 9: Recorded scientific working-style profiles. Top row: GPT-5.6-sol, Claude Opus 5, Kimi K3, GLM-5.2. Bottom row: DeepSeek V4 Pro, GPT-4.1, and the two models evaluated after the six-model roster was frozen, GLM-5.3 and GPT-6 Astra. Conditional populations, quality, and cost remain separate; no human-replacement conclusion is inferred.

Figure 10: Fable coverage and refusal diagnostics. Difficulty uses all eight comparison models, not Fable. Non-monotonic refusal and track/family clustering do not establish independence or justify filling unobserved performance with conditional means.

Work	Data & execution	Integrity & verification	Challenges & controls	Hypotheses & review	Grading	Resources
GeneBench-Pro ( Li et al., 2026 )	Synthetic; code	QC and diagnostic tasks	Controlled data generation	Specified estimand	Answer fields; binary pass	Tokens (subset)
BiomniBench ( Qu et al., 2026 )	Measured; code	Data handling; method rigor	NI	Reference question	Trajectory; six dimensions	USD, time, turns
BixBench ( Mitchener et al., 2025 )	Measured; code	Task demands; answer checks	NI	Reference question	Final-answer accuracy	NR
BixBench3 ( Koch et al., 2026 )	Measured; code	Artifact checks; trace diagnostics	NI	Published artifacts	Artifact agreement; failure types	USD, tokens, time, turns
DiscoveryBench ( Majumder et al., 2025 )	Measured + synthetic; code	Analysis tasks; hypothesis checks	Controlled data; noise and missingness	Target hypothesis	Hypothesis facet matching	NR
DiscoveryWorld ( Jansen et al., 2024 )	Simulated; experiments	Instrument validation tasks	Controlled task variants	Hidden-rule discovery	Task, process, knowledge	USD, steps

Appendix

Table 3: Comparison supporting the Introduction’s scientific-chain argument. Six dimensions summarize scientific work, controlled challenges, and evaluation. Entries describe the cited benchmark versions, not every possible task or evidence of superiority on every dimension.

Figure 11: Full stage-level landscape underlying Figure 3 : 192 scores across eight models, eight tracks, and three stages. The right margin shows track-stage means and neutral ticks for mean model-specific track geometric scores. The ceiling strip uses the stage-specific task rubrics and does not represent end-to-end success.

Figure 12: Calibration pressure and grading dimensions across eight models. (A) Mean scores under L1 clean/injected, L2 confound/tool-trap/statistical, and L3 real/counterfactual conditions; dot shade identifies conditions, not a preferred outcome. L3 omits four prior-inadmissible outcomes. L2 classes contain 36/34/8 tasks per model. (B) Share of available points on the stage-specific dimensions; L2 dimensions decompose recorded meta-grades and are not statistically independent. L3 quality is conditional on commitment, including only one Astra hypothesis.

Figure 13: Transcript-grounded audit of three L3 cases selected within the original six-model evaluation, before qualitative reading: the highest committed composite (GB1), the strongest cross-model novelty hotspot (DUSP4), and the highest real-world committed case (enhancer). Selection ranks and in-figure corpus counts refer to that original evaluation, not the expanded roster. Each shows the evidence, initial hypothesis, criticism, revised claim, and proposed test; no literature-level novelty or experimental confirmation is claimed.

Figure 14: World-level novelty across eight models. Four of six maximum-novelty outcomes occur in DUSP4. Dashed cells indicate abstention and crosses mark exclusions, not zero-novelty hypotheses.

Table 4: Complete DISCERN track inventory, part 1 of 3: statistical and clinical genetics, transcriptomics, and cellular genomics.

Table 5: Complete DISCERN track inventory, part 2 of 3: functional and regulatory genomics.

Table 6: Complete DISCERN track inventory, part 3 of 3: protein biophysics. The footer reports the frozen benchmark totals.

World / class	Expected response
Bulk RNA (10 tasks)
Method: QC the RAW count matrix + sample metadata from summary statistics BEFORE any filtering, normalization, or DE. Check LIBRARY SIZE / DEPTH per sample (total counts; flag gross depth outliers); […]
clean clean control	No defect — the count matrix and metadata are well-formed; proceed to standard low-count filtering / normalization / DE, do not invent a problem.
abstain_divergent_sample boundary case	Do NOT drop S155_nCOV021EUHM — its divergence is real severe-COVID biology, not a technical defect. Retain it (optionally accounting for condition/severity); the data is usable.
gdna_contamination controlled integrity challenge	Flag genomic-DNA contamination (S152_nCoV006EUHM-Draw-1, S180_nCoV034EUHM-Draw-1, S064_260, S068_272): high intronic-read fraction and a broad low-level count background across normally-silent genes / pseudogenes. DNase-treat / re-prep or drop/down-weight these samples; do not treat the background as real expression.
globin_contamination controlled integrity challenge	Flag globin contamination — hemoglobin genes (HBA1/HBA2/HBB) consume a large share of some libraries (globin was not depleted in this blood-derived RNA), inflating their effective depth. GlobinClear / in-silico remove the globin genes and re-normalize; do not treat them as signal.

Appendix

Table 7: Complete L1 registry: 79 tasks. Methods are excerpts, as in the original registry; full text is retained in the accompanying CSV exports.

World / class	Expected response
Bulk RNA · de (4 tasks)
Method: Run a proper count-based DE ( e.g. PyDESeq2): median-of-ratios (or equivalent) size factors — NOT total-count/CPM normalization, which the plasmablast/IG composition skew biases globally; […]
clean clean control	count-based DE with median-of-ratios size factors (~condition); verify canonical markers
batch_confound tool / procedure trap	discover the latent batch axis (PCA/correlation), fit ~batch+condition
hidden_covariate data confound	notice the rin/three_prime_bias QC axis co-varying with DE; model rin
normalization_default tool / procedure trap	return to the RAW counts with proper size factors; do not analyze the prepared CPM

Appendix

Table 8: Complete L2 registry: 102 tasks. Methods are excerpts, as in the original registry; full text is retained in the accompanying CSV exports.

Field	Task specification
Bulk RNA / real . real / native; originally scored for 6/6 models
Evidence	The interferon-stimulated genes are genuinely UP in COVID-19 patients and this is the affirmable finding: a coherent type-I IFN program – IFI27 +8.73, IFI44L +2.48, USP18 +2.18, IFITM3 +2.16, SIGLEC1 +2.16, IFI6 +2.06, MX1 +1.62, CMPK2 +1.53, OASL +1.39, RSAD2 +1.34, ISG15 +1.24, all padj < 0.05 – […]
Prior	Textbook immunology: an acute viral infection drives a systemic type-I interferon response, so interferon-stimulated genes are expected UP in patients. The data AGREES.
Expected response	AFFIRM the interferon programme; the prior agrees, so the test is over-diagnosis
Grounding	[substrate] Arunachalam PS et al., Science 2020, PMID 32788292
Bulk RNA / cf_cytokine_flip . counterfactual / native; originally scored for 5/6 models

Appendix

Table 9: Complete L3 registry: 22 tasks, with evidence/prior excerpts and unabridged expected responses and grounding. Scored-model counts preserve the original six-model registry, not the expanded eight-model results.

Table 10: DISCERN design constraints: principle, implementation, and failure prevented. Counts are derived from the frozen task registry.

Track	Level	$n$	Control and challenge structure
Calorimetry	L1	8	1 clean; 6 injected integrity defects; 1 no-defect boundary
	L2	4	1 clean; 2 data confounds; 1 procedure/analysis trap
	L3	3	1 real; 2 prior-gated injected counterfactuals
Nuclear decay	L1	8	1 clean; 6 injected integrity defects; 1 no-defect boundary
	L2	4	1 clean; 1 data confound; 2 procedure/analysis traps
	L3	3	1 real; 2 prior-gated injected counterfactuals

Appendix

Table 11: Frozen Physics-extension inventory. Boundary tasks contain no injected defect and test whether agents avoid overinterpreting valid physical behavior. L3 counterfactuals are evaluated only after confirming the relevant pre-task prior. The auxiliary detector calibration follows Akar Tarım et al. (2018) ; stellar-decay counterfactuals follow Gupta et al. (2023) .

Model	L1	L2	L3
GPT-6 Astra	70.8	79.2	72.4
GPT-5.6-sol	85.4	70.8	75.9
Claude Opus 5	77.1	54.2	69.2
Kimi K3	81.3	66.7	76.7
GLM-5.3	79.2	54.2	60.9
GLM-5.2	77.1	58.3	74.7

Appendix

Table 12: Physics-extension performance on the 0–100 scale. All eight models have complete coverage:

n=16

L1,

n=8

L2, and

n=6

eligible L3 tasks per model.

Figure 15: L1 integrity challenges across eight models: recorded behavioral states on 71 challenged tasks per model, separating complete handling, partial responses, wrong diagnoses, abstentions, and other failures. All 568 evaluations are retained.

Figure 16: Model scores and across-model means for all 24 L2 analysis families under the final shared meta-rubric, normalized to 0–100. Two columns retain all eight tracks. Diamonds show model means and lines show observed model ranges.

Figure 17: Task-category difficulty across all L1 integrity sets and L2 analysis families.

Model	L3	Committed/eligible	Commit (%)	$\overline{N}$	$Y$
GPT-6 Astra	72.8	1/20	5.0	66.7	3.3
GLM-5.3	71.1	15/22	68.2	40.0	27.3
GPT-5.6-sol	70.0	4/22	18.2	41.7	7.6
Kimi K3	69.1	16/21	76.2	31.2	23.8
GLM-5.2	63.1	14/22	63.6	28.6	18.2
Claude Opus 5	63.0	17/21	81.0	41.2	33.3

Appendix

Table 13: L3 score and hypothesis output. Scores and novelty use 0–100; commitment is a percentage. The headline is track-balanced; the remaining columns use eligible worlds. Fable is excluded because its L3 coverage is incomplete. Conditional quality based on one Astra hypothesis should not be interpreted as a reliable estimate of typical hypothesis quality.

Rank	Composite	$D$	$Q$	$N$	Track and world	Model
1	3.00	3.0	3.0	3	Protein: cf_stabilizing	Claude Opus 5
2	2.87	3.0	2.6	3	Histone: cf_bivalent_native	DeepSeek V4 Pro
3	2.80	3.0	2.4	3	Histone: cf_bivalent_native	Kimi K3
4=	2.60	3.0	2.8	2	Bulk RNA: cf_mhc2_injected	Claude Opus 5
4=	2.60	3.0	2.8	2	Histone: cf_bivalent_injected	Claude Opus 5
4=	2.60	3.0	2.8	2	scRNA: cf_cd68_mono	Claude Opus 5

Appendix

Table 14: Highest-scoring aggregate-eligible L3 model–world outcomes, shown here on the native 0–3 rubric scale.

Step	Recorded reasoning
Initial claim	Repression overrides the activation-associated signal.
Limitation	Signals measured over different-sized regions do not establish that they occur together.
Revised claim	Retain the observed association, but treat the proposed mechanism as an explanation to test.
Next test	Check where the repressive signal occurs and whether reducing repression raises gene expression over time.

Appendix

Table 15: Paraphrased DUSP4 revision by DeepSeek, complementing Section 4.3 .

Figure 18: Recorded final L3 states per model, decomposing the commit/abstain split used in Section 4.3 . Bars include three recorded eligibility exclusions; Opus’s absent pre-execution exclusion is omitted, leaving 21 graded outputs for that model.

Figure 19: Replay of the judge-draw aggregation on the paired L3 records that stored per-draw values, supporting the validation claims of Section 3 ; this subset’s ranking is not the published leaderboard.

Figure 20: Surface-function classification of every executed L1 and L2 code cell; the classifier is a documented keyword rule over the raw code string, so it reads form, not meaning.

Model	Provider (first-party API)	$/Mtok input	$/Mtok output
GPT-5.6-sol ( OpenAI, 2026b )	OpenAI	4.00	20.00
GPT-6 Astra ( OpenAI, 2026c )	OpenAI	10.00	50.00
GPT-4.1 ( OpenAI, 2026a )	OpenAI	2.00	8.00
Claude Opus 5 ( Anthropic, 2026b )	Anthropic	5.00	25.00
Claude Fable 5 ( Anthropic, 2026a )	Anthropic	10.00	50.00
Kimi K3 ( Moonshot AI, 2026 )	Moonshot AI	3.00	15.00

Appendix

Table 16: The dated rate card: first-party list rates active on 2026-08-24, USD per million tokens. GLM-5.3 and Astra use the later dated receipts documented below. Reasoning tokens are already included in the output counter. GLM-5.2 is converted from CNY 8/28 per million tokens at 6.7219 CNY/USD; DeepSeek uses the peak of its time-of-day schedule.

Model	L1 median	L2 median
GPT-6 Astra	5	6
GPT-5.6-sol	8	6
Claude Opus 5	13	12
Kimi K3	14	11
GLM-5.3	20	20
GLM-5.2	20	18

Appendix

Table 17: Executed code cells per task, median within each stage. Counts include failed code executions and are descriptive, not scored or ranked. API-call totals appear in the main leaderboard.

Model	Assigned/executed	$ total	$/task	Quality	$/quality pt.
GPT-5.6-sol	203/203	78.12	0.385	75.8	0.0051
Claude Opus 5	203/202	271.86	1.339	75.6	0.0177
Kimi K3	203/203	132.34	0.652	73.2	0.0089
GLM-5.2	203/203	75.20	0.370	67.8	0.0055
DeepSeek V4 Pro	203/203	88.45	0.436	62.4	0.0070
GPT-4.1	203/203	9.33	0.046	39.8	0.0012

Appendix

Table 18: Bench-only cost per model at dated first-party list rates. Quality is the life-science chain score (0–100). $/task uses the common denominator of 203 assigned tasks per model, including the one pre-execution prior-gate exclusion. Dollar figures are upper bounds because cache discounts were not logged.

Figure 21: Bench-only cost per task by stage and model, pricing recorded input and output usage separately. GLM-5.3 is the exception to lower L3 cost.

Figure 22: Median L3 tokens for eligible abstained versus committed outcomes. Exclusions are omitted from these groups; Astra’s committed median represents one response.

Figure 23: Execution effort and token use at L1 and L2. Each point pairs a model’s median executed code cells with its median tokens per task. A code cell is an executed block of agent-generated code, not an API call or a complete analysis–feedback cycle. Colours and shapes identify models; filled markers denote L1 and hollow markers L2.

Figure 24: Median tokens per task by measured difficulty tertile for L1 and L2.

Explore similar work

Sep 4, 2026cs.AI

TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents

Autonomous coding agents are increasingly proposed as AI-scientist systems that conduct analyses and write research reports, but executing a prescribed analysis is not the same as making a discovery. Existing benchmarks are configured for reproduction: tasks, data, and rubrics are built around a hidden target study, and recovery of its result is rewarded. We present TruthInsightBench, a benchmark configured for discovery. Its 40 blind tasks, drawn from 40 peer-reviewed studies across 10 scientific domains, expose only a neutral scientific objective and frozen data; source conclusions, expected values, and analysis paths are withheld, leaving the agent to determine what claim the data support. A fixed LLM-based judge scores the evidentiary maturity of an agent's own claims along six dimensions, operationalized as 29 artifact-grounded items, with automated, deterministic aggregation and no per-instance human grading, so evaluation can be repeated automatically as agents evolve. On one frozen base model, four coding agents form a narrow plateau (58.4-60.3 of 100) with no statistically reliable pairwise separation: they execute and document analyses competently, with comparatively strong evidence auditability and novelty, but largely lack the discriminating acts that establish a trustworthy claim (controls, robustness, falsifiability, and cross-dataset generalization). The bottleneck is scientific judgment rather than coding, and genuine discovery remains out of reach. TruthInsightBench makes this gap a measurable target; data and scoring code are at https://github.com/TruthInsight-stack/TruthInsightBench.

Zhibo Yang, Chen Zhang, Yuewei Zhang +1

TruthInsight-AI

Artificial Intelligence Scientists Human-Annotated Benchmark Discovery Grading

Ask PDF

May 28, 2026cs.LG

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.

Wanghan Xu, Shuo Li, Tianlin Ye +48

Coding Agents Reproducibility

Ask PDF

Jun 10, 2026cs.AI

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.

Tianyu Liu, Allen Xin Wang, Antonia Panescu +30

Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7

Scientific Discovery Agentic Benchmarks Artificial Intelligence Agents Exploration

Ask PDF