ScienceClaw: Benchmarking Continual Self-Evolution of AI-for-Science Agents Across the Natural and Social Sciences

Mingda Zhang; Wenjin Liu; Tiesunlong Shen; Zikai Xiao; Zhenghong Lin; Qing Xu; Erik Cambria; Xiaoying Tang

cs.AIOct 6, 2026

ScienceClaw: Benchmarking Continual Self-Evolution of AI-for-Science Agents Across the Natural and Social Sciences

Authors: Mingda Zhang, Wenjin Liu, Tiesunlong Shen, Zikai Xiao, Zhenghong Lin, Qing Xu, Erik Cambria, Xiaoying Tang, +1 more

Abstract

Large language model agents are accelerating scientific automation, yet verified executions rarely become persistent program-level improvements, and existing evaluations do not examine this process across sequential tasks in both the natural and social sciences. We formalize ScienceClaw as fixed-parameter program self-evolution that unifies task solving, scientific verification, and program updates. ScienceClaw-Eval spans 23 disciplines and measures scientific correctness, evolutionary gain, retention, cross-dataset transfer, and evolution cost through sequential streams and independent reset evaluation. Our framework repairs executable workflows through multi-turn interaction, converts re-execution-verified failure--success trajectories into linked Skill and Operator candidates, and retains an update only when source-task replay reproduces the repair and independent scientific tasks improve. Code is available at https://github.com/beita6969/ScienceClaw.

Explore similar work

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
May 28, 2026Wanghan Xu, Shuo Li, Tianlin Ye +48LLM Agent Evaluation AI Agent Benchmarks
EvoMaster: A Foundational Evolving Agent Framework for Agentic Science at Scale
Date pendingXinyu Zhu, Yuzhu Cai, Zexi Liu +20Continual Learning for LLM Agents AI Agents for Scientific Discovery
School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China. · SciLand, Shanghai, China. · DP Technology, Beijing, China.
Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Jun 10, 2026Tianyu Liu, Allen Xin Wang, Antonia Panescu +30AI for Science AI Agent Evaluation
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7

May 28, 2026, cs.LGJ/K move · Enter open · S save

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

Wanghan Xu, Shuo Li, Tianlin Ye, Qinglong Cao+47

LLM Agent Evaluation AI Agent Benchmarks Autonomous Scientific Discovery Autonomous Research Agents

Open paper AskPDF arXiv

AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.

Abstract

Explore similar work

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

EvoMaster: A Foundational Evolving Agent Framework for Agentic Science at Scale

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales