Before AI can be trusted to discover new science, it should be able to reproduce what science already knows. Reproduction provides a rigorous checkpoint for an AI scientist: the target claim is known, yet recovering it still requires translating an incomplete scientific description into an executable experiment and producing evidence that can be independently verified. Existing research-reproduction benchmarks largely focus on AI and machine learning, where experiments share relatively homogeneous software stacks. Extending this capability across scientific disciplines introduces a different challenge: agents must operate domain-specific scientific software, make physical and numerical choices, diagnose long-running computations, and distinguish valid evidence from merely plausible outputs. We introduce PAPERBENCHX, a benchmark of 93 expert-constructed paper-reproduction tasks spanning 12 research areas across electromagnetics, photonics, chemistry, materials science, biology, and robotics. Each task provides the source paper, a public task specification, and a domain-native scientific environment involving specialized scientific software. Agents must reconstruct a key experiment, execute and diagnose it, and deliver a runnable workflow that supports the target scientific claim. To evaluate scientific validity rather than surface-level agreement, we replay every submission from scratch and score only regenerated evidence against expert-validated criteria. Across 10 frontier agent configurations, the best configuration fully reproduces only 14% tasks under the 90% threshold. Meanwhile, mean rubric scores range from 27.53% to 59.08%, indicating that substantial partial progress rarely translates into complete scientific reproduction. These results reveal a large gap between paper understanding and reliably producing reproducible scientific evidence.
Figures & tables
Figure 1: Frontier agents on PaperBenchX . Bars show full scores ( >90% ); boxes show partial scores. GPT-6-Astra and GPT-5.6-sol use Codex; all others use Claude Code.
Figure 2: From task package to replay-grounded score. An agent receives the paper, task specification, environment, and starter submission; interacts with domain-native scientific software; and returns a runnable package. Clean replay regenerates the evidence used to score the scientific rubric. The displayed leaves and scores are illustrative.
Benchmark
Research scope
Primary evaluation target
Paper-level reproduction
Domain-native scientific execution
Replay-grounded scientific verification
Machine-learning research and paper-reproduction benchmarks
MLAgentBench ( Huang et al., 2023 )
Machine learning
ML experimentation tasks
✗
✗
✗
RE-Bench ( Wijk et al., 2024 )
Machine learning
Research-engineering tasks
✗
✗
✗
PaperBench ( Starace et al., 2025 )
Machine learning
Paper reproduction
✓
✗
✗
SA-Bench ( Hu et al., 2026 )
Machine learning
Paper reproduction
✓
✗
✓
Cross-domain scientific benchmarks
Table 1: Comparison with prior agent benchmarks. Checkmarks indicate that a benchmark explicitly evaluates the named capability.
Figure 3: Coverage and evaluation structure of PaperBenchX . A , The 93 tasks span 12 research areas in six scientific domains. B , Their 93 source papers span 44 publication venues; bar colors correspond to the 12 research areas in A. C , The task rubrics contain 3,168 scorable criteria (median 38 per task); points are tasks and black ticks mark research-area medians.
Figure 4: Task construction and quality control. Experts turn a published claim into a runnable task by defining its scientific scope, provisioning the domain-native environment, and specifying the public task contract. Reference replay, calibration checks, and independent expert review verify executability, fairness, and evidence requirements before the task is admitted to the benchmark.
Overall (%)
Stage score (%)
Average effort
Model
Harness
Partial
Full
Ref.
Modeling
Execution
Validation
Turns
Time (h)
GPT-6-Astra
Codex
59.08
13.98
0.00
72.72
69.35
52.76
348.7
5.96
Fable 5.1
Claude Code
58.17
12.90
11.83
71.00
69.92
51.06
113.0
4.65
Fable 5
Claude Code
57.96
11.83
13.98
72.13
68.99
50.39
112.0
5.09
GPT-5.6-sol
Codex
55.07
9.68
0.00
69.85
71.23
45.29
237.8
3.28
Opus 5
Claude Code
52.52
7.53
0.00
66.99
67.16
47.76
259.2
6.52
Table 2: Model-level results on the shared set of 93 tasks. Partial, Full, Ref., and all stage scores are reported as percentages; metric definitions and outcome handling follow Section 4.1 .
Figure 5: Domain-specific capability profiles. (a) Mean rubric score by model and research area, ordered from hardest to easiest; circles mark Fable refusals scored using the Opus 5 fallback. (b) Domain difficulty, defined as 100 minus the cross-model mean. (c) Overall scores, matching Table 2 . Names are abbreviated.
Figure 6: Stage performance, interaction length, and workflow allocation. (a) Stage-score distributions over 930 outcomes; thick and thin lines denote the interquartile and 10th–90th percentile ranges, and circles and diamonds denote medians and means. (b) Partial score versus the trimmed average turns in Table 2 . (c) Mean tool-mediated time allocation across 200 sampled trajectories (50 per model), with phases coded from timestamped tool actions using fixed rules.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Role
Agent
Evaluator
Paper and addendum
Scientific scope, measurements, and evidence requirements
Table A1: Task components and their visibility during evaluation. Evaluator-only components are included in the public demonstration package after evaluation.
Field
Public contract
Scientific target
Guided-mode resonances of a three-pair multilayer dielectric grating in Meep
Cases
Unetched control plus full-depth air-slot widths of 0.030, 0.050, 0.070, and 0.150 μ m
Required evidence
Native spectra and logs, matched reference/device fluxes, complex reflected amplitudes, resonance fits, independent decay modes, and refinement records
Meep 1.17.1, 8 CPUs, 32 GiB RAM; 7 hours for the agent, 5 hours for native calculation, and 6 hours for replay and review
Failure reporting
Missing, dark, overlapping, or non-identifiable features remain unresolved and cannot be replaced with paper values or invented poles
Appendix
Table A2: Condensed public contract for the GMR demonstration task.
ID
Task
Public demo
01 RF antennas and arrays
1-1
5g-smartphone-eight-element-mimo
1-2
cpw-quad-port-uwb-mimo
yes
1-3
dual-band-wlan-miniaturized-patch
1-4
f-stub-uwb-mimo
1-5
inkjet-flexible-dual-band-cp-patch
Appendix
Table A3: Complete 93-task benchmark inventory used in the reported evaluation.
Area
Task
Source publication and DOI
Radio-frequency antennas and arrays
01-cpw-quad-port-uwb-mimo
CPW Fed Compact UWB 4-Element MIMO Antenna with High Isolation ; 10.3390/s21082688
Radio-frequency metasurfaces and scattering
02-anisotropic-coding-diffusion-metasurface
Broadband diffusion metasurface based on a single anisotropic element and optimized by the Simulated Annealing algorithm ; 10.1038/srep23896
Radio-frequency filters and frequency-selective surfaces
03-dual-passband-angular-stable-fss
A Dual-Passband Frequency Selective Surface with High Angular Stability and Polarization Insensitivity ; 10.3390/mi15060690
Integrated photonic devices
04-swg-anisotropic-phase-shifter-meep
Ultra-broadband nanophotonic phase shifter based on subwavelength metamaterial waveguides ; 10.1364/PRJ.373223
Photonic crystals and topological photonics
05-pt-bragg-unidirectional-invisibility-meep
Unidirectional Invisibility induced by PT-Symmetric Periodic Structures ; 10.1103/PhysRevLett.106.213901
Optical metasurfaces, gratings, and plasmonics
06-gmr-grating-fano-meep
Controlling Fano resonances in multilayer dielectric gratings towards optical bistable devices ; 10.1038/s41598-018-34787-9
Appendix
Table A4: The twelve public demonstration tasks and their source publications.
Figure 7: Representative outputs from domain-native environments. Panels show solver fields and spectra, numerical analysis outputs, and embodied-task scenes produced by scientific workflows in the benchmark.
ID
Execution stack
Agent h
Verifier h
CPU
RAM GiB
Disk GiB
GPU
1-2
Ansys HFSS
24
25
32
128
200
2-1
Ansys HFSS
72
73
32
128
200
3-1
Ansys HFSS
48
49
32
128
300
4-13
Meep
7
6
8
32
50
5-9
Meep
5
5
8
32
50
6-4
Meep
7
6
8
32
50
Appendix
Table A5: Execution stacks and declared resource limits for the twelve public task packages in Table A4
Check family
Perturbation and expected behavior
Contract and parser integrity
Preserve rubric hashes and complete leaf mappings; reject duplicate keys, nonfinite values, and invalid numerical grids while accepting documented equivalent formats.
Native evidence and variant identity
Reject incomplete solver logs, mismatched setup names, explicit non-convergence, and copied variant outputs; accept a shared solve only when the task contract permits it and the variants remain identifiable.
Evidence boundary
Exclude cached outputs and links outside permitted roots; keep private references and evaluator credentials inaccessible; fail closed when network isolation cannot be established.
Completion and provenance
Give no completion credit to an empty submission; require delivery records, input hashes, and preprocessing conventions to agree with the task contract.
Local failure accounting
Corrupt one input, metric, or figure panel and verify that only dependent rubric leaves lose credit.
Interactive-task contract
Use evaluator-owned episode records and protocol identifiers; do not allow candidate summaries to replace required trajectory or visual evidence.
Appendix
Table A6: Regression cases used to check evidence admission and local scoring. Valid controls accompany invalid fixtures so that permissible variation is not mistaken for unsupported evidence.
Figure 8: Task-level completion landscape. (a) Maximum Partial score attained on each of the 93 tasks, ordered by research area. (b) Number of configurations exceeding 90% on the same task. The horizontal line marks the operational full-reproduction threshold; 71 tasks have no configuration above it.
Figure 9: Stage profiles in three trace case studies. Bars report Modeling, Execution, and Validation scores; diamonds report the aggregate task score.
AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research.
AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research. To spur the development of useful agents, we need benchmarks that are challenging, but more crucially, directly correspond to real-world tasks of interest. This paper introduces such a benchmark, designed to measure the accuracy of AI agents in tackling a crucial yet surprisingly challenging aspect of scientific research: computational reproducibility. This task, fundamental to the scientific process, involves reproducing the results of a study using the provided code and data. We introduce CORE-Bench (Computational Reproducibility Agent Benchmark), a benchmark consisting of 270 tasks based on 90 scientific papers across three disciplines (computer science, social science, and medicine). Tasks in CORE-Bench consist of three difficulty levels and include both language-only and vision-language tasks. We provide an evaluation system to measure the accuracy of agents in a fast and parallelizable way, saving days of evaluation time for each run compared to a sequential implementation. We evaluated two baseline agents: the general-purpose AutoGPT and a task-specific agent called CORE-Agent. We tested both variants using two underlying language models: GPT-4o and GPT-4o-mini. The best agent achieved an accuracy of 21% on the hardest task, showing the vast scope for improvement in automating routine scientific tasks. Having agents that can reproduce existing work is a necessary step towards building agents that can conduct novel research and could verify and improve the performance of other research agents. We hope that CORE-Bench can improve the state of reproducibility and spur the development of future research agents.
Reproducibility is essential to scientific progress, yet the growing volume and complexity of scientific publications make exhaustive manual verification increasingly impractical. Although recent advances in large language model (LLM) agents enable automated experiment reproduction, existing evaluations largely focus on final repositories and are typically limited to machine learning (ML). We introduce AgentActionBench, a process-oriented benchmark for evaluating agent-based experiment reproduction across ML and AI4Science domains. Our framework uses an MCP-based Action Recorder to capture agents' behaviour throughout the reproduction process and evaluates the resulting traces with paper-specific rubrics. AgentActionBench contains 150 papers, including 120 ML papers and 30 AI4Science papers. A human-annotated subset covering 10% of the benchmark provides validation data, while model-assisted augmentation expands the full benchmark to more than 10,000 rubric items. Experimental results show that current systems remain limited, with execution as the primary bottleneck. Meanwhile, the strong Pearson and Spearman correlations between model-generated and human-annotated rubrics validate the reliability of our scalable rubric-generation approach.
Hanhua Hong, Yizhi Li, Luu Gia Huy +3
The University of Manchester · IQuest Research · Vietnam National University +2