cs.AIOct 29, 2026
SavePAPERBENCHX: Can AI Reproduce Science Across Disciplines?
Organizations: UniPat AI
Abstract
Before AI can be trusted to discover new science, it should be able to reproduce what science already knows. Reproduction provides a rigorous checkpoint for an AI scientist: the target claim is known, yet recovering it still requires translating an incomplete scientific description into an executable experiment and producing evidence that can be independently verified. Existing research-reproduction benchmarks largely focus on AI and machine learning, where experiments share relatively homogeneous software stacks. Extending this capability across scientific disciplines introduces a different challenge: agents must operate domain-specific scientific software, make physical and numerical choices, diagnose long-running computations, and distinguish valid evidence from merely plausible outputs. We introduce PAPERBENCHX, a benchmark of 93 expert-constructed paper-reproduction tasks spanning 12 research areas across electromagnetics, photonics, chemistry, materials science, biology, and robotics. Each task provides the source paper, a public task specification, and a domain-native scientific environment involving specialized scientific software. Agents must reconstruct a key experiment, execute and diagnose it, and deliver a runnable workflow that supports the target scientific claim. To evaluate scientific validity rather than surface-level agreement, we replay every submission from scratch and score only regenerated evidence against expert-validated criteria. Across 10 frontier agent configurations, the best configuration fully reproduces only 14% tasks under the 90% threshold. Meanwhile, mean rubric scores range from 27.53% to 59.08%, indicating that substantial partial progress rarely translates into complete scientific reproduction. These results reveal a large gap between paper understanding and reliably producing reproducible scientific evidence.
Figures & tables
Figure 1: Frontier agents on PaperBenchX . Bars show full scores ( ); boxes show partial scores. GPT-6-Astra and GPT-5.6-sol use Codex; all others use Claude Code.
Figure 2: From task package to replay-grounded score. An agent receives the paper, task specification, environment, and starter submission; interacts with domain-native scientific software; and returns a runnable package. Clean replay regenerates the evidence used to score the scientific rubric. The displayed leaves and scores are illustrative.
| Benchmark | Research scope | Primary evaluation target | Paper-level reproduction | Domain-native scientific execution | Replay-grounded scientific verification |
|---|---|---|---|---|---|
| Machine-learning research and paper-reproduction benchmarks | |||||
| MLAgentBench ( Huang et al., 2023 ) | Machine learning | ML experimentation tasks | ✗ | ✗ | ✗ |
| RE-Bench ( Wijk et al., 2024 ) | Machine learning | Research-engineering tasks | ✗ | ✗ | ✗ |
| PaperBench ( Starace et al., 2025 ) | Machine learning | Paper reproduction | ✓ | ✗ | ✗ |
| SA-Bench ( Hu et al., 2026 ) | Machine learning | Paper reproduction | ✓ | ✗ | ✓ |
| Cross-domain scientific benchmarks | |||||
Table 1: Comparison with prior agent benchmarks. Checkmarks indicate that a benchmark explicitly evaluates the named capability.
Figure 3: Coverage and evaluation structure of PaperBenchX . A , The 93 tasks span 12 research areas in six scientific domains. B , Their 93 source papers span 44 publication venues; bar colors correspond to the 12 research areas in A. C , The task rubrics contain 3,168 scorable criteria (median 38 per task); points are tasks and black ticks mark research-area medians.
Figure 4: Task construction and quality control. Experts turn a published claim into a runnable task by defining its scientific scope, provisioning the domain-native environment, and specifying the public task contract. Reference replay, calibration checks, and independent expert review verify executability, fairness, and evidence requirements before the task is admitted to the benchmark.
| Overall (%) | Stage score (%) | Average effort | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Harness | Partial | Full | Ref. | Modeling | Execution | Validation | Turns | Time (h) |
| GPT-6-Astra | Codex | 59.08 | 13.98 | 0.00 | 72.72 | 69.35 | 52.76 | 348.7 | 5.96 |
| Fable 5.1 | Claude Code | 58.17 | 12.90 | 11.83 | 71.00 | 69.92 | 51.06 | 113.0 | 4.65 |
| Fable 5 | Claude Code | 57.96 | 11.83 | 13.98 | 72.13 | 68.99 | 50.39 | 112.0 | 5.09 |
| GPT-5.6-sol | Codex | 55.07 | 9.68 | 0.00 | 69.85 | 71.23 | 45.29 | 237.8 | 3.28 |
| Opus 5 | Claude Code | 52.52 | 7.53 | 0.00 | 66.99 | 67.16 | 47.76 | 259.2 | 6.52 |
Table 2: Model-level results on the shared set of 93 tasks. Partial, Full, Ref., and all stage scores are reported as percentages; metric definitions and outcome handling follow Section 4.1 .
Figure 5: Domain-specific capability profiles. (a) Mean rubric score by model and research area, ordered from hardest to easiest; circles mark Fable refusals scored using the Opus 5 fallback. (b) Domain difficulty, defined as 100 minus the cross-model mean. (c) Overall scores, matching Table 2 . Names are abbreviated.
Figure 6: Stage performance, interaction length, and workflow allocation. (a) Stage-score distributions over 930 outcomes; thick and thin lines denote the interquartile and 10th–90th percentile ranges, and circles and diamonds denote medians and means. (b) Partial score versus the trimmed average turns in Table 2 . (c) Mean tool-mediated time allocation across 200 sampled trajectories (50 per model), with phases coded from timestamped tool actions using fixed rules.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Role | Agent | Evaluator |
|---|---|---|---|
| Paper and addendum | Scientific scope, measurements, and evidence requirements | ✓ | ✓ |
| Instruction | Goal, deliverables, and submission interface | ✓ | ✓ |
| Environment | Simulator, dependencies, and resource envelope | ✓ | ✓ |
| Starter submission | Editable source tree and replay entry point | ✓ | ✓ |
| Rubric | Hierarchical, independently scored scientific requirements | ✗ | ✓ |
| Judge prompt | Evidence-inspection and decision rules | ✗ | ✓ |
Table A1: Task components and their visibility during evaluation. Evaluator-only components are included in the public demonstration package after evaluation.
| Field | Public contract |
|---|---|
| Scientific target | Guided-mode resonances of a three-pair multilayer dielectric grating in Meep |
| Cases | Unetched control plus full-depth air-slot widths of 0.030, 0.050, 0.070, and 0.150 m |
| Required evidence | Native spectra and logs, matched reference/device fluxes, complex reflected amplitudes, resonance fits, independent decay modes, and refinement records |
| Replay interface | No-argument bash/home/submission/reproduce.sh , executed offline |
| Compute envelope | Meep 1.17.1, 8 CPUs, 32 GiB RAM; 7 hours for the agent, 5 hours for native calculation, and 6 hours for replay and review |
| Failure reporting | Missing, dark, overlapping, or non-identifiable features remain unresolved and cannot be replaced with paper values or invented poles |
Table A2: Condensed public contract for the GMR demonstration task.
| ID | Task | Public demo |
|---|---|---|
| 01 RF antennas and arrays | ||
| 1-1 | 5g-smartphone-eight-element-mimo | |
| 1-2 | cpw-quad-port-uwb-mimo | yes |
| 1-3 | dual-band-wlan-miniaturized-patch | |
| 1-4 | f-stub-uwb-mimo | |
| 1-5 | inkjet-flexible-dual-band-cp-patch | |
Table A3: Complete 93-task benchmark inventory used in the reported evaluation.
| Area | Task | Source publication and DOI |
|---|---|---|
| Radio-frequency antennas and arrays | 01-cpw-quad-port-uwb-mimo | CPW Fed Compact UWB 4-Element MIMO Antenna with High Isolation ; 10.3390/s21082688 |
| Radio-frequency metasurfaces and scattering | 02-anisotropic-coding-diffusion-metasurface | Broadband diffusion metasurface based on a single anisotropic element and optimized by the Simulated Annealing algorithm ; 10.1038/srep23896 |
| Radio-frequency filters and frequency-selective surfaces | 03-dual-passband-angular-stable-fss | A Dual-Passband Frequency Selective Surface with High Angular Stability and Polarization Insensitivity ; 10.3390/mi15060690 |
| Integrated photonic devices | 04-swg-anisotropic-phase-shifter-meep | Ultra-broadband nanophotonic phase shifter based on subwavelength metamaterial waveguides ; 10.1364/PRJ.373223 |
| Photonic crystals and topological photonics | 05-pt-bragg-unidirectional-invisibility-meep | Unidirectional Invisibility induced by PT-Symmetric Periodic Structures ; 10.1103/PhysRevLett.106.213901 |
| Optical metasurfaces, gratings, and plasmonics | 06-gmr-grating-fano-meep | Controlling Fano resonances in multilayer dielectric gratings towards optical bistable devices ; 10.1038/s41598-018-34787-9 |
Table A4: The twelve public demonstration tasks and their source publications.
Figure 7: Representative outputs from domain-native environments. Panels show solver fields and spectra, numerical analysis outputs, and embodied-task scenes produced by scientific workflows in the benchmark.
| ID | Execution stack | Agent h | Verifier h | CPU | RAM GiB | Disk GiB | GPU |
|---|---|---|---|---|---|---|---|
| 1-2 | Ansys HFSS | 24 | 25 | 32 | 128 | 200 | |
| 2-1 | Ansys HFSS | 72 | 73 | 32 | 128 | 200 | |
| 3-1 | Ansys HFSS | 48 | 49 | 32 | 128 | 300 | |
| 4-13 | Meep | 7 | 6 | 8 | 32 | 50 | |
| 5-9 | Meep | 5 | 5 | 8 | 32 | 50 | |
| 6-4 | Meep | 7 | 6 | 8 | 32 | 50 |
Table A5: Execution stacks and declared resource limits for the twelve public task packages in Table A4
| Check family | Perturbation and expected behavior |
|---|---|
| Contract and parser integrity | Preserve rubric hashes and complete leaf mappings; reject duplicate keys, nonfinite values, and invalid numerical grids while accepting documented equivalent formats. |
| Native evidence and variant identity | Reject incomplete solver logs, mismatched setup names, explicit non-convergence, and copied variant outputs; accept a shared solve only when the task contract permits it and the variants remain identifiable. |
| Evidence boundary | Exclude cached outputs and links outside permitted roots; keep private references and evaluator credentials inaccessible; fail closed when network isolation cannot be established. |
| Completion and provenance | Give no completion credit to an empty submission; require delivery records, input hashes, and preprocessing conventions to agree with the task contract. |
| Local failure accounting | Corrupt one input, metric, or figure panel and verify that only dependent rubric leaves lose credit. |
| Interactive-task contract | Use evaluator-owned episode records and protocol identifiers; do not allow candidate summaries to replace required trajectory or visual evidence. |
Table A6: Regression cases used to check evidence admission and local scoring. Valid controls accompany invalid fixtures so that permissible variation is not mistaken for unsupported evidence.
Figure 8: Task-level completion landscape. (a) Maximum Partial score attained on each of the 93 tasks, ordered by research area. (b) Number of configurations exceeding on the same task. The horizontal line marks the operational full-reproduction threshold; 71 tasks have no configuration above it.
Figure 9: Stage profiles in three trace case studies. Bars report Modeling, Execution, and Validation scores; diamonds report the aggregate task score.