A Judge Agent Closes the Reliability Gap in AI-Generated Scientific Simulation

Chengshuai Yang

cs.SEMar 26, 2026

A Judge Agent Closes the Reliability Gap in AI-Generated Scientific Simulation

Authors: Chengshuai Yang

Organizations: NextGen PlatformAI C Corp, USA

Abstract

Large language models can generate scientific simulation code, but the generated code silently fails on most non-textbook problems. We show that classical mathematical validation -- well-posedness, convergence, and error certification -- can be fully automated by a Judge Agent, reducing the silent-failure rate from 42% to 1.5% across 134 test cases spanning 12 scientific domains. The headline result comes from a prospective benchmark: 72 blinded tasks submitted by 12 independent scientists yield an 89% success rate (95% CI: [80%, 95%]) with automated error bounds, versus 53% without the Judge. On clinical CT (the only powered experiment, n = 200), the pipeline reaches 99% of expert quality. The residual 1.5% concentrates at bifurcation points where certifiability breaks down. We formalize this boundary through the simulability class S and introduce spec.md, a structured specification format that makes any scientific computation problem machine-readable and solver-independent. Code, data, and all 72 benchmark tasks are publicly archived.

Figures & tables

Panel of Figure 1 — Figure 1: Three-agent pipeline grounded in the simulability class $\mathfrak{S}$ . The Plan Agent translates natural language into a spec.md file (realizing S1). The Judge Agent verifies S2–S4 via 5 pre-execution gates and audits the solution via post-execution quality checks (realizing S4). Rejection triggers redesign (up to 3 rounds) or a certificate that the problem lies outside $\mathfrak{S}$ .

Panel of Figure 2 — Figure 2: spec.md format and a concrete example. (a) Generic structure: 6 mandatory sections encode the S1 six-tuple $\mathcal{S}=(\Omega,\mathcal{E},\mathcal{B},\mathcal{I},\mathcal{O},\varepsilon)$ ; 2 optional sections link to the primitive basis (Proposition 3 ) and benchmark variations. The format uses YAML-compatible key-value syntax within Markdown section headers. (b) A real spec.md file for CT reconstruction from the benchmark archive. All 72 prospective tasks and 12 development problems are archived in this format. Formal grammar in Supplementary Section S8.

Panel of Figure 3 — Figure 3: The Judge Agent’s contribution grows with problem difficulty. Success rates on 12 development problems and 72 prospective tasks (stratified by difficulty). Without the Judge, the success rate drops from 89% to 53% overall, and from 79% to 29% on frontier tasks—those closest to $\partial\mathfrak{S}$ . The gap widens monotonically with proximity to the boundary.

Panel of Figure 4 — Figure 4: CT reconstruction comparison (modified Shepp–Logan phantom [ 32 ] , 128 projections, Poisson noise, $n=200$ ). (a) Ground truth. (b) Framework with Judge ( $\in\mathfrak{S}$ verified): correct angular model, Ram-Lak filter, non-negativity enforced. (c) Without Judge: wrong angular range (360° assumed instead of 180°), producing doubled edges and streak artifacts; 5,326 non-negativity violations. (d) PSNR distribution: with Judge $25.6\pm 0.3$ dB vs. without Judge $13.1\pm 0.1$ dB ( $n=200$ , $p<10^{-30}$ , Wilcoxon).

Domain	$t_{\text{fw}}$	$t_{\text{expert}}$	$\rho$	Quality	$n$
Clinical CT	12 min	5 d	600 $\times$	99%	200
Seismic FWI	45 min	2 wk	450 $\times$	95%	5
Combustion (GRI-Mech)	25 min	5.5 d	320 $\times$	94%	15
Granular flow	18 min	1 wk	560 $\times$	92%	3
Helium ground state	8 min	4 d	720 $\times$	97%	1
BFS turbulent flow	22 min	2 wk	916 $\times$	91%	3

Table 1: Setup-time efficiency ratio

\rho

across 12 validated scientific domains.

\rho=t_{\text{expert}}/t_{\text{framework}}

. All domains lie within

\mathfrak{S}

; quality ratio is the framework PSNR (or domain-appropriate metric) divided by expert PSNR.

Outcome	Count	%	Redesign	Quality	Notes
Correct + bounded + quality	58	80.6%	1.4 $\pm$ 0.8	Pass	$\in\mathfrak{S}$ verified
Correct + bounded, quality flag	6	8.3%	2.1 $\pm$ 1.0	Flag	5/6 resolved after consult
$\mathcal{A}_{J}$ rejected (correct)	3	4.2%	—	—	Outside $\mathfrak{S}$ (ill-posed)
$\mathcal{A}_{J}$ rejected (fixable)	2	2.8%	—	—	Ambiguous NL input
Failed (resource limit)	2	2.8%	—	—	Exceeded 64 GB memory
Failed (wrong answer)	1	1.4%	3	Fail	Near event horizon

Table 2: Prospective benchmark: 72 blinded tasks from 12 scientists, 12 institutions, 9 countries. Managed by an independent coordinator. The Judge Agent verified

\mathfrak{S}

membership for each accepted task.

Panel of Figure 5 — Figure 5: Failure analysis across 134 test cases. (a) $\mathfrak{S}$ -verification reduces silent failures in two stages: pre-execution gates (42% $\to$ 6%) and post-execution quality audit (6% $\to$ 1.5%). (b) Quality ratio vs. estimated Lipschitz constant $\hat{L}_{\text{DAG}}$ (log scale). The two residual failures (red) occur at $\partial\mathfrak{S}$ —bifurcation points where $\hat{L}_{\text{DAG}}\to\infty$ —while all interior- $\mathfrak{S}$ problems (blue) achieve $\geq 75\%$ quality. Orange triangles: flagged by quality audit. The dashed line indicates the empirical boundary where certifiability (S4) degrades.

Symbol	Primitive	Operation
$\partial$	Differentiate	Spatial/temporal derivatives (finite differences, spectral, finite elements)
$\int$	Integrate	Quadrature, line/surface/volume integrals (Gauss, Monte Carlo)
$L$	Solve (linear)	Direct or iterative solution of $\mathbf{A}\mathbf{x}=\mathbf{b}$ (LU, CG, GMRES, multigrid)
$N$	Evaluate (nonlinear)	Pointwise nonlinear function evaluation ( $\|\psi\|^{2}\psi$ , reaction rates, constitutive laws)
$E$	Evolve	Time integration (Euler, Runge–Kutta, BDF, symplectic, exponential integrators)
$F$	Transform	Basis change (FFT, wavelet, spherical harmonics, Radon)

Table 3: The 12 computational primitives. Every problem in the benchmark decomposes into a DAG over these primitives. Each primitive is necessary (witness problems in Supplementary Section S4); the basis covers 25 numerical method families (Supplementary Table S1).

	Full		No $\mathcal{A}_{J}$		Opus+spec		Opus raw		PINN
	Cor.	Bnd.	Cor.	Bnd.	Cor.	Bnd.	Cor.	Bnd.	Cor.
CT reconstruction	✓	✓	✓	—	✓	—	✓	—	✓
Marmousi FWI	✓	✓	—	—	—	—	—	—	—
GRI-Mech ignition	✓	✓	✓	—	✓	—	—	—	—
Granular flow	✓	✓	—	—	—	—	—	—	—
He ground state	✓	✓	$\sim$	—	$\sim$	—	$\sim$	—	—

Table 4: Extended Data Table 1: Controlled comparison on 12 development problems (all satisfying S1–S4). Five conditions isolate the Judge’s contribution. “Correct”: solution within domain-specific tolerance. “Bound”: automated error bound (S4) provided. Each condition run 3

\times

with different seeds; majority vote reported. Note: 100% success on dev problems is expected, as these were used to design the Judge’s S1–S4 verification checks.

	This work	COMSOL	FEniCS	OF a	PINNs	LLM agents
NL input	✓	—	—	—	—	✓
Cross-domain	$\mathfrak{S}$ tested	Multi-phys.	PDE	CFD	PDE	Partial
Error bound	✓	—	A post.	—	—	—
$\mathfrak{S}$ -verification	✓	—	—	—	—	—
No code required	✓	—	—	—	—	✓
$\rho$ (median)	480 $\times$	—	—	—	—	—

Table 5: Extended Data Table 2: Comparison with existing tools. Each excels in its domain; this work adds automated

\mathfrak{S}

-verification to AI-generated code. Quantitative reference: on a standard Poisson problem (

\nabla^{2}u=f

[0,1]^{2}

128^{2}

grid), framework error is

3.1\times 10^{-5}

; FEniCS achieves

2.8\times 10^{-5}

; both match the

O(h^{2})

theoretical rate.

Phase	Goal	Mechanism	Milestone
I	Cross-domain validation	Three-agent pipeline, 12 primitives, S1–S4 verification, prospective benchmark	89% on 72 held-out tasks
II	Additional powered validations	Extend $n\geq 200$ powered experiments to $\geq$ 2 domains beyond CT	Powered CIs in $\geq$ 3 domains
III	Bifurcation certification	Implement continuation-based rejection, Lyapunov detection, ensemble perturbation	1.5% $\to$ 0% residual failures
IV	Specification portability	Adapters for FEniCS, COMSOL, OpenFOAM consuming spec.md ; independent consumers	$\geq$ 3 independent spec.md consumers
V	Domain expansion	Domain-specific Judge extensions for acoustics, particle physics, materials	$\geq$ 3 domains beyond imaging
VI	Primitive completeness	Proof that 12 primitives realize $\mathfrak{S}$ for all convergent discretizations	Coverage theorem

Table 6: Extended Data Table 3: Research and engineering roadmap. Phases are ordered by evidence strength: validate the core claim first, then harden and extend. Phase I is demonstrated in this work; later phases are research directions.

Obstruction	Condition	Fails	Example
Turing undecidability	S3	No convergent approximation	Halting problem
BSS uncomputability	S3	No finite algorithm over $\mathbb{R}$	Mandelbrot boundary membership
Quantum measurement	S2	Solution not unique/determined	Individual photon detection
Infinite-precision barrier	S4	Error bound not computable	Problems depending on Chaitin’s $\Omega$

Table S1 : Mapping of obstructions to

\mathfrak{S}

conditions.

#	Domain	Problem	Primitives	Target	Achieved	Data
1	Epidemiology	COVID-19 SEIR-D (8 counties)	$N,E,O,B$	10% peak	12.4%	R
2	Classical Mech.	Granular flow (2D, $N$ =5000)	$\partial,N,E,K,B,G$	$10^{-3}$	$7.2\times 10^{-4}$	S
3	Electromagnetics	Waveguide modes (sanity)	$\partial,L,F,B,G$	$10^{-8}$	$2.1\times 10^{-9}$	S
4	Quantum Chem.	He ground state (CI)	$\partial,L,\Pi,B,G,O$	1 mHa	0.4 mHa	S
5	Fluid Dynamics	BFS turbulent ( $Re$ =5100)	$\partial,E,N,K,B,G,\Pi$	5% TKE	3.8%	R †
6	Structural Mech.	Topology-opt. bracket	$\partial,L,O,B,G$	$10^{-3}$	$6.1\times 10^{-4}$	S

Table S2 : Table S0. Seven additional domain validations, all within

\mathfrak{S}

\mathfrak{S}

conditions verified by the Judge Agent for each.

Method Family	Primitive Decomposition
Finite Difference (FD)	$\partial,L,E,B,G$
Finite Element (FEM)	$\partial,\int,L,B,G$
Finite Volume (FVM)	$\int,L,E,B,G$
Spectral methods	$F,L,E,B$
Discontinuous Galerkin (DG)	$\partial,\int,L,E,B,G$
Boundary Element (BEM)	$\int,L,B,G$

Table S3 : Table S1. 25 numerical method families decomposed over the 12-primitive basis. All 25 families have been verified to satisfy

\mathfrak{S}

conditions S2–S4 for standard problem instances, establishing Proposition 3 (main text).

Problem	Replicator A	Replicator B	Replicator C	Variability
CT reconstruction	31.5 dB	31.8 dB	31.7 dB	$\pm$ 0.2 dB
Marmousi FWI	26.2 dB	26.5 dB	26.4 dB	$\pm$ 0.2 dB
GRI-Mech ignition	6.5%	6.1%	6.3%	$\pm$ 0.2%
Granular flow	$7.0\times 10^{-4}$	$7.4\times 10^{-4}$	$7.2\times 10^{-4}$	$\pm$ 3%
He ground state	0.38 mHa	0.42 mHa	0.40 mHa	$\pm$ 5%
BFS turbulent	3.6%	4.0%	3.8%	$\pm$ 5%

Table S4 : Table S11. Independent replication results across three instances. All problems verified as

\in\mathfrak{S}

; all produced bounded-error solutions.

Scientist	Problem Posed	$\mathcal{A}_{J}$	Bnd.	$\rho$	Expert Check
Geophysicist	Rayleigh wave in layered soil	$\in\mathfrak{S}$	Yes	380 $\times$	Correct
Bioengineer	Drug diffusion through tumor	$\in\mathfrak{S}$	Yes	520 $\times$	Correct
Astrophysicist	Bondi accretion onto compact obj.	$\in\mathfrak{S}$	Yes	450 $\times$	Correct $\dagger$
Materials sci.	Spinodal decomp. (Cahn–Hilliard)	$\in\mathfrak{S}$	Yes	600 $\times$	Correct
Chemical eng.	Packed-bed reactor, mass transfer	$\in\mathfrak{S}$	Yes	290 $\times$	Partial $\ddagger$

Table S5 : Table S4. Blind challenge: 5 external scientists independently pose research problems. All problems verified as

\in\mathfrak{S}

by the Judge Agent.

Feature	This work	COMSOL	FEniCS	OpenFOAM	Wolfram
Input	NL text	GUI + equations	Python + weak form	Config files	Wolfram Lang.
Domain	$\mathfrak{S}$ verified	Multi-physics	PDE only	CFD only	Broad
Error guarantee	Formal (Prop. S5 )	None	A posteriori only	None	None
$\mathfrak{S}$ -verification	Automated	None	None	None	None
$\rho$ (setup time)	480 $\times$	—	—	—	—
Expertise req.	Domain only	Domain + numerical	Domain + FEM + Python	Domain + CFD	Domain + Wolfram

Table S6 : Table S4b. Framework comparison with domain-specific tools. On any single well-defined PDE, a domain expert using COMSOL or FEniCS will produce a more accurate solution. The pipeline’s contribution is

\mathfrak{S}

-verified cross-domain automation from natural language, with a median

\rho=480\times

setup-time advantage.

#	Model	Complexity	Framework (dB)	Expert (dB)	Ratio
1	Marmousi-2	High (faulted layers)	26.4	27.8	95%
2	Synthetic salt body	Very high (salt dome)	23.1	25.2	92%
3	Thrust fault model	High (overturned layers)	25.4	27.0	94%
4	Thin layers	Medium (thin beds)	27.5	28.6	96%
5	Smooth gradient	Low (1D variation)	28.2	28.9	98%
	Mean $\pm$ s.d.		$25.8\pm 1.9$	$27.3\pm 1.6$	$95\%$

Table S7 : Table S5. Five seismic velocity models used for Marmousi FWI validation. All within

\mathfrak{S}

(regularized inverse problem with bounded

L_{\text{DAG}}

$T$ (K)	$p$ (atm)	$\phi$	Framework Error (%)	Expert Error (%)
1000	1	1.0	9.2	5.8
1000	10	1.0	8.7	5.1
1000	50	1.0	7.3	4.9
1200	1	1.0	6.8	4.2
1200	10	1.0	6.1	3.5
1200	50	1.0	5.9	3.3

Table S8 : Table S6. GRI-Mech 3.0 ignition delay: per-condition results. Error =

|(\tau_{\text{framework}}-\tau_{\text{exp}})/\tau_{\text{exp}}|\times 100\%

. All conditions within

\mathfrak{S}

; the Judge correctly rejects explicit RK4 at stiffness

>10^{6}

(condition 3 violation) and selects BDF.

County	Peak Error (%)	Cum. RMSE (%)	$R^{2}$	Expert Peak Err. (%)
King County (WA)	7.3	11.2	0.91	5.1
Maricopa (AZ)	9.8	15.3	0.87	7.2
Fulton County (GA)	11.5	17.8	0.84	8.5
Harris County (TX)	12.1	18.4	0.82	9.0
Los Angeles (CA)	12.8	19.6	0.80	9.4
Cook County (IL)	13.4	20.1	0.79	9.8

Table S9 : Table S7. COVID-19 SEIR-D per-county results. Peak timing error =

|(t_{\text{peak,fw}}-t_{\text{peak,data}})/t_{\text{peak,data}}|

. All within

\mathfrak{S}

; error is due to model-class limitations (SEIR-D cannot capture intervention timing), not numerical failure.

	GPT-4 (raw)		Claude-3.5 (raw)
Problem	Correct	Bound	Correct	Bound
CT reconstruction	Yes	No	Yes	No
Marmousi FWI	No	No	No	No
GRI-Mech ignition	No	No	Partial	No
Granular flow	No	No	No	No
He ground state	Partial	No	Partial	No

Table S10 : Table S8. LLM backbone comparison: Claude-3.5-Sonnet vs. GPT-4 in raw code-generation mode (no Judge, no

\mathfrak{S}

-verification).

Difficulty	Tasks	Correct+Bounded	Quality Pass	Med. Quality Ratio
Textbook (interior $\mathfrak{S}$ )	18	18 (100%)	18 (100%)	98%
Standard (interior $\mathfrak{S}$ )	30	27 (90%)	25 (83%)	95%
Frontier (near $\partial\mathfrak{S}$ )	24	19 (79%)	15 (63%)	89%
Total	72	64 (89%)	58 (81%)	94%

Table S11 : Table S9. Prospective benchmark results stratified by difficulty tier and proximity to

\partial\mathfrak{S}

Configuration	Valid	Bounded	$\mathfrak{S}$ condition tested
Full $\mathcal{A}_{J}$ (all gates + audit)	12/12	12/12	S1–S4 verified
Remove Gate 1 (dimensional)	11.4/12	10.8/12	S1: specifiability
Remove Gate 2 (BC/IC)	11.0/12	10.2/12	S2: well-posedness
Remove Gate 3 (well-posedness)	8.2/12	7.4/12	S2+S3: stability + approx.
Remove Gate 4 (classification)	11.8/12	11.6/12	S1: correct encoding
Remove Gate 5 (cost estimation)	12/12	11.4/12	S4: certifiability feasibility

Table S12 : Table S10. Per-gate ablation of Judge Agent on 12 development problems. Gates mapped to

\mathfrak{S}

conditions S1–S4 (

n=5

trials each, mean

\pm

s.d.).

#	Category	Problem Description	Outcome
(a) Ambiguous/incomplete NL (20 tests, targeting S1 via $\mathcal{A}_{P}$ )
1	(a)	“Solve the wave equation” (no BC, no IC, no domain)	Redesign $\to$ correct
2	(a)	“ $k=0.1$ , solve diffusion” ( $k$ ambiguous: conductivity or rate?)	Redesign $\to$ correct
3	(a)	“Large Reynolds number flow past cylinder” (no Re value)	Redesign $\to$ correct
4	(a)	“Knudsen regime gas dynamics” (jargon misinterpreted)	Human clarification
5	(a)	“Heat transfer in a room” (no geometry or materials)	Redesign $\to$ correct

Table S13 : Table S3. Full list of 50 adversarial test inputs, designed to probe the boundary

\partial\mathfrak{S}

. Each category targets specific

\mathfrak{S}

conditions.

Backbone	Correct	Bounded	Median Quality Ratio
Claude-3.5-Sonnet (default)	12/12	12/12	94%
Claude-3-Opus	12/12	12/12	93%
GPT-4-Turbo	11/12	11/12	91%

Table S14 : Table S14. LLM backbone sensitivity on 12 development problems.

\mathfrak{S}

-verification is backbone-independent.

Difficulty tier	Full pipeline	No $\mathcal{A}_{J}$	GPT-4+spec	PINN
Textbook (interior $\mathfrak{S}$ , 18)	18/18 (100%)	14/18 (78%)	13/18 (72%)	11/18 (61%)
Standard (interior $\mathfrak{S}$ , 30)	27/30 (90%)	17/30 (57%)	13/30 (43%)	7/30 (23%)
Frontier (near $\partial\mathfrak{S}$ , 24)	19/24 (79%)	7/24 (29%)	5/24 (21%)	1/24 (4%)
Total	64/72 (89%)	38/72 (53%)	31/72 (43%)	19/72 (26%)

Table S15 : Table S12. Extended controlled comparison on 72 prospective tasks. Without

\mathfrak{S}

-verification (No

\mathcal{A}_{J}

), success drops to 53%.

Configuration	Correct + Bounded	Quality Pass	$\mathfrak{S}$ cond.
Full $\mathcal{A}_{J}$ (all gates + audit)	64/72 (89%)	58/72 (81%)	S1–S4
Remove well-posedness gate	50/72 (69%)	44/72 (61%)	S2+S3
Remove BC/IC compatibility	56/72 (78%)	48/72 (67%)	S2
Remove dimensional consistency	60/72 (83%)	52/72 (72%)	S1
Remove classification	62/72 (86%)	56/72 (78%)	S1
Remove cost estimation	64/72 (89%)	54/72 (75%)	S4 feasibility

Table S16 : Table S13. Per-gate ablation of Judge Agent on 72 prospective tasks. Gates mapped to

\mathfrak{S}

conditions S1–S4 (

n=3

trials each).

Scientific Code Generation

Scientific code generation develops systems that turn scientific questions, papers, and workflows into executable, scientifically valid programs.

See all 51 papers

Aug 5, 2026

SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models

SciCode is the standard measure of the scientific-coding ability of language models: research-level problems that demand both frontier scientific theory and its implementation as working numerical code. It is a component of the Artificial Analysis Intelligence Index and a standing evaluation in government and national-laboratory suites. Yet its scores have recently plateaued: the strongest 2026 models cluster tightly around 60% subproblem accuracy, and a successor model ties its predecessor. We trace this stagnation to defects in the benchmark itself. A per-problem, domain-expert audit of all 65 test problems uncovers 263 defects; 192 of them, spread across 91% of the main problems, cause correct, instruction-following solutions to be wrongly rejected---through non-reproducible gold answers, over-tight tolerances, or self-contradictory specifications. Critically, 78% of these score-suppressing defects require specialized physics or mathematics knowledge to detect, not mere clerical proofreading. We corrected every confirmable defect to produce SciCode-Verified. The corrections add only the specifications a well-posed problem requires, repair grading, and tighten the tests that were too lenient; every change is recorded with its justification and independently re-checked by a second domain expert. We re-evaluate twelve frontier model snapshots on the corrected benchmark and find a substantial recovery: subproblem accuracy rises from 45--60% to 84--98%, and main-problem accuracy from 9--27% to 69--92%. State-of-the-art models are far more proficient in scientific coding than SciCode has suggested---the bottleneck was not model capability, but the quality of the evaluation instrument. We release SciCode-Verified with its complete audit trail as the corrected public standard.

Sihan Hu, Lyuhan Huang, Youjin Deng +1

Hefei National Laboratory College of Mechanical and Electrical Engineering University of Science and Technology of China · Hefei National Laboratory Institute of Theoretical Physics University of Science and Technology of China Chinese Academy of Sciences Hefei 230026, China · Chinese Academy of Sciences Beijing 100190, China

LLM Evaluation Scientific Code Generation Benchmark Validity Code Generation Benchmarks

Ask PDF

Mar 16, 2026

An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc

While LLMs have accelerated scientific code generation, comprehensively evaluating generated code remains challenging. Many benchmarks emphasize functional correctness or task completion, which is insufficient for code built on production HPC libraries, where solver selection, API conventions, memory management, parallel awareness, and performance also matter. We introduce PETSCAgent-Bench, a multidimensional benchmark and agent-based framework for assessing whether AI-generated scientific code uses a production HPC library as an expert would. A tool-augmented evaluator compiles, executes, and measures code and combines deterministic checks with LLM-based assessments in a 14-evaluator pipeline spanning five categories: correctness, performance, code quality, algorithmic appropriateness, and library-specific conventions. A2A and MCP enable black-box evaluation of compatible coding agents. Across realistic PETSc problems, frontier models generate readable, well-structured code but struggle with correctness on challenging problems and with library-specific conventions even when code compiles and runs---limitations that conventional pass/fail evaluation does not capture.

Hong Zhang, Barry Smith, Satish Balay +4

Argonne National Laboratory, Lemont, IL, USA

High-Performance Computing Code Generation AI Agent Evaluation Scientific Code Generation Code Generation Evaluation Code Generation Benchmarks

Ask PDF

Mar 29, 2026

PRBench: End-to-end Paper Reproduction in Physics Research

AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research.

Shi Qiu, Junyi Deng, Yiwei Deng +48

AI Agent Benchmarks AI Coding Agents Scientific Reproducibility Scientific Reasoning in Language Models

Ask PDF

Abstract

Scientific Code Generation

Explore similar work

SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models

An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc

PRBench: End-to-end Paper Reproduction in Physics Research