GPUPhysBench: Benchmarking Coding Agents for Correct and Efficient GPU Physics Simulation

Yuchen Sun; Jinjin He; Sinan Wang; Bo Zhu

cs.DCSep 28, 2026

GPUPhysBench: Benchmarking Coding Agents for Correct and Efficient GPU Physics Simulation

Authors: Yuchen Sun, Jinjin He, Sinan Wang, Bo Zhu

Organizations: Georgia Institute of Technology

Abstract

Writing fast GPU code for physical simulation is difficult: implementations must preserve numerical accuracy while handling irregular data access, synchronization, and iterative solvers. We introduce GPUPhysBench, a benchmark of 50 tasks testing whether coding agents can meet these demands. Tasks cover fluids, deformable solids, and granular materials, from individual simulation operators to complete simulators. Agents write, compile, test, and optimize GPU code with access to a NVIDIA GPU under fixed time budgets. We report pass rates and runtime performance relative to expert-optimized reference implementations. In a single-attempt evaluation of six frontier model-harness pairs, the two strongest pass all 50 tasks, but even the fastest reaches at least 0.9 the reference speed on only 22% of them, and no submission is more than 5% faster than the reference. The largest gaps arise in collision detection, constraint solving, and iterative solvers. GPUPhysBench brings physical simulation workloads to coding-agent evaluation, testing both the ability to implement numerical methods correctly and the ability to make them run efficiently.

Figures & tables

Panel of Figure 1 — Figure 1: Multi-step simulations driven by GPUPhysBench reference implementations.

Model	Harness	Correctness $\uparrow$	Pass Rate $\uparrow$	$\mathbf{fast}_{0.5}$ $\uparrow$	$\mathbf{fast}_{0.9}$ $\uparrow$	$\mathbf{fast}_{1.05}$ $\uparrow$
Claude-Opus-5	Claude Code	100%	100%	66%	22%	0%
GPT-5.6-Sol	Codex CLI	100%	100%	42%	16%	0%
Gemini-3.5-Flash	Gemini CLI	88%	88%	28%	16%	0%
DeepSeek-V4.1-Flash	DeepSeek Harness	86%	86%	38%	16%	0%
Qwen-3.8-Max	Qwen Code	52%	52%	34%	14%	0%
GLM-5.3	OpenCode	88%	86%	34%	16%	0%

Table 1: Results on GPUPhysBench over 50 tasks, with one attempt per task. The best result in each column is underlined.

Figure 2: Distribution of the 50 tasks across six computational categories.

Attempts	Time Budget	Coding Agent	Correctness $\uparrow$	Pass Rate $\uparrow$	$\mathbf{fast}_{0.5}$ $\uparrow$	$\mathbf{fast}_{0.9}$ $\uparrow$	$\mathbf{fast}_{1.05}$ $\uparrow$
1	$0.5\times$	Opus	98%	98%	54%	18%	0%
	$0.5\times$	GPT	100%	100%	36%	16%	0%
	$1.0\times$	Opus	100%	100%	66%	22%	0%
	$1.0\times$	GPT	100%	100%	42%	16%	0%
	$1.5\times$	Opus	100%	100%	76%	34%	4%
	$1.5\times$	GPT	100%	100%	40%	18%	0%

Table 2: Ablations of time budget and number of attempts. Budgets are relative to the default of 30 minutes for operators and 60 minutes for full simulators. With three attempts, a task counts as solved if any run solves it, and

\mathrm{fast}_{p}

uses the best passing speedup per task.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Local Grid and Lattice Computations
Task	Opus	GPT	Gemini	DeepSeek	Qwen	GLM
advect_rk1	0.87	0.80	0.55	0.88	–	–
advect_rk2	0.83	0.74	0.70	0.83	0.67	0.82
advect_rk3	0.85	0.44	0.42	0.73	0.62	–
advect_rk4	0.89	0.76	0.75	0.44	0.86	0.63
advect_mc	0.81	0.51	0.66	0.64	–	0.54

Appendix

Table 3: Per-task results. Each entry is the hidden-input speedup

s_{i}

of a passing submission relative to the reference (higher is better;

1.00

matches the reference). “–” marks a submission that fails the numerical checks or produces no valid output, and “A” marks a numerically correct submission that fails an audit. Tasks marked ∗ are checked by the residual of the returned solution rather than by comparison with the reference.

Model	Run	Pass Rate	$\mathrm{fast}_{0.5}$	$\mathrm{fast}_{0.9}$	$\mathrm{fast}_{1.05}$	Geo. mean
Claude-Opus-5	Run 1	100%	66%	22%	0%	0.53
	Run 2	98%	64%	24%	2%	0.53
	Run 3	100%	64%	28%	0%	0.52
	Mean $\pm$ s.d.	99.3 $\pm$ 1.2	64.7 $\pm$ 1.2	24.7 $\pm$ 3.1	0.7 $\pm$ 1.2	0.53 $\pm$ 0.01
	Best of 3, public selection	100%	76%	36%	2%	0.63
	Best of 3, hidden selection	100%	76%	36%	2%	0.63

Appendix

Table 4: Run-to-run variation at the default time limit. Runs 1–3 are independent single attempts; run 1 is the run in Table 1 . Metrics are percentages over all 50 tasks; the geometric mean is the speedup over passing tasks. Best of 3 keeps, per task, one run that passes all numerical checks and audits: public selection picks the highest public-input speedup, and hidden selection picks the highest hidden-input speedup, as in Table 2 . Both report the hidden-input speedup of the chosen run. These post-hoc selections require evaluator verdicts and reference timings unavailable to the agent during development.

Model	Pass Rate	$\mathrm{fast}_{0.5}$	$\mathrm{fast}_{0.9}$
Claude-Opus-5	100% / 100%	66% / 65%	22% / 16%
GPT-5.6-Sol	100% / 100%	42% / 35%	16% / 8%
Gemini-3.5-Flash	88% / 84%	28% / 14%	16% / 8%
DeepSeek-V4.1-Flash	86% / 85%	38% / 29%	16% / 8%
Qwen-3.8-Max	52% / 53%	34% / 28%	14% / 8%
GLM-5.3	86% / 87%	34% / 26%	16% / 8%

Appendix

Table 5: Task-weighted / family-weighted metrics. The family-weighted score groups near-variant tasks into one family each: the five advection schemes, the six LBM streaming, collision and full-step tasks, the four CCD primitive pairs, and the two Poisson solves. Every other task is its own family, giving 37 families of equal weight; within a family, tasks are weighted equally.

Model	Harness	Time (min)	Budget used	Deadline hits	Input tok.	Cached	Output tok.
Claude-Opus-5	Claude Code	15.7	43%	0	1.65M	96%	45k
GPT-5.6-Sol	Codex CLI	18.0	50%	0	2.72M	97%	33k
Gemini-3.5-Flash	Gemini CLI	8.9	25%	0	2.88M	90%	16k
DeepSeek-V4.1-Flash	DeepSeek Harness	18.5	54%	0	7.34M	99%	116k
Qwen-3.8-Max	Qwen Code	30.0	92%	19	1.65M	49%	41k
GLM-5.3	OpenCode	23.8	66%	4	2.96M	97%	66k

Appendix

Table 6: Agent development cost per task in the runs of Table 1 , as medians over the 50 tasks. Time is the wall-clock development time, also given as a fraction of the task budget; deadline hits count runs stopped at the budget. Input tokens include cached prompt tokens, whose share over all tasks is given separately. Token counts are reported by each harness; for runs stopped at the deadline, some harnesses report only the usage recorded before termination.

Figure 3: Distribution of the worst error-to-tolerance ratio over the checks on both input sets, for every evaluated submission that produces an output, in half-decade bins. The vertical line marks the tolerance; the shaded band is the range between the largest passing and the smallest failing ratio, which contains no submission. Comparison checks measure the relative

\ell_{2}

difference from the reference output; residual checks measure the true residual of a returned linear or Newton solution against the solver tolerance, and passing solvers stop just below it by design. Values are clipped to

[10^{-5},10^{8.5}]

; the rightmost residual failure is non-finite.

Task	Model	$\bm{t_{\mathrm{ref}}/t_{\mathrm{gen}}}$ $\uparrow$
PIC/FLIP particle-to-grid transfer	Claude-Opus-5	$0.556\times$
PIC/FLIP particle-to-grid transfer	Gemini-3.5-Flash	$0.134\times$
Dirichlet Poisson solve	Claude-Opus-5	$0.049\times$
Dirichlet Poisson solve	Gemini-3.5-Flash	$0.036\times$

Appendix

Table 7: Final performance of the selected implementations on hidden inputs. For passing submissions, higher

t_{\mathrm{ref}}/t_{\mathrm{gen}}

is better, and

1\times

matches the reference.

No.	Category	Test name	Description
1	Global Solves and Pressure Projection	poisson_dirichlet	Solve a Poisson system with homogeneous Dirichlet boundaries using preconditioned CG.
2	Global Solves and Pressure Projection	poisson_neumann	Solve a pure-Neumann Poisson system using preconditioned CG and return a zero-mean solution.
3	Global Solves and Pressure Projection	viscosity_implicit	Solve implicit viscous diffusion for velocity components on a MAC grid.
4	Global Solves and Pressure Projection	mass_spring_newtonian_implicit	Advance an implicit mass-spring step using Newton’s method.
5	Global Solves and Pressure Projection	fem_newtonian_implicit	Advance an implicit tetrahedral FEM step using Newton’s method.
6	Global Solves and Pressure Projection	mass_spring_pd	Advance a mass-spring step using projective dynamics local-global iterations.

Appendix

Table 8: Complete GPUPhysBench task list. Descriptions summarize the computation specified in each task prompt.

Task	Public / hidden size	Hidden-input variation	Checks and tolerances
Local Grid and Lattice Computations
advect_rk1	MAC velocity grid $256^{3}$	seed; Fourier modes 16 $\to$ 20 (same RMS, same $k_{\max}$ )	rel. $L_{2}$ of $u_{x},u_{y},u_{z}$ ( $\times 3$ ): $10^{-6}$
advect_rk2	MAC velocity grid $256^{3}$	seed; Fourier modes 16 $\to$ 20 (same RMS, same $k_{\max}$ )	rel. $L_{2}$ of $u_{x},u_{y},u_{z}$ ( $\times 3$ ): $10^{-6}$
advect_rk3	MAC velocity grid $256^{3}$	seed; Fourier modes 16 $\to$ 20 (same RMS, same $k_{\max}$ )	rel. $L_{2}$ of $u_{x},u_{y},u_{z}$ ( $\times 3$ ): $10^{-6}$
advect_rk4	MAC velocity grid $256^{3}$	seed; Fourier modes 16 $\to$ 20 (same RMS, same $k_{\max}$ )	rel. $L_{2}$ of $u_{x},u_{y},u_{z}$ ( $\times 3$ ): $10^{-6}$
advect_mc	MAC velocity grid $256^{3}$	seed; Fourier modes 16 $\to$ 20 (same RMS, same $k_{\max}$ )	rel. $L_{2}$ of $u_{x},u_{y},u_{z}$ ( $\times 3$ ): $10^{-4}$

Appendix

Table 9: Inputs, hidden-input variation and correctness checks of every task. Each call advances or evaluates one step. Sizes are those the driver passes to the submission; the hidden inputs have the same size unless a hidden size is given. The hidden inputs come from the public generator with the listed constants changed (public

\to

hidden); grids, particle lattices, mesh topology, material parameters, time steps and solver settings are otherwise kept. A check passes when its value is below the listed absolute tolerance on both input sets. Unless stated otherwise it is the FP64 relative error

\|g-r\|_{2}/\|r\|_{2}

of the submission output

g

against the reference output

r

on the same inputs;

\Delta q=q-q_{0}

is the change from the input, and (

\times k

) marks

k

separately checked components.

Solver	Total (ms)	Setup (ms)	Solve (ms)	Iterations
Reference (geometric multigrid PCG)	19.1	0.3	18.0	10
AMGX PCG + classical AMG	171.8	101.1	65.8	17
AMGX PCG + aggregation AMG	323.5	64.8	253.6	28

Appendix

Table 10: Reference solver of poisson_dirichlet versus AMGX. Total also includes data preparation (layout conversion, or CSR assembly and upload) and output.

Task	Library (example)	Reference (ms)	Library (ms)
mpm_explicit	Taichi ( mpm3d.py )	9.08	63.12
dem	Warp ( example_dem.py )	1.15	3.03
fem_explicit	Taichi ( implicit_fem.py )	0.95	3.59

Appendix

Table 11: Reference implementations versus ports of Warp and Taichi examples. For fem_explicit , the step includes computing the rest-state quantities, which the task requires and the Taichi example precomputes once; without them, the Taichi step takes 2.01 ms.

Panel of Figure 4 — Figure 4: Multi-step simulations driven by GPUPhysBench reference implementations, one per row, with the benchmark task named in parentheses. Each row shows four representative frames; times in (c) are in units of the channel height divided by the inflow speed.

Explore similar work

PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinement
Date pendingTianyidan Xie, Peiyu Wang, Hu Jiaxin +9Physics Simulation Llm-Driven Code Synthesis
Nanjing University · Skywork AI · Jilin University +2
AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents
May 16, 2026Sharareh Younesian, Wenwen Ouyang, Sina Rafati +11Graphics Processing Unit Kernels Agentic Optimization
AMD
GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models
Aug 6, 2026Shuai Wang, Yaxin Feng, Xuekun Jiang +12Physics Simulation Video World Models
1Shanghai Artificial Intelligence Laboratory · 2Hong Kong University of Science and Technology · 3Shanghai Jiao Tong University +1

Date pending, cs.ROJ/K move · Enter open · S save

PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinement

Tianyidan Xie, Peiyu Wang, Hu Jiaxin, Yuyi Qian+8

Nanjing University · Skywork AI · Jilin University+2

Physics Simulation Llm-Driven Code Synthesis Mixture-Of-Agents First Comprehensive Benchmark Rigid-Body Dynamics

Open paper AskPDF arXiv

Translating natural-language descriptions of physical phenomena into executable simulation code requires both programming expertise and physical reasoning. Current large language models (LLMs) lack this combination: they frequently produce code that runs but simulates the wrong physics. We introduce PhysCodeBench, the first benchmark for this task, with 1,200 expert-validated examples spanning four physical domains. Its evaluation suite, PhysCodeEval, goes beyond executability and visual fidelity to measure physical correctness directly from the engine state via conservation-law residuals and expert-written assertions, and supports cross-engine evaluation to disentangle physics reasoning from API fluency. As a reference method, we propose the Self-Corrective Multi-Agent Refinement Framework (SMRF), which decouples physics-aware error correction from code generation through specialized agents. This design is motivated by our finding that targeted correction, rather than generic iterative refinement, is the key driver of physical accuracy. SMRF nearly triples the physical-assertion pass rate of the best proprietary baseline (70.6% vs.\ 23.8%) and retains its advantage under cross-engine transfer.

Abstract

Explore similar work

PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinement

AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents

GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models