SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

Yadi Cao; Sicheng Lai; Jiahe Huang; Yang Zhang; Zach Lawrence; Rohan Bhakta; Izzy F. Thomas; Mingyun Cao

physics.comp-phMar 11, 2026

SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

Authors: Yadi Cao, Sicheng Lai, Jiahe Huang, Yang Zhang, Zach Lawrence, Rohan Bhakta, Izzy F. Thomas, Mingyun Cao,

Organizations: University of California San Diego · The Chinese University of Hong Kong, Shenzhen · Peking University · University of California, Los Angeles ·

Abstract

Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental resources. As a result, metrics like pass@k become impractical under realistic budget constraints. To address this gap, we introduce SimulCost, the first benchmark targeting cost-sensitive parameter tuning in physics simulations. SimulCost compares LLM tuning cost-sensitive parameters against traditional scanning approach in both accuracy and computational cost, spanning 2,643 single-round (initial guess) and 2,304 multi-round (adjustment by trial-and-error) tasks across 11 simulators from fluid dynamics, solid mechanics, and plasma physics, whose costs are analytically defined and platform-independent. A twelfth simulator, a production plasma code measurable only by wall clock, is reported separately. Frontier LLMs achieve 45-62% success rates in single-round mode, dropping to 34-50% under high accuracy requirements, rendering their initial guesses unreliable especially for high accuracy tasks. Multi-round mode improves rates to 66-81%, but LLMs are 1.5-2.7x slower than traditional scanning, making them uneconomical choices. We also investigate parameter group correlations for knowledge transfer potential, and the impact of in-context examples and reasoning effort, providing practical implications for deployment and fine-tuning. We open-source SimulCost as a static benchmark and extensible toolkit to facilitate research on improving cost-aware agentic designs for physics simulations, and for expanding new simulation environments. Code and data are available at https://github.com/Rose-STL-Lab/SimulCost-Bench

Figures & tables

Figure 1: Overview of SimulCost . Our benchmark evaluates LLM agents on cost-sensitive parameter tuning across 12 physics simulators spanning fluid dynamics, solid mechanics, and plasma physics. Given a simulation task, tuning mode, and accuracy requirement, the LLM proposes tunable parameters in either single-round (initial guess) or multi-round (trial-and-error) mode. The challenge is balancing accuracy and cost: overly coarse parameters fail accuracy requirements, while overly fine parameters waste computation.

Benchmark	Domain	Tool Cost	Platform Indep.	Toolkit
TaskBench ( Shen et al., 2024 )	General Tools	✗	–	✗
CATP-LLM ( Wu et al., 2025a )	General Tools	✓	✗	✗
ScienceAgentBench ( Chen et al., 2025b )	Data Science	✗	–	✗
DiscoveryBench ( Majumder et al., 2024 )	Data Science	✗	–	✗
Auto-Bench ( Chen et al., 2025a )	Causal Discovery	✓	✓	✗
MLE-Bench ( Chan et al., 2025 )	ML Engineering	✓ ∗	✗	✗

Table 1: Comparison of benchmarks for LLM agents in scientific and engineering domains. Tool Cost: whether it tracks computational costs of external tools (separate from LLM inference). Platform Indep.: whether tool cost measurement is platform-independent (not based on wall time). Toolkit: whether it provides an extensible simulation environment. –: not applicable (no tool cost). ∗ Ambiguous: costs are reported but mixed with other components or based on wall time. † Claims computational cost but reports only LLM token costs in USD.

Panel of Figure 2 — Figure 2: Overall performance across models, accuracy levels, and inference modes. SR = Single-Round, MR = Multi-Round; High/Med/Low = accuracy level requirements.

Panel of Figure 3 — Figure 3: Key findings. (a) Forest plot of multi-round improvement by model and accuracy; all differences significant at $p<0.05$ except Claude-3.7-Sonnet at low accuracy. (b)–(c) Task group summary: aggregated success rate and efficiency by parameter type; single-round (solid) vs. multi-round (hollow). (d) Task correlations: within-group vs. between-group shows no significant difference. (e)–(f) In-context learning (ICL) variants improve single-round success but degrade multi-round; cost information in examples preserves efficiency.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Solver	Single	Multi	Single-only Params	Both Params	Low	Med	High
Analytical cost (main results)
Burgers 1D	339	339	–	beta , cfl , k , n_space	$8.0\times 10^{-2}$	$4.0\times 10^{-2}$	$1.0\times 10^{-2}$
Diff-React 1D	90	90	–	cfl , n_space , tol	vary ∗	vary ∗	vary ∗
Euler 1D	197	197	–	beta , cfl , k , n_space	$8.0\times 10^{-2}$	$2.0\times 10^{-2}$	$1.0\times 10^{-2}$
Euler 2D	232	232	–	cfl , cg_tolerance , n_grid_x	$8.0\times 10^{-2}$	$4.0\times 10^{-2}$	$8.0\times 10^{-3}$
FEM 2D	103	103	–	cfl , dx	vary ∗	vary ∗	vary ∗

Appendix

Table 2: Summary of solvers in the SimulCost benchmark. Single/Multi columns show task counts per mode. Single-only parameters are used exclusively in single-round evaluation; Both parameters support both modes. Low/Med/High columns show accuracy thresholds. ∗ Thresholds vary by task type. The upper block defines the tool cost analytically by counting dominant-operation FLOPs and carries the main results; the lower block is a compiled binary whose FLOPs cannot be estimated in closed form, so its cost is wall-clock time on fixed hardware, which is not comparable across solvers and is reported separately in Appendix C.15 . See Appendix A.2 for physics background and citations.

Component	Specification
CPU	Intel Core i9-14900KF (24-core, 2.4–6.0GHz)
Memory	64GB DDR5 (2 $\times$ 32GB, 5200MHz)
GPU	NVIDIA GeForce RTX 4090 (24GB GDDR6X)

Appendix

Table 3: Computational environment specifications.

Model/Acc level	Low		Med		High		Ave
Metrics	S	E	S	E	S	E	S	E
GPT-5	69.8	0.49	64.9	0.38	49.9	0.33	61.5	0.39
Claude-3.7-Sonnet	65.2	0.39	51.8	0.36	41.0	0.37	52.7	0.38
Llama-3-70B-Instruct	58.2	0.16	41.9	0.17	34.4	0.29	44.8	0.20
Qwen3-32B	61.6	0.46	42.8	0.39	34.6	0.33	46.3	0.39
GPT-OSS-120B	64.7	0.52	47.1	0.39	33.8	0.43	48.5	0.44

Appendix

Table 4: The overall results across the 11 solvers with analytical costs. Abbreviations: S - Success Rate (%), E - Efficiency. Low/Med/High - accuracy levels. Measurements reported for the single-round inference mode.

Model/Acc level	Low		Med		High		Ave
Metrics	S	E	S	E	S	E	S	E
GPT-5	85.5	0.69	80.9	0.59	75.9	0.72	80.8	0.66
Claude-3.7-Sonnet	65.6	0.37	66.0	0.32	66.6	0.43	66.1	0.37
Llama-3-70B-Instruct	69.2	0.40	73.2	0.38	69.5	0.54	70.6	0.43
Qwen3-32B	71.0	0.66	67.7	0.49	64.2	0.65	67.7	0.60
GPT-OSS-120B	74.2	0.86	78.1	0.54	79.1	0.64	77.1	0.67

Appendix

Table 5: The overall results across the 11 solvers with analytical costs. Abbreviations: S - Success Rate (%), E - Efficiency. Low/Med/High - accuracy levels. Measurements reported are for multi-round tunable parameters only.

Model / Acc level	Low		Med		High		Ave
Inference Modes	SR	MR	SR	MR	SR	MR	SR	MR
GPT-5	69.8	85.5	64.9	80.9	49.9	75.9	61.5	80.8
Claude-3.7-Sonnet	65.2	65.6	51.8	66.0	41.0	66.6	52.7	66.1
Llama-3-70B-Instruct	58.2	69.2	41.9	73.2	34.4	69.5	44.8	70.6
Qwen3-32B	61.6	71.0	42.8	67.7	34.6	64.2	46.3	67.7
GPT-OSS-120B	64.7	74.2	47.1	78.1	33.8	79.1	48.5	77.1

Appendix

Table 6: Comparison between single-round and multi-round inference modes: Success Rate . Abbreviations: Low/Med/High - accuracy levels, SR/MR - single-round and multi-round modes.

Model / Acc level	Low		Med		High		Ave
Inference Modes	SR	MR	SR	MR	SR	MR	SR	MR
GPT-5	0.49	0.69	0.38	0.59	0.33	0.72	0.39	0.66
Claude-3.7-Sonnet	0.39	0.37	0.36	0.32	0.37	0.43	0.38	0.37
Llama-3-70B-Instruct	0.16	0.40	0.17	0.38	0.29	0.54	0.20	0.43
Qwen3-32B	0.46	0.66	0.39	0.49	0.33	0.65	0.39	0.60
GPT-OSS-120B	0.52	0.86	0.39	0.54	0.43	0.64	0.44	0.67

Appendix

Table 7: Comparison between single-round and multi-round inference modes: Efficiency . Abbreviations: Low/Med/High - accuracy levels, SR/MR - single-round and multi-round modes.

Model / Accuracy	Single	Multi	$\Delta$	$p$ -value
GPT-5	63.5%	88.7%	+25.2%	$<$ 0.001
Claude-3.7-Sonnet	57.7%	66.5%	+8.8%	0.068
Llama-3-70B-Instruct	52.4%	79.6%	+27.3%	$<$ 0.001
Qwen3-32B	47.6%	71.2%	+23.6%	$<$ 0.001
GPT-OSS-120B	52.8%	83.1%	+30.3%	$<$ 0.001
Overall (High)	46.8%	77.4%	+30.5%	$<$ 0.001

Appendix

Table 8: Multi-round improvement over single-round (McNemar’s test). Single/Multi = success rate (%);

\Delta

= improvement. Top: per-model averages across accuracy levels. Bottom: overall by accuracy level.

Model	Effect	SE	$p$ -value	Dataset Var.
All	+21.3%	0.006	$<$ 0.001	0.041
GPT-5	+24.7%	0.011	$<$ 0.001	0.000
Claude-3.7-Sonnet	+7.0%	0.014	$<$ 0.001	0.000
Llama-3-70B	+25.2%	0.012	$<$ 0.001	0.000
Qwen3-32B	+22.2%	0.012	$<$ 0.001	0.000
GPT-OSS-120B	+27.4%	0.012	$<$ 0.001	0.000

Appendix

Table 9: Random-intercept linear probability model, success

\sim

mode

+(1|\texttt{dataset})

. Effect is the difference in mean success between modes. The per-model fits return a dataset variance of exactly zero, a boundary solution the optimiser reports as singular, so they detect no between-dataset variation rather than establishing its absence.

Model	Accuracy	$\Delta$ SR	$p$ (uncorr.)	$p$ (FDR)	Sig.
GPT-5	Low	+23.5%	$<$ 0.001	$<$ 0.001	Yes
GPT-5	Medium	+23.1%	$<$ 0.001	$<$ 0.001	Yes
GPT-5	High	+29.0%	$<$ 0.001	$<$ 0.001	Yes
Claude-3.7-Sonnet	Low	-3.3%	0.068	0.068	No
Claude-3.7-Sonnet	Medium	+11.7%	$<$ 0.001	$<$ 0.001	Yes
Claude-3.7-Sonnet	High	+18.1%	$<$ 0.001	$<$ 0.001	Yes

Appendix

Table 10: McNemar’s test with Benjamini-Hochberg FDR correction. With 15 comparisons (5 models

\times

3 accuracy levels), we apply FDR correction at

\alpha=0.05

. All comparisons remain significant after correction except Claude-3.7-Sonnet at low accuracy.

	Single-Round				Multi-Round
Dataset	Base	ICL	No-Cost	Mixed	Base	ICL	No-Cost	Mixed
Euler 1D	38.1	50.3	67.0	66.4	50.8	39.8	39.7	39.8
Heat 1D	69.9	75.0	85.9	86.6	87.7	81.0	85.0	72.8
MPM 2D	34.3	49.7	63.0	71.3	35.2	36.4	35.4	28.7
NS Trans. 2D	38.3	68.3	73.4	71.2	78.5	72.7	70.3	69.7

Appendix

Table 11: ICL performance breakdown by dataset and inference mode. Values show success rate (%) averaged across accuracy levels. Base = no ICL; ICL = accuracy-matched examples with cost; No-Cost = ICL without cost info; Mixed = examples from all accuracy levels.

Panel of Figure 4 — Figure 4: Detailed ICL ablation results. (a) Success rate improves with ICL in single-round mode but degrades in multi-round. (b) Efficiency drops when cost information is omitted from examples (No-Cost variant).

	Single-Round						Multi-Round
Effort Level	Low		Med		High		Low		Med		High
	S	E	S	E	S	E	S	E	S	E	S	E
Minimal	74.2	0.39	60.5	0.29	48.0	0.19	81.7	0.66	84.5	0.50	84.8	0.70
Medium	68.1	0.43	62.1	0.28	50.0	0.15	86.5	0.71	81.8	0.52	75.1	0.64
High	70.4	0.44	60.8	0.29	58.6	0.19	87.9	0.62	78.3	0.51	76.0	0.56

Appendix

Table 12: GPT-5 reasoning effort ablation by accuracy level and inference mode. S = Success Rate (%), E = Efficiency. Bold indicates best per column.

Figure 5: Reasoning effort ablation by accuracy level. Left: Single-round mode. Right: Multi-round mode. No consistent pattern emerges—increased reasoning effort does not reliably improve parameter selection.

Method	Success Rate	Rel. Cost
Brute-force	93.5%	1.00
Bisection	97.8%	1.08
Nelder-Mead 1D	97.8%	1.42

Appendix

Table 13: Non-LLM optimization baselines on monotonic multi-round parameters.

Model	Eff (SR)	RC (SR)	Eff (MR)	RC (MR)
GPT-OSS-120B	0.44	0.49	0.67	0.91
Qwen3-32B	0.39	0.57	0.60	0.98
GPT-5	0.39	1.17	0.66	1.18
Claude-3.7-Sonnet	0.38	0.72	0.37	2.08
Llama-3-70B	0.20	1.11	0.43	2.89

Appendix

Table 14: Conditional efficiency (Eff) vs. unconditional relative cost (RC). RC

<

1 means total LLM cost is below brute-force including failed runs.

Mode	Success Rate Shift	Efficiency Shift
Single-round	+0.069	$-$ 0.047
Multi-round	+0.072	$-$ 0.090

Appendix

Table 15: Prompt sensitivity: mean shift from original results across 25 variants.

Metric	$\Delta$ (Tool Search $-$ Internal)
Success Rate	+0.086
Efficiency	$-$ 0.520
Relative Cost	+2.363

Appendix

Table 16: Tool-augmented (LLM initial guess + search) vs. LLM internal adjustment (multi-round only).

Profile	Precision	LLM Guess	Near-Optimal	Cost Ratio
p3	High	0.10	0.50	13.8 $\times$
p1	Low	0.25	0.50	5.7 $\times$

Appendix

Table 17: Tool-augmented tuning examples showing conservative undershoots.

	BO			LLM Avg
Solver	Low	Med	High	Low	Med	High
Burgers 1D	100	99	87	78	77	74
Diff-React 1D	76	48	50	81	74	75
Euler 1D	74	65	51	60	67	58
Euler 2D	93	98	100	91	90	89
FEM 2D	64	53	9	73	71	65

Appendix

Table 18: Bayesian Optimization vs LLM performance by solver (multi-round mode). SR = Success Rate (%), Eff = Efficiency. LLM Avg is the mean across GPT-5, Claude, Llama, Qwen, and GPT-OSS.

	BO			LLM Avg
Solver	Low	Med	High	Low	Med	High
Burgers 1D	0.36	0.41	0.67	0.78	0.70	0.85
Diff-React 1D	0.30	0.97	0.64	1.20	1.18	1.00
Euler 1D	0.99	1.08	1.18	1.21	0.78	0.79
Euler 2D	0.60	0.62	0.57	0.42	0.38	0.49
FEM 2D	0.30	0.20	0.48	0.46	0.51	0.73

Appendix

Table 19: Bayesian Optimization vs LLM efficiency by solver (multi-round mode). Values show cost efficiency (higher is better; 1.0 = brute-force search cost).

Group	Parameters	Description
Spatial	n_space , dx , nx , n_grid_x , …	Grid/mesh resolution parameters affecting spatial discretization accuracy. Most common across all solvers.
Temporal	cfl , dt	Timestep and CFL parameters controlling temporal stability and accuracy.
Tolerance	error_threshold , cg_tolerance , cg_atol , residual_threshold , …	Convergence tolerances for iterative solvers to control solution accuracy.
Misc	beta , k , t_init , relax , …	Physics-specific parameters (limiter coefficients, relaxation factors) varying by solver.

Appendix

Table 20: Task group classification. Parameters are grouped by their role in numerical simulation, enabling analysis of LLM performance across parameter types.

Panel of Figure 6 — Figure 6: Performance by task group. Spatial and Tolerance parameters show consistently strong performance, while Misc parameters lag especially at higher accuracy levels.

Panel of Figure 7 — Figure 7: Average Pearson correlation between task groups. Diagonal = within-group; off-diagonal = between-group.

Panel of Figure 8 — Figure 8: Full pairwise task correlation matrix. Tasks grouped by parameter type: Spatial, Temporal, Tolerance, Misc. Black lines separate groups.

Model	Success Rate (%)			Efficiency
	Single	Multi	$\Delta$	Single	Multi
GPT-5	61.5	80.8	+19.2	0.39	0.66
Claude-3.7-Sonnet	52.7	66.1	+13.4	0.38	0.37
Llama-3-70B-Instruct	44.8	70.6	+25.8	0.20	0.43
Qwen3-32B	46.3	67.7	+21.3	0.39	0.60
GPT-OSS-120B	48.5	77.1	+28.6	0.44	0.67

Appendix

Table 21: Overall model performance (corresponds to Figure 2 ).

Task Group	Success Rate (%)			Efficiency
	Single	Multi	$\Delta$	Single	Multi
Spatial	55.5	75.7	+20.2	0.28	0.62
Temporal	57.3	73.8	+16.5	0.42	0.45
Tolerance	59.4	76.5	+17.2	0.32	0.46
Misc	32.7	61.1	+28.4	0.86	0.77

Appendix

Table 22: Performance by task group (corresponds to Figure 6 ).

Variant	Success Rate (%)			Efficiency
	Single	Multi	$\Delta$	Single	Multi
Base	46.7	64.1	+17.3	0.328	0.367
Idealized ICL	63.8	53.2	-10.6	0.445	0.377
Cost-Ignorant ICL	73.4	53.1	-20.3	0.364	0.470
Mixed-Accuracy ICL	75.4	46.3	-29.1	0.572	0.435

Appendix

Table 23: ICL variant comparison using Claude-3.7-Sonnet (corresponds to Figure 3 ).

Solver	Model	Low		Med		High		Avg
		S	E	S	E	S	E	S	E
Burgers 1D	GPT-5	53.9	1.07	41.7	0.71	23.8	0.40	39.8	0.67
	Claude	48.9	1.46	25.0	0.94	18.9	0.78	30.9	1.02
	Llama	62.2	0.55	56.7	0.26	21.7	0.85	46.9	0.50
	Qwen	33.3	1.26	25.0	0.68	15.6	0.95	24.6	0.93
	GPT-OSS	48.3	1.31	22.8	0.71	21.7	0.82	30.9	0.91

Appendix

Table 24: Single-round results for Fluid Dynamics solvers. S: Success Rate (%), E: Efficiency.

Solver	Model	Low		Med		High		Avg
		S	E	S	E	S	E	S	E
Burgers 1D	GPT-5	85.0	0.83	86.1	0.75	72.2	0.98	81.1	0.85
	Claude	77.8	0.72	65.0	0.54	62.2	0.65	68.3	0.63
	Llama	79.4	0.53	80.0	0.53	57.5	0.68	72.3	0.57
	Qwen	66.7	0.84	63.9	0.91	59.2	1.10	63.2	0.94
	GPT-OSS	71.7	0.70	71.7	0.49	83.3	0.67	75.6	0.62

Appendix

Table 25: Multi-round results for Fluid Dynamics solvers. S: Success Rate (%), E: Efficiency.

Solver	Model	Low		Med		High		Avg
		S	E	S	E	S	E	S	E
Heat 1D	GPT-5	88.0	0.38	80.0	0.36	55.4	0.05	74.5	0.19
	Claude	96.0	0.45	68.0	0.99	45.6	0.53	69.9	0.62
	Llama	100.0	0.01	92.0	0.05	82.4	0.23	91.5	0.05
	Qwen	98.0	0.47	68.0	0.78	35.9	0.29	67.3	0.47
	GPT-OSS	100.0	0.71	65.3	0.56	33.0	0.94	66.1	0.72

Appendix

Table 26: Single-round results for Heat & Diffusion solvers. S: Success Rate (%), E: Efficiency.

Solver	Model	Low		Med		High		Avg
		S	E	S	E	S	E	S	E
Heat 1D	GPT-5	96.0	0.88	92.0	0.71	88.1	0.86	92.0	0.81
	Claude	92.0	0.82	88.0	1.01	83.2	0.88	87.7	0.90
	Llama	92.0	0.66	93.3	0.51	86.8	0.53	90.7	0.56
	Qwen	92.0	0.85	88.7	0.73	84.6	0.81	88.4	0.80
	GPT-OSS	92.0	0.97	93.3	0.83	92.3	0.85	92.5	0.88

Appendix

Table 27: Multi-round results for Heat & Diffusion solvers. S: Success Rate (%), E: Efficiency.

Solver	Model	Low		Med		High		Avg
		S	E	S	E	S	E	S	E
FEM 2D	GPT-5	73.0	0.13	80.0	0.14	39.7	0.47	64.3	0.20
	Claude	73.0	0.19	58.4	0.14	52.6	0.31	61.3	0.20
	Llama	65.5	0.10	63.9	0.26	67.9	0.22	65.8	0.18
	Qwen	70.0	0.15	63.9	0.22	51.3	0.21	61.7	0.19
	GPT-OSS	42.0	0.23	51.9	0.39	49.6	0.20	47.8	0.26

Appendix

Table 28: Single-round results for Solid Mechanics solvers. S: Success Rate (%), E: Efficiency.

Solver	Model	Low		Med		High		Avg
		S	E	S	E	S	E	S	E
FEM 2D	GPT-5	90.0	0.44	91.5	0.60	100.0	0.72	93.8	0.58
	Claude	57.5	0.43	63.2	0.32	53.8	0.38	58.2	0.38
	Llama	46.0	0.10	48.5	0.21	37.2	0.38	43.9	0.20
	Qwen	75.5	0.45	66.9	0.61	60.3	0.68	67.5	0.57
	GPT-OSS	85.0	0.40	89.2	0.72	86.8	0.95	87.0	0.65

Appendix

Table 29: Multi-round results for Solid Mechanics solvers. S: Success Rate (%), E: Efficiency.

Solver	Model	Low		Med		High		Avg
		S	E	S	E	S	E	S	E
HM Linear	GPT-5	97.6	0.35	91.1	0.49	85.2	0.65	91.3	0.48
	Claude	81.3	0.06	61.9	0.22	64.9	0.22	69.4	0.14
	Llama	77.2	0.04	54.2	0.59	36.7	0.70	56.0	0.25
	Qwen	81.3	0.05	56.2	0.80	45.0	0.65	60.8	0.30
	GPT-OSS	88.4	0.14	62.3	0.28	44.7	0.27	65.1	0.22

Appendix

Table 30: Single-round results for Plasma Physics solvers. S: Success Rate (%), E: Efficiency.

Solver	Model	Low		Med		High		Avg
		S	E	S	E	S	E	S	E
HM Linear	GPT-5	98.8	0.53	97.8	0.81	100.0	0.86	98.9	0.72
	Claude	59.9	0.06	72.1	0.17	85.6	0.21	72.5	0.13
	Llama	66.7	0.12	65.3	0.35	64.4	0.41	65.5	0.26
	Qwen	84.9	0.09	68.3	0.14	63.0	0.86	72.1	0.22
	GPT-OSS	100.0	0.42	98.7	0.62	100.0	0.58	99.6	0.53

Appendix

Table 31: Multi-round results for Plasma Physics solvers. S: Success Rate (%), E: Efficiency.

Solver	Model/Acc level	Low		Med		High		Ave
	Metrics	S	E	S	E	S	E	S	E
EPOCH 1D	GPT-5	82.2	0.05	73.9	0.02	80.4	0.01	78.9	0.03
	Claude	77.8	0.28	74.2	0.29	70.8	0.29	74.3	0.29
	Llama	51.1	0.15	42.9	0.13	61.6	0.10	51.9	0.13
	GPT-OSS	93.3	0.40	84.1	0.56	79.2	0.55	85.5	0.50

Appendix

Table 32: Results for the wall-clock-cost solvers, reported separately from the main results because their tool cost is wall-clock time on fixed hardware rather than an analytical operation count, and is therefore not comparable across solvers. Abbreviations: S - Success Rate (%), E - Efficiency. Low/Med/High - accuracy levels. Measurements reported for the single-round inference mode.

Solver	Model/Acc level	Low		Med		High		Ave
	Metrics	S	E	S	E	S	E	S	E
EPOCH 1D	GPT-5	80.7	0.25	73.9	0.22	78.3	0.14	77.7	0.20
	Claude	94.1	0.80	81.4	1.03	67.6	0.94	81.0	0.92
	Llama	75.6	0.39	70.4	0.51	67.7	0.56	71.2	0.48
	GPT-OSS	97.8	2.84	94.8	3.30	92.8	4.71	95.1	3.53

Appendix

Table 33: Results for the wall-clock-cost solvers, reported separately from the main results because their tool cost is wall-clock time on fixed hardware rather than an analytical operation count, and is therefore not comparable across solvers. Abbreviations: S - Success Rate (%), E - Efficiency. Low/Med/High - accuracy levels. Measurements reported for the multi-round inference mode.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,497 papers

WorldSolver: Can LLM Agents Simulate the Physical Dynamics via Solver Generation?
Oct 6, 2026Siru Jiang, Yongzhe Lyu, Shuo Lu +6Physics-Based Simulation Code Generation
NLPR & MAIS, CASIA · PKU · Huawei Noah’s Ark Lab
EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation
May 8, 2026Yi Liu, TingFeng Hui, Wei Zhang +4AI Agent Evaluation AI Agent Benchmarks
Beijing University of Posts and Telecommunications · The Hong Kong University of Science and Technology · Chongqing University
Solving Physics Olympiad via Reinforcement Learning on Physics Simulators
Apr 13, 2026Mihir Prabhudesai, Aryan Satpathy, Yangmin Li +6Scientific Reasoning in Language Models Synthetic Data Generation for Language Models
Carnegie Mellon University

Oct 6, 2026J/K move · Enter open · S save

WorldSolver: Can LLM Agents Simulate the Physical Dynamics via Solver Generation?

Siru Jiang, Yongzhe Lyu, Shuo Lu, Yubin Wang+5

NLPR & MAIS, CASIA · PKU · Huawei Noah’s Ark Lab

Physics-Based Simulation Code Generation AI Agent Benchmarks LLM Agent Evaluation

Open paper AskPDF arXiv

LLM-based agents are increasingly advancing scientific and engineering problem solving, with physics simulation emerging as a challenging yet practical testbed for reproducing complex physical phenomena with application in embodied AI, games and films. As the workhorse of such simulation, a solver computes how the state of a dynamic system evolves over time. Building such solvers requires physical understanding to identify appropriate models, mathematical reasoning to formulate the underlying dynamics, and software engineering to implement them as executable code, yet this capability of LLM agents remains underexplored. To this end, we introduce WorldSolver, a benchmark of 168 simulation tasks derived from physical phenomena in 61 classic computer graphics papers, spanning 7 physical domains. Each task contains a code scaffold that provides a fixed simulation environment for the scene, with the solver implementation left for the agent to complete. Specifically, we evaluate them along three dimensions: Execution Checks for successful execution, Visual Fidelity for reproducing the intended dynamic behavior in the rendered simulation, and Physical Plausibility for physics-grounded verification of the generated dynamics. Experiments on frontier agents reveal that producing executable solvers is difficult itself, and satisfying visual and physical correctness is even harder. GPT-5.6-Sol and Claude-Opus-5 perform comparatively better than the other evaluated agents, yet achieve overall scores of only 48.7% and 46.7%, respectively. WorldSolver is an early step toward agentic solver generation, and we hope it helps drive progress toward agents that can faithfully simulate the dynamic physical world. Code is available at https://github.com/sirujiang/WorldSolver.

Abstract

AI Agent Benchmarks

Explore similar work

WorldSolver: Can LLM Agents Simulate the Physical Dynamics via Solver Generation?

EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation

Solving Physics Olympiad via Reinforcement Learning on Physics Simulators