Organizations: University of California San Diego · The Chinese University of Hong Kong, Shenzhen · Peking University · University of California, Los Angeles · ETH Zurich · California Institute of Technology
Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental resources. As a result, metrics like pass@k become impractical under realistic budget constraints. To address this gap, we introduce SimulCost, the first benchmark targeting cost-sensitive parameter tuning in physics simulations. SimulCost compares LLM tuning cost-sensitive parameters against traditional scanning approach in both accuracy and computational cost, spanning 2,643 single-round (initial guess) and 2,304 multi-round (adjustment by trial-and-error) tasks across 11 simulators from fluid dynamics, solid mechanics, and plasma physics, whose costs are analytically defined and platform-independent. A twelfth simulator, a production plasma code measurable only by wall clock, is reported separately. Frontier LLMs achieve 45-62% success rates in single-round mode, dropping to 34-50% under high accuracy requirements, rendering their initial guesses unreliable especially for high accuracy tasks. Multi-round mode improves rates to 66-81%, but LLMs are 1.5-2.7x slower than traditional scanning, making them uneconomical choices. We also investigate parameter group correlations for knowledge transfer potential, and the impact of in-context examples and reasoning effort, providing practical implications for deployment and fine-tuning. We open-source SimulCost as a static benchmark and extensible toolkit to facilitate research on improving cost-aware agentic designs for physics simulations, and for expanding new simulation environments. Code and data are available at https://github.com/Rose-STL-Lab/SimulCost-Bench
Figures & tables
Figure 1: Overview of SimulCost . Our benchmark evaluates LLM agents on cost-sensitive parameter tuning across 12 physics simulators spanning fluid dynamics, solid mechanics, and plasma physics. Given a simulation task, tuning mode, and accuracy requirement, the LLM proposes tunable parameters in either single-round (initial guess) or multi-round (trial-and-error) mode. The challenge is balancing accuracy and cost: overly coarse parameters fail accuracy requirements, while overly fine parameters waste computation.
Benchmark
Domain
Tool Cost
Platform Indep.
Toolkit
TaskBench ( Shen et al., 2024 )
General Tools
✗
–
✗
CATP-LLM ( Wu et al., 2025a )
General Tools
✓
✗
✗
ScienceAgentBench ( Chen et al., 2025b )
Data Science
✗
–
✗
DiscoveryBench ( Majumder et al., 2024 )
Data Science
✗
–
✗
Auto-Bench ( Chen et al., 2025a )
Causal Discovery
✓
✓
✗
MLE-Bench ( Chan et al., 2025 )
ML Engineering
✓ ∗
✗
✗
Table 1: Comparison of benchmarks for LLM agents in scientific and engineering domains. Tool Cost: whether it tracks computational costs of external tools (separate from LLM inference). Platform Indep.: whether tool cost measurement is platform-independent (not based on wall time). Toolkit: whether it provides an extensible simulation environment. –: not applicable (no tool cost). ∗ Ambiguous: costs are reported but mixed with other components or based on wall time. † Claims computational cost but reports only LLM token costs in USD.
Figure 2: Overall performance across models, accuracy levels, and inference modes. SR = Single-Round, MR = Multi-Round; High/Med/Low = accuracy level requirements.
Figure 3: Key findings. (a) Forest plot of multi-round improvement by model and accuracy; all differences significant at p<0.05 except Claude-3.7-Sonnet at low accuracy. (b)–(c) Task group summary: aggregated success rate and efficiency by parameter type; single-round (solid) vs. multi-round (hollow). (d) Task correlations: within-group vs. between-group shows no significant difference. (e)–(f) In-context learning (ICL) variants improve single-round success but degrade multi-round; cost information in examples preserves efficiency.
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
Solver
Single
Multi
Single-only Params
Both Params
Low
Med
High
Analytical cost (main results)
Burgers 1D
339
339
–
beta , cfl , k , n_space
8.0×10−2
4.0×10−2
1.0×10−2
Diff-React 1D
90
90
–
cfl , n_space , tol
vary ∗
vary ∗
vary ∗
Euler 1D
197
197
–
beta , cfl , k , n_space
8.0×10−2
2.0×10−2
1.0×10−2
Euler 2D
232
232
–
cfl , cg_tolerance , n_grid_x
8.0×10−2
4.0×10−2
8.0×10−3
FEM 2D
103
103
–
cfl , dx
vary ∗
vary ∗
vary ∗
Appendix
Table 2: Summary of solvers in the SimulCost benchmark. Single/Multi columns show task counts per mode. Single-only parameters are used exclusively in single-round evaluation; Both parameters support both modes. Low/Med/High columns show accuracy thresholds. ∗ Thresholds vary by task type. The upper block defines the tool cost analytically by counting dominant-operation FLOPs and carries the main results; the lower block is a compiled binary whose FLOPs cannot be estimated in closed form, so its cost is wall-clock time on fixed hardware, which is not comparable across solvers and is reported separately in Appendix C.15 . See Appendix A.2 for physics background and citations.
Table 4: The overall results across the 11 solvers with analytical costs. Abbreviations: S - Success Rate (%), E - Efficiency. Low/Med/High - accuracy levels. Measurements reported for the single-round inference mode.
Model/Acc level
Low
Med
High
Ave
Metrics
S
E
S
E
S
E
S
E
GPT-5
85.5
0.69
80.9
0.59
75.9
0.72
80.8
0.66
Claude-3.7-Sonnet
65.6
0.37
66.0
0.32
66.6
0.43
66.1
0.37
Llama-3-70B-Instruct
69.2
0.40
73.2
0.38
69.5
0.54
70.6
0.43
Qwen3-32B
71.0
0.66
67.7
0.49
64.2
0.65
67.7
0.60
GPT-OSS-120B
74.2
0.86
78.1
0.54
79.1
0.64
77.1
0.67
Appendix
Table 5: The overall results across the 11 solvers with analytical costs. Abbreviations: S - Success Rate (%), E - Efficiency. Low/Med/High - accuracy levels. Measurements reported are for multi-round tunable parameters only.
Model / Acc level
Low
Med
High
Ave
Inference Modes
SR
MR
SR
MR
SR
MR
SR
MR
GPT-5
69.8
85.5
64.9
80.9
49.9
75.9
61.5
80.8
Claude-3.7-Sonnet
65.2
65.6
51.8
66.0
41.0
66.6
52.7
66.1
Llama-3-70B-Instruct
58.2
69.2
41.9
73.2
34.4
69.5
44.8
70.6
Qwen3-32B
61.6
71.0
42.8
67.7
34.6
64.2
46.3
67.7
GPT-OSS-120B
64.7
74.2
47.1
78.1
33.8
79.1
48.5
77.1
Appendix
Table 6: Comparison between single-round and multi-round inference modes: Success Rate . Abbreviations: Low/Med/High - accuracy levels, SR/MR - single-round and multi-round modes.
Model / Acc level
Low
Med
High
Ave
Inference Modes
SR
MR
SR
MR
SR
MR
SR
MR
GPT-5
0.49
0.69
0.38
0.59
0.33
0.72
0.39
0.66
Claude-3.7-Sonnet
0.39
0.37
0.36
0.32
0.37
0.43
0.38
0.37
Llama-3-70B-Instruct
0.16
0.40
0.17
0.38
0.29
0.54
0.20
0.43
Qwen3-32B
0.46
0.66
0.39
0.49
0.33
0.65
0.39
0.60
GPT-OSS-120B
0.52
0.86
0.39
0.54
0.43
0.64
0.44
0.67
Appendix
Table 7: Comparison between single-round and multi-round inference modes: Efficiency . Abbreviations: Low/Med/High - accuracy levels, SR/MR - single-round and multi-round modes.
Model / Accuracy
Single
Multi
Δ
p -value
GPT-5
63.5%
88.7%
+25.2%
< 0.001
Claude-3.7-Sonnet
57.7%
66.5%
+8.8%
0.068
Llama-3-70B-Instruct
52.4%
79.6%
+27.3%
< 0.001
Qwen3-32B
47.6%
71.2%
+23.6%
< 0.001
GPT-OSS-120B
52.8%
83.1%
+30.3%
< 0.001
Overall (High)
46.8%
77.4%
+30.5%
< 0.001
Appendix
Table 8: Multi-round improvement over single-round (McNemar’s test). Single/Multi = success rate (%); Δ = improvement. Top: per-model averages across accuracy levels. Bottom: overall by accuracy level.
Model
Effect
SE
p -value
Dataset Var.
All
+21.3%
0.006
< 0.001
0.041
GPT-5
+24.7%
0.011
< 0.001
0.000
Claude-3.7-Sonnet
+7.0%
0.014
< 0.001
0.000
Llama-3-70B
+25.2%
0.012
< 0.001
0.000
Qwen3-32B
+22.2%
0.012
< 0.001
0.000
GPT-OSS-120B
+27.4%
0.012
< 0.001
0.000
Appendix
Table 9: Random-intercept linear probability model, success ∼ mode +(1∣dataset) . Effect is the difference in mean success between modes. The per-model fits return a dataset variance of exactly zero, a boundary solution the optimiser reports as singular, so they detect no between-dataset variation rather than establishing its absence.
Model
Accuracy
Δ SR
p (uncorr.)
p (FDR)
Sig.
GPT-5
Low
+23.5%
< 0.001
< 0.001
Yes
GPT-5
Medium
+23.1%
< 0.001
< 0.001
Yes
GPT-5
High
+29.0%
< 0.001
< 0.001
Yes
Claude-3.7-Sonnet
Low
-3.3%
0.068
0.068
No
Claude-3.7-Sonnet
Medium
+11.7%
< 0.001
< 0.001
Yes
Claude-3.7-Sonnet
High
+18.1%
< 0.001
< 0.001
Yes
Appendix
Table 10: McNemar’s test with Benjamini-Hochberg FDR correction. With 15 comparisons (5 models × 3 accuracy levels), we apply FDR correction at α=0.05 . All comparisons remain significant after correction except Claude-3.7-Sonnet at low accuracy.
Single-Round
Multi-Round
Dataset
Base
ICL
No-Cost
Mixed
Base
ICL
No-Cost
Mixed
Euler 1D
38.1
50.3
67.0
66.4
50.8
39.8
39.7
39.8
Heat 1D
69.9
75.0
85.9
86.6
87.7
81.0
85.0
72.8
MPM 2D
34.3
49.7
63.0
71.3
35.2
36.4
35.4
28.7
NS Trans. 2D
38.3
68.3
73.4
71.2
78.5
72.7
70.3
69.7
Appendix
Table 11: ICL performance breakdown by dataset and inference mode. Values show success rate (%) averaged across accuracy levels. Base = no ICL; ICL = accuracy-matched examples with cost; No-Cost = ICL without cost info; Mixed = examples from all accuracy levels.
Figure 4: Detailed ICL ablation results. (a) Success rate improves with ICL in single-round mode but degrades in multi-round. (b) Efficiency drops when cost information is omitted from examples (No-Cost variant).
Single-Round
Multi-Round
Effort Level
Low
Med
High
Low
Med
High
S
E
S
E
S
E
S
E
S
E
S
E
Minimal
74.2
0.39
60.5
0.29
48.0
0.19
81.7
0.66
84.5
0.50
84.8
0.70
Medium
68.1
0.43
62.1
0.28
50.0
0.15
86.5
0.71
81.8
0.52
75.1
0.64
High
70.4
0.44
60.8
0.29
58.6
0.19
87.9
0.62
78.3
0.51
76.0
0.56
Appendix
Table 12: GPT-5 reasoning effort ablation by accuracy level and inference mode. S = Success Rate (%), E = Efficiency. Bold indicates best per column.
Figure 5: Reasoning effort ablation by accuracy level. Left: Single-round mode. Right: Multi-round mode. No consistent pattern emerges—increased reasoning effort does not reliably improve parameter selection.
Method
Success Rate
Rel. Cost
Brute-force
93.5%
1.00
Bisection
97.8%
1.08
Nelder-Mead 1D
97.8%
1.42
Appendix
Table 13: Non-LLM optimization baselines on monotonic multi-round parameters.
Model
Eff (SR)
RC (SR)
Eff (MR)
RC (MR)
GPT-OSS-120B
0.44
0.49
0.67
0.91
Qwen3-32B
0.39
0.57
0.60
0.98
GPT-5
0.39
1.17
0.66
1.18
Claude-3.7-Sonnet
0.38
0.72
0.37
2.08
Llama-3-70B
0.20
1.11
0.43
2.89
Appendix
Table 14: Conditional efficiency (Eff) vs. unconditional relative cost (RC). RC < 1 means total LLM cost is below brute-force including failed runs.
Mode
Success Rate Shift
Efficiency Shift
Single-round
+0.069
− 0.047
Multi-round
+0.072
− 0.090
Appendix
Table 15: Prompt sensitivity: mean shift from original results across 25 variants.
Table 18: Bayesian Optimization vs LLM performance by solver (multi-round mode). SR = Success Rate (%), Eff = Efficiency. LLM Avg is the mean across GPT-5, Claude, Llama, Qwen, and GPT-OSS.
BO
LLM Avg
Solver
Low
Med
High
Low
Med
High
Burgers 1D
0.36
0.41
0.67
0.78
0.70
0.85
Diff-React 1D
0.30
0.97
0.64
1.20
1.18
1.00
Euler 1D
0.99
1.08
1.18
1.21
0.78
0.79
Euler 2D
0.60
0.62
0.57
0.42
0.38
0.49
FEM 2D
0.30
0.20
0.48
0.46
0.51
0.73
Appendix
Table 19: Bayesian Optimization vs LLM efficiency by solver (multi-round mode). Values show cost efficiency (higher is better; 1.0 = brute-force search cost).
Group
Parameters
Description
Spatial
n_space , dx , nx , n_grid_x , …
Grid/mesh resolution parameters affecting spatial discretization accuracy. Most common across all solvers.
Temporal
cfl , dt
Timestep and CFL parameters controlling temporal stability and accuracy.
Convergence tolerances for iterative solvers to control solution accuracy.
Misc
beta , k , t_init , relax , …
Physics-specific parameters (limiter coefficients, relaxation factors) varying by solver.
Appendix
Table 20: Task group classification. Parameters are grouped by their role in numerical simulation, enabling analysis of LLM performance across parameter types.
Figure 6: Performance by task group. Spatial and Tolerance parameters show consistently strong performance, while Misc parameters lag especially at higher accuracy levels.
Figure 7: Average Pearson correlation between task groups. Diagonal = within-group; off-diagonal = between-group.
Figure 8: Full pairwise task correlation matrix. Tasks grouped by parameter type: Spatial, Temporal, Tolerance, Misc. Black lines separate groups.
Model
Success Rate (%)
Efficiency
Single
Multi
Δ
Single
Multi
GPT-5
61.5
80.8
+19.2
0.39
0.66
Claude-3.7-Sonnet
52.7
66.1
+13.4
0.38
0.37
Llama-3-70B-Instruct
44.8
70.6
+25.8
0.20
0.43
Qwen3-32B
46.3
67.7
+21.3
0.39
0.60
GPT-OSS-120B
48.5
77.1
+28.6
0.44
0.67
Appendix
Table 21: Overall model performance (corresponds to Figure 2 ).
Task Group
Success Rate (%)
Efficiency
Single
Multi
Δ
Single
Multi
Spatial
55.5
75.7
+20.2
0.28
0.62
Temporal
57.3
73.8
+16.5
0.42
0.45
Tolerance
59.4
76.5
+17.2
0.32
0.46
Misc
32.7
61.1
+28.4
0.86
0.77
Appendix
Table 22: Performance by task group (corresponds to Figure 6 ).
Variant
Success Rate (%)
Efficiency
Single
Multi
Δ
Single
Multi
Base
46.7
64.1
+17.3
0.328
0.367
Idealized ICL
63.8
53.2
-10.6
0.445
0.377
Cost-Ignorant ICL
73.4
53.1
-20.3
0.364
0.470
Mixed-Accuracy ICL
75.4
46.3
-29.1
0.572
0.435
Appendix
Table 23: ICL variant comparison using Claude-3.7-Sonnet (corresponds to Figure 3 ).
Table 32: Results for the wall-clock-cost solvers, reported separately from the main results because their tool cost is wall-clock time on fixed hardware rather than an analytical operation count, and is therefore not comparable across solvers. Abbreviations: S - Success Rate (%), E - Efficiency. Low/Med/High - accuracy levels. Measurements reported for the single-round inference mode.
Solver
Model/Acc level
Low
Med
High
Ave
Metrics
S
E
S
E
S
E
S
E
EPOCH 1D
GPT-5
80.7
0.25
73.9
0.22
78.3
0.14
77.7
0.20
Claude
94.1
0.80
81.4
1.03
67.6
0.94
81.0
0.92
Llama
75.6
0.39
70.4
0.51
67.7
0.56
71.2
0.48
GPT-OSS
97.8
2.84
94.8
3.30
92.8
4.71
95.1
3.53
Appendix
Table 33: Results for the wall-clock-cost solvers, reported separately from the main results because their tool cost is wall-clock time on fixed hardware rather than an analytical operation count, and is therefore not comparable across solvers. Abbreviations: S - Success Rate (%), E - Efficiency. Low/Med/High - accuracy levels. Measurements reported for the multi-round inference mode.
LLM-based agents are increasingly advancing scientific and engineering problem solving, with physics simulation emerging as a challenging yet practical testbed for reproducing complex physical phenomena with application in embodied AI, games and films. As the workhorse of such simulation, a solver computes how the state of a dynamic system evolves over time. Building such solvers requires physical understanding to identify appropriate models, mathematical reasoning to formulate the underlying dynamics, and software engineering to implement them as executable code, yet this capability of LLM agents remains underexplored. To this end, we introduce WorldSolver, a benchmark of 168 simulation tasks derived from physical phenomena in 61 classic computer graphics papers, spanning 7 physical domains. Each task contains a code scaffold that provides a fixed simulation environment for the scene, with the solver implementation left for the agent to complete. Specifically, we evaluate them along three dimensions: Execution Checks for successful execution, Visual Fidelity for reproducing the intended dynamic behavior in the rendered simulation, and Physical Plausibility for physics-grounded verification of the generated dynamics. Experiments on frontier agents reveal that producing executable solvers is difficult itself, and satisfying visual and physical correctness is even harder. GPT-5.6-Sol and Claude-Opus-5 perform comparatively better than the other evaluated agents, yet achieve overall scores of only 48.7% and 46.7%, respectively. WorldSolver is an early step toward agentic solver generation, and we hope it helps drive progress toward agents that can faithfully simulate the dynamic physical world. Code is available at https://github.com/sirujiang/WorldSolver.
Scalable AI agents training relies on interactive environments that faithfully simulate the consequences of agent actions. Manually crafted environments are expensive to build, brittle to extend, and fundamentally limited in diversity. A promising direction is to replace manually crafted environments with LLM-simulated counterparts. However, this paradigm hinges on an unexamined core assumption: LLMs can accurately simulate environmental feedback. In practice, LLM-simulated environments suffer from hallucinations, logical inconsistencies, and silent state drift failures that corrupt agent reward signals and compound the construction costs that the paradigm was designed to eliminate. To address this gap, we propose EnvSimBench with four contributions: 1) We provide the first formal definition and operationalization of Environment Simulation Ability (EnvSim Ability) as a quantifiable research objective. 2) We construct EnvSimBench, a rigorous benchmark covering 400 samples across 167 diverse environments, equipped with verifiable labels and fine-grained difficulty stratification along three axes. 3) Systematic evaluations reveal that all state-of-the-art language models suffer from a universal state change cliff: they achieve near-perfect accuracy on tasks when the environment state remains invariant, yet fail catastrophically when multiple states need simultaneous updates. This finding exposes EnvSim Ability as a critical yet largely unaddressed capability gap. 4) We design a constraint-driven simulation pipeline that substantially reduces hallucination, boosts environment synthesis yield by 6.8%, and cuts costs by over 90%. Overall, EnvSimBench serves as both a diagnostic framework and a practical optimization path for reliable LLM-based environment simulation, establishing a foundation for scalable agent training. Code and data are available at https://github.com/cookieApril/EnvSimBench
Yi Liu, TingFeng Hui, Wei Zhang +4
Beijing University of Posts and Telecommunications · The Hong Kong University of Science and Technology · Chongqing University
We have witnessed remarkable advances in LLM reasoning capabilities with the advent of DeepSeek-R1. However, much of this progress has been fueled by the abundance of internet question-answer (QA) pairs, a major bottleneck going forward, since such data is limited in scale and concentrated mainly in domains like mathematics. In contrast, other sciences such as physics lack large-scale QA datasets to effectively train reasoning-capable models. In this work, we show that physics simulators can serve as a powerful alternative source of supervision for training LLMs for physical reasoning. We generate random scenes in physics engines, create synthetic question-answer pairs from simulated interactions, and train LLMs using reinforcement learning on this synthetic data. Our models exhibit zero-shot sim-to-real transfer to real-world physics benchmarks: for example, training solely on synthetic simulated data improves performance on IPhO (International Physics Olympiad) problems by 5-10 percentage points across model sizes. These results demonstrate that physics simulators can act as scalable data generators, enabling LLMs to acquire deep physical reasoning skills beyond the limitations of internet-scale QA data. Code available at: https://sim2reason.github.io/.