We developed CAE-Bench, a benchmark to evaluate AI agents on Computer-Aided Engineering (CAE) solver development. A CAE solver is more than a PDE solver; it must be able to generalize across various inputs, such as different boundary conditions, geometries, and material properties. CAE-Bench evaluates the generalizability of AI-developed CAE solvers by providing the same task, such as a heat transfer problem, multiple cases, with each case specifying an input. The AI agent develops a solver for a fixed task using only a few public cases, and the resulting solver is then tested on held-out cases, i.e., cases it has never seen before. We evaluate 12 models across a range of reasoning settings on 80 computational tasks. Our experiments show that even if a solver developed by an AI agent performs well on the public cases, it may even get 0 score on held-out cases. For tasks such as fluid dynamics problems, even the solver developed by GPT-6 Astra performs poorly on held-out cases. A data format and an SDK for task registration are provided, so the researcher can use the benchmark on their own infrastructure. We hope that CAE-Bench will help the community to develop AI agents that can develop reusable CAE solvers that generalize to various inputs.
Fluid domain; positive constant density and viscosity
Conditions
Supported mechanical boundary conditions, body forcing, and pressure conventions
Condition types, boundary assignments, values, and applicable pressure constraint
Observations
Definitions and units of velocity, pressure, and other supported quantities
Requested quantities, locations, and point, average, or integral observations
Table 1: Cases for the 2-dimensional steady incompressible Newtonian-flow task.
Figure 1: The workflow of CAE-Bench
Task category
Tasks
Evaluation cases
Public
Held out
Prescribed PDEs
13
266
3
263
Fluid mechanics
43
340
43
297
Fixed-problem tasks
24
24
24
0
Computational total
80
630
70
560
Infrastructure tests
2
17
1
16
Table 2: Number of tasks and cases of CAE-Bench (RC7).
Figure 2: Input variations for periodic advection and generalized-Newtonian flow tasks.
Task category
Source
Reference preparation
Heat conduction (2 tasks, 56 cases)
Manufactured solutions and analytical series
Evaluate the prescribed fields at the requested nodes and times.
PDEBench adaptations (11 tasks, 210 cases)
Source trajectories and recomputations described in the accompanying records
Match the equations, initial states, time coordinates, units, and observations to the task.
Basic steady flow (2 tasks, 20 cases)
Analytical solutions, Lethe, and Elmer
Evaluate analytical fields or convert numerical solutions into the requested observations.
Other fluid-mechanics tasks (41 tasks, 320 cases)
OpenFOAM, Elmer, SPARTA, PyClaw, Thetis, and formula evaluation
Evaluate formulas or convert solver fields into the requested values, integrals, or statistics.
Fixed-problem adaptations (24 tasks, 24 cases)
Analytical representations, NGSolve, Dedalus, and OpenFOAM
Apply the problem definitions described in the accompanying records. Export the requested observations.
Table 3: Reference sources and preparation for the computational suite.
Figure 3: Case outcomes across the 80 computational tasks in RC7.
Model
Reasoning effort
Varying physical inputs (56)
PDE (13)
Fluid mechanics (43)
Fixed-problem (24)
Overall
GPT-5.6 Luna
medium
29.31
57.39
20.82
68.11
40.95
GPT-6 Luna
medium
36.08
79.13
23.06
75.22
47.82
GPT-5.6 Sol
medium
44.37
87.27
31.40
97.22
60.23
GPT-6 Sol
medium
39.90
72.58
30.02
100.00
57.93
GPT-6.1 Sol
medium
50.35
83.75
40.25
100.00
65.24
GPT-6 Astra
medium
49.79
82.69
39.85
100.00
64.85
Table 4: Model scores on the 80 computational tasks in RC7. For each model, we average case scores within each task and give equal weight to the evaluated tasks in each column. Scores are on a 0–100 scale.
Figure 4: Comparison of reasoning-effort settings for GPT-5.6 Luna and GPT-6 Luna on 80 computational tasks. The tasks include 56 with varying physical inputs and 24 fixed-problem tasks. Each task has equal weight in the average score.
Figure 5: Recorded API costs are in USD per task on a logarithmic scale.
Figure 6: Public and held-out performance on tasks with varying physical inputs. (a) For tasks with both case types, we average public and held-out scores separately within each task and then average across tasks with equal weight. (b) Each bar shows the distribution of held-out case outcomes on tasks where every public case received full score. The numbers on the right give the held-out case count for each bar.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
File under supplement/
Contents
reference_records.json
Reference data index and source
evidence/
Record of reference data making process
experiment_results.json
Experiment results and model configurations
heldout_losses.json
Held-out scores (tasks with full credit on all public cases)
score_group_diagnostics.json
Report of each group (errors and checks)
score_sensitivity.json
Result under different scoring rules
Appendix
Table 5: Supplementary files provided with CAE-Bench.
Model
RMS 4%/8%
RMS 5%/10%
RMS 6%/12%
Max 5%/10%
GPT-5.6 Luna
28.52
29.31
29.99
27.88
GPT-6 Luna
35.41
36.08
36.60
34.16
GPT-5.6 Sol
42.56
44.37
46.46
38.83
GPT-6 Sol
38.16
39.90
41.22
34.43
GPT-6.1 Sol
47.26
50.35
52.70
43.01
GPT-6 Astra
46.93
49.79
52.05
42.77
Appendix
Table 6: Score sensitivity on tasks with varying physical inputs for the 12 cases in Table 4 . Each entry averages the task scores with equal task weights. Scores use a 0–100 scale. We recompute the scores from stored group errors for the same submitted programs. Column headings give the full-score and zero-score thresholds, respectively. The bold column uses the official RMS 5%/10% scoring rule. Failures of execution, output-format checks, and exact-value checks retain 0 score.
Varying physical inputs
Fixed problems
Model
Effort
Score
USD/task
Score
USD/task
Overall
Reports
GPT-5.6 Luna
none
26.25
0.0193
54.17
0.0110
34.63
80/80
low
25.80
0.0133
62.50
0.0103
36.81
80/80
medium
29.31
0.0216
68.11
0.0154
40.95
80/80
high
35.49
0.0666
95.77
0.0352
53.58
80/80
xhigh
39.67
0.1346
100.00
0.0545
57.77
80/80
Appendix
Table 7: Scores and mean recorded API costs on RC7. Scores are on a 0–100 scale. The last column gives the number of valid task reports out of 80.
Model
Tasks with full score on all public cases
Mean held-out score (0–100)
GPT-5.6 Luna
15
57.35
GPT-6 Luna
17
60.98
GPT-5.6 Sol
22
59.16
GPT-6 Sol
20
60.13
GPT-6.1 Sol
22
66.97
GPT-6 Astra
22
67.44
Appendix
Table 8: Mean held-out scores after full score on all public cases for the cases in Table 4 .
Computer-aided engineering (CAE) simulation is among the largest and most demanding areas of engineering, where setting up a solver such as OpenFOAM, FEniCS, or COMSOL takes real expertise. Large language model (LLM) agents promise to turn a natural-language request into a working simulation, and recent CAE agents add simulation-specific machinery: multi-agent decomposition, domain retrieval, and scripted reflection. That machinery suited weak base models; modern harnesses already supply multi-turn reasoning, tool use, and execution feedback. We ask what a CAE simulation agent still needs beyond a generic harness. With information access and repair budget held fixed, a single-agent harness matches or beats multi-agent specialized systems (FoamBench 96.4% vs.\ 88.2%). Ablations trace this to capabilities the harness already provides: execution-feedback repair lifts FoamBench from 71.8% with no repair round to 96.4%, while scripted reflection adds nothing. The one input that still helps is domain knowledge supplied as solver tutorials, our largest measured gain (80.9% to 96.4%).
Jiasheng Shi, Tianhan Zhang
DP Technology · Beijing, China · School of Astronautics, Beihang University +1
PDE-to-solver code generation aims to automatically synthesize executable numerical solvers from partial differential equation (PDE) specifications. This task requires not only understanding the mathematical structure of PDEs, but also selecting appropriate discretization schemes and solver configurations, and correctly implementing the resulting formulations in finite-element method (FEM) libraries. Existing code generation benchmarks mainly evaluate syntactic correctness, or success on predefined test cases. To our knowledge, there is currently no publicly available benchmark specifically for PDE-to-solver code generation, and general-purpose code benchmarks do not fully capture the unique challenges of numerical PDE solution, such as ensuring solver accuracy, efficiency, and compatibility with professional FEM libraries. We introduce PDEAgent-Bench, to the best of our knowledge, the first multi-metric, multi-library benchmark for PDE-to-solver code generation. PDEAgent-Bench contains 645 instances across 6 mathematical categories and 11 PDE families, with common FEM libraries for DOLFINx, Firedrake, and deal.II. Each instance provides an agent-facing problem specification, a reference solution on a prescribed evaluation grid, and case-specific accuracy and runtime targets. PDEAgent-Bench adopts a staged evaluation framework in which generated solvers must sequentially pass executability, numerical accuracy, and computational efficiency checks. Experiments with representative LLMs and code agents show that models can often produce runnable code, but their pass rate drops substantially once accuracy and efficiency requirements are enforced. These results indicate that current agents remain limited in producing numerically reliable and efficient PDE solvers, and that PDEAgent-Bench provides a reproducible testbed grounded in the practical requirements of numerical PDE solving.
Zhen Hang, Yushan Yashengjiang, Junhui Li +21
University of Science and Technology of China · Beijing University of Posts and Telecommunications · Tencent +10
The AEC-Bench is a multimodal benchmark for evaluating agentic systems on real-world tasks in the Architecture, Engineering, and Construction (AEC) domain. The benchmark covers tasks requiring drawing understanding, cross-sheet reasoning, and construction project-level coordination. This report describes the benchmark motivation, dataset taxonomy, evaluation protocol, and baseline results across several domain-specific foundation model harnesses. We use AEC-Bench to identify consistent tools and harness design techniques that uniformly improve performance across foundation models in their own base harnesses, such as Claude Code and Codex. We openly release our benchmark dataset, agent harness, and evaluation code for full replicability at https://github.com/nomic-ai/aec-bench under an Apache 2 license.