Existing Simulink benchmarks mainly evaluate whether generated models compile, execute, or resemble a reference model. These criteria do not establish whether a model satisfies its engineering requirements. We introduce SimuVerity, a benchmark of 101 text-to-executable Simulink model-generation tasks across ten engineering domains. For each task, executable-system profiles ground the engineering specification and four families of native simulation scenarios. A hierarchical evaluator first checks artifact delivery, native executability, and engineering qualification, then scores qualified models across six dimensions covering accuracy, output quality, mechanistic fidelity, control and causal integrity, operating-domain robustness, and dynamic response. We evaluate six agent systems with SimuVerity. The best system achieves an overall score of only 42.86. The results show that structural similarity is a poor proxy for engineering performance: capability bottlenecks arise both in producing qualified implementations and in satisfying multidimensional requirements after qualification. Meanwhile, some high-scoring models still exhibit severe visual-layout disorder. SimuVerity provides a systematic basis for assessing agents' engineering capabilities and diagnosing failures in executable Simulink model generation.
Figures & tables
Figure 1: Task composition and agent performance in SimuVerity . (a) Task distribution across ten engineering domains; outer-ring line lengths indicate mean domain scores. (b) Six-dimensional performance and overall scores of six agents.
Figure 2: The SimuVerity pipeline. Benchmark construction, native simulation testing, and six-dimensional performance evaluation.
Engineering domains (101 tasks)
Power electronics & control
9
Automotive systems
11
Mechanical systems
11
Battery systems
10
Robotics
10
Aerospace systems
10
Thermal systems
11
Communication systems
9
Fluid systems
11
Biomedical systems
9
Table 1: Task distribution across the ten engineering domains of SimuVerity .
Figure 3: Evaluation of the closed-loop tiltrotor VTOL model. Native simulation evidence, six-dimensional scores, and failure diagnosis for the candidate model. Four of six scenarios are shown.
Agent system
Overall
Delivered
Executable
G pass
Opus 4.8 + Claude Code
42.86
94.72%
86.47%
76.90%
GPT-5.5 + Codex
41.72
98.68%
94.72%
75.58%
DeepSeek-V4-Pro + Claude Code
28.40
97.03%
88.12%
61.39%
Qwen3.8-Max + Claude Code
25.01
89.77%
80.86%
53.80%
GLM-5.3-Flash + Claude Code
4.98
96.04%
48.18%
13.20%
Qwen3.8-27B-FP8 + Claude Code
1.60
30.36%
13.53%
6.27%
Table 2: Overall scores and prerequisite pass rates. Agent results are three-run means; Reference reports the mean score of 101 task-specific reference systems.
Agent system
Scope
A
Q
M
C
R
D
Opus 4.8 + Claude Code
End-to-end
47.95
63.24
52.34
48.82
46.72
45.90
Post-G
62.28
82.21
65.25
58.92
60.73
58.65
GPT-5.5 + Codex
End-to-end
48.80
61.94
56.34
44.87
46.05
44.81
Post-G
64.57
81.96
72.43
53.62
60.92
59.07
DeepSeek-V4-Pro + Claude Code
End-to-end
36.08
49.31
38.05
38.57
33.75
33.93
Post-G
58.78
80.32
59.32
57.59
55.07
54.50
Table 3: End-to-end and post-G performance on the six engineering dimensions. Each dimension is averaged over its registered applicable-task set.
Structural relation
n
G pass
End-to-end
Post-G
Top quartile
Zero
Reference-Aligned
39
87.18%
52.13
59.80
12
7
Partially Similar
43
83.72%
44.05
52.61
9
10
Structurally Distinct
13
69.23%
42.37
61.20
3
4
Table 4: Structural relation and engineering performance. Results cover 95 delivered Opus Run 1 candidates; post-G means include only candidates that pass engineering qualification.
Figure 4: Robotic ball-balancing system before and after layout optimization. Only block positions and existing line routes are adjusted; the model implementation remains unchanged.
Tool condition
Artifact delivery
Native executability
G pass
Overall score
Full MCP
100.00%
100.00%
100.00%
78.46
No-Simulation MCP
53.33%
33.33%
13.33%
7.88
Batch-only
60.00%
50.00%
40.00%
18.23
Table 5: Ablations of native simulation feedback and tool access. Results report three-run mean prerequisite pass rates and overall scores for Claude Opus 4.8 with Claude Code on ten cross-domain, high-performing tasks.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Work
Task scope
Native execution
Evaluation focus
SLGPT ( Shrestha & Csallner, 2021 )
Simulink file generation
Toolchain execution
Fault discovery
SimuGen ( Ren et al., 2025 )
Image-to-model reconstruction
Basic execution
Blocks and connections
ASWE-Bench ( Abdalla et al., 2026 )
Automotive control logic
Simulink simulation
Boundary and time-series tests
SimuBench ( Liang & Zhao, 2026 )
Multi-domain text-to-model
Basic execution
Reference similarity and completeness
VerilogEval ( Liu et al., 2023 )
RTL code generation
Transient simulation
Functional correctness
RTLLM ( Lu et al., 2024 )
RTL design generation
Compilation and simulation
Syntax, function, and design quality
Appendix
Table 6: Comparison with related executable-engineering and Simulink benchmarks.
XAERO-04: Closed-Loop Tiltrotor VTOL Flight
Engineering goal
Build a VTOL model coupling flight missions, closed-loop control, tilt actuation, and six-degree-of-freedom vehicle dynamics. Support hover waypoint flight, hover-to-fixed-wing transition, and fixed-wing waypoint or orbit flight.
Inputs and outputs
Inputs are mission type, initial flight mode, mission profile, payload, wind enablement, and simulation duration. Outputs are position-tracking, transition-state, vehicle-state, and actuator-command time series.
Core constraints
Rotor propulsion, independent front-rotor tilt, aerodynamic control surfaces, and vehicle dynamics must form connected paths, with vehicle states participating in feedback control. The model must cover the specified payload and wind conditions, maintain bounded tracking errors and continuous transitions, and avoid prescribed or replayed flight trajectories.
Validation scenarios
Six scenarios: nominal hover, payload boundary, transition with wind, forward transition, combined payload and wind, and a fixed-wing turn.
Applicable dimensions
A/Q/M/R/D.
Appendix
Table 11
XAERO-07: Electric-Aircraft Mission and Powertrain
Engineering goal
Build an aircraft powertrain coupling the flight mission, aerodynamic loads, electric propulsion, and power distribution. Evaluate how battery capacity and payload affect mission execution.
Inputs and outputs
Inputs are battery capacity and payload. Outputs are flight range, battery state, motor current, and low-voltage auxiliary-load power.
Core constraints
Battery capacity affects both stored energy and mass, while payload affects power demand. Electrical and mechanical paths must actively participate in execution.
Validation scenarios
Five scenarios: nominal design, increased capacity, increased payload, joint variation, and an intermediate design point.
Applicable dimensions
A/Q/M/C/R/D.
Appendix
Table 12
XAUTO-07: Electric Axle Drive with Regenerative Braking
Engineering goal
Build a battery-powered electric axle drive coupling closed-loop motor control, the drivetrain, and longitudinal vehicle dynamics for acceleration and regenerative braking.
Inputs and outputs
Inputs scale acceleration and braking demand. Outputs include actual pedal inputs; reference, measured, and estimated torque and torque limits; three-phase currents; and vehicle speed.
Core constraints
A high-voltage source, DC bus, controlled three-phase converter, interior permanent-magnet synchronous motor, reduction gear, and vehicle load form a connected electromechanical path. Phase currents and rotor states participate in feedback control. Vehicle speed follows finite-mass vehicle dynamics, without prescribed speed or replayed torque and current trajectories.
Validation scenarios
Five scenarios: nominal startup and regeneration transition, reduced acceleration demand, reduced regenerative braking, moderate-demand cyclic operation, and asymmetric pedal excitation.
Applicable dimensions
A/Q/M/C/R/D.
Appendix
Table 13
Primary scenario category
Count
Operating-envelope scenarios
205
Interaction and fault scenarios
189
Temporal-process scenarios
79
Mechanism and causal-check scenarios
122
Total
595
Appendix
Table 7: Formal scenarios across the 101 tasks, counted by primary category.
Performance dimension
A
Q
M
C
R
D
Applicable tasks
101
101
93
35
101
98
Appendix
Table 8: Task counts for the six applicable performance dimensions.
Primary category
Scenario
Engineering checks
Main dims.
Temporal process
Nominal startup and regeneration transition
Startup torque establishment, transition timing from motoring to regeneration, and torque tracking.
D
Mechanism and causal check
Reduced acceleration demand
Paired comparison with the nominal scenario of torque and speed responses to acceleration demand, together with electromechanical coupling.
M/C
Mechanism and causal check
Reduced regenerative braking
Paired comparison with the nominal scenario of generating torque, final vehicle speed, and braking-response delay.
C/D
Operating envelope
Moderate-demand cyclic operation
Torque tracking, ripple, and current balance at a moderate operating point within the input range.
A/Q
Interaction and fault
Asymmetric pedal excitation
Output quality and performance retention relative to nominal operation under asymmetric acceleration and braking demands.
R
Appendix
Table 9: Electric-axle scenarios, engineering checks, and main performance associations. All scenarios undergo output-completeness checks and contribute to dimension scores and cross-scenario evaluation.
Dim.
Engineering metrics
Score derivation for this candidate
Score
A
Torque-tracking and torque-estimation errors
Measured-to-reference tracking error and estimated-to-measured torque error are converted to utilities, then averaged across the five scenarios.
100.00
Q
Torque ripple, peak-to-average ratio, phase-current crest factor and balance, and ride smoothness
Torque and current fluctuations, three-phase current imbalance, and vehicle jerk are evaluated. Their utilities are averaged across the five scenarios.
100.00
M
Functional-path connectivity, motoring and generating operation, and electromechanical coupling
Connectivity, bidirectional operation, torque–acceleration relationships, and current–torque relationships are combined, with greater weight on weaker components. Electromechanical coupling is the main shortfall here.
78.25
C
Acceleration/braking input fidelity and corresponding vehicle responses
Checks assess whether acceleration and braking inputs are applied as required and whether torque and speed respond accordingly. Weaker input responses receive greater weight. Braking-input fidelity and braking response are weaker here.
71.88
R
Retention of accuracy, output quality, electromechanical coupling, and bidirectional operation
Performance retention relative to nominal operation is calculated for each scenario, then aggregated through a weighted combination of the minimum and mean. Stronger motoring with light braking gives the lowest retention, although the overall decline is small.
99.90
D
Startup torque-establishment and regenerative-braking delays
Three delays are checked: nominal startup, nominal braking, and entry into generating operation under reduced braking demand.
100.00
Appendix
Table 10: Six-dimensional performance and score derivation for the Opus Run 1 electric-axle candidate. Scores are rounded to two decimal places.
Statistic
Comparison
n
Estimate
95% CI
Spearman ρ
Expert 1 vs. Expert 2
30
0.87
[0.71, 0.96]
Spearman ρ
Evaluator vs. Expert 1
30
0.88
[0.71, 0.96]
Spearman ρ
Evaluator vs. Expert 2
30
0.94
[0.85, 0.98]
Spearman ρ
Evaluator vs. expert mean
30
0.94
[0.88, 0.98]
Spearman ρ
Evaluator vs. expert mean (nonzero candidates)
18
0.88
[0.72, 0.94]
Appendix
Table 11: Rank correlations between evaluator scores and domain-expert ratings.
Setting
Claude Opus 4.8
GPT-5.5
DeepSeek V4-Pro
Qwen3.8 Max
GLM-5.3 Flash
Qwen3.8 27B-FP8
Reasoning effort
max
xhigh
max
xhigh
max
xhigh
Context window (tokens)
1M
1,050,000
1M
1M
1,048,576
262,144
Appendix
Table 12: Reasoning effort and context windows. Each model uses its highest supported reasoning level and its official native context-window setting.
System
Run 1
Run 2
Run 3
Mean
95% CI
Consistent G
Claude Opus 4.8
44.34
41.05
43.18
42.86
[37.03, 48.73]
92.08%
GPT-5.5
39.40
42.87
42.91
41.72
[35.29, 48.31]
89.11%
DeepSeek-V4-Pro
27.35
29.52
28.32
28.40
[22.01, 34.76]
82.18%
Qwen3.8-Max
24.57
25.85
24.62
25.01
[18.85, 31.37]
86.14%
GLM-5.3-Flash
3.36
6.19
5.38
4.98
[2.40, 8.04]
81.19%
Qwen3.8-27B-FP8
1.58
1.56
1.67
1.60
[0.41, 2.99]
89.11%
Appendix
Table 13: Scores for the three runs, mean, task-level 95% confidence interval, and the proportion of tasks with consistent G outcomes across runs.
Comparison
Difference
95% CI
p
Claude Opus 4.8 – GPT-5.5
1.13
[-5.80, 7.68]
0.86
GPT-5.5 – DeepSeek-V4-Pro
13.33
[6.93, 19.89]
<0.001
DeepSeek-V4-Pro – Qwen3.8-Max
3.38
[-3.63, 10.39]
0.46
Qwen3.8-Max – GLM-5.3-Flash
20.04
[13.73, 26.54]
<0.001
GLM-5.3-Flash – Qwen3.8-27B-FP8
3.37
[0.92, 6.03]
0.012
Appendix
Table 14: Differences between adjacent agent systems. 95% confidence intervals are obtained by paired task-level bootstrap, and p -values are from the Wilcoxon signed-rank test.
System
Original
Arithmetic
Equal-weight geometric
Minimum
Claude Opus 4.8
42.86
50.34
44.00
25.38
GPT-5.5
41.72
50.65
42.49
26.48
DeepSeek-V4-Pro
28.40
37.70
29.40
18.87
Qwen3.8-Max
25.01
33.01
25.61
16.06
GLM-5.3-Flash
4.98
7.37
5.38
3.20
Qwen3.8-27B-FP8
1.60
3.29
1.67
0.74
Appendix
Table 15: Overall scores under the original and three alternative aggregation rules; candidates failing G receive zero.
Condition
Run 1
Run 2
Run 3
Mean
95% CI
Tasks below Full MCP
Full MCP
82.19
76.04
77.16
78.46
[76.13, 80.93]
—
No-Simulation MCP
6.63
10.85
6.17
7.88
[0.00, 20.06]
10/10
Batch-only
16.81
20.67
17.22
18.23
[0.00, 40.73]
8/10
Appendix
Table 16: Scores for the three runs, mean, task-level 95% confidence interval, and tasks with scores below Full MCP on the tool-ablation subset.
(a) End-to-end overall score
Domain
Claude Opus 4.8
GPT-5.5
DeepSeek V4-Pro
Qwen3.8 Max
GLM-5.3 Flash
Qwen3.8 27B-FP8
Power electronics & control
25.99
30.51
21.63
1.79
5.72
0.00
Mechanical
32.85
34.25
23.74
35.91
0.00
0.00
Robotics
55.49
49.79
47.75
36.83
0.00
0.00
Thermal
49.44
59.48
36.13
36.52
3.52
0.00
Fluid
48.92
46.35
39.79
24.81
19.48
5.87
Appendix
Table 17: End-to-end and post-G overall scores across ten domains for six systems. An em dash indicates no qualified candidate in any run; 0.00 denotes a defined zero score.
Engineering domain
N
Overall
G pass
Post-G
Capability regime
Power electronics & control
9
25.99
44.44%
58.49
Qualification dominated
Mechanical systems
11
32.85
69.70%
47.46
Compound
Robotics
10
55.49
86.67%
64.05
—
Thermal systems
11
49.44
78.79%
62.90
—
Fluid systems
11
48.92
72.73%
67.26
Qualification dominated
Automotive systems
11
53.03
90.91%
58.33
—
Appendix
Table 18: Cross-domain engineering profiles for Claude Opus 4.8 with Claude Code. “Post-G” averages include only candidates that pass engineering qualification; an em dash marks domains with no dominant bottleneck type.
Figure 5: Summary of the experimental results. (a) Overall performance, (b) structural relation, (c) tool access, and (d) prerequisite-gate outcomes.
Case
A
Q
M
C
R
D
Wheel-loader coordination ( XMECH-11 )
95.55
99.89
100.00
50.00
91.40
100.00
Electric-aircraft mission and powertrain ( XAERO-07 )
100.00
33.33
75.00
100.00
66.67
100.00
Serial robotic arm ( XMECH-10 )
0.00
0.00
0.00
—
0.00
0.00
Radar system ( XCOMM-01 )
0.00
55.18
25.00
—
0.00
63.69
Appendix
Table 19: Six-dimensional scores of the four representative candidates. All four candidates pass the G gate; an em dash indicates a non-applicable dimension.
Relation
Decision rule
Reference-Aligned (RA)
The candidate preserves a recognizable correspondence in major block choices and abstraction level, system topology, subsystem decomposition, and signal/feedback routing. Differences are limited mainly to local substitutions, parameterization, auxiliary blocks, or bounded reorganization.
Partially Similar (PS)
The candidate retains some central components or engineering paths, but materially changes the overall architecture, abstraction level, subsystem organization, or signal/feedback routing. Similarity is local rather than system-wide.
Structurally Distinct (SD)
The candidate differs substantially in system-level formulation, abstraction level, topology, subsystem organization, or signal/feedback routing. Shared elements are mainly task-mandated interfaces, generic domain components, or the engineering objective.
Reference: detailed physical loader, engine, transmission, and surface subsystems; candidate: a lumped MATLAB-function plant, a different system-level structural implementation.
Valid diversity
XAERO-07
Structurally Distinct
Pass
80.07
Reference: physical Simscape aircraft distribution network; candidate: signal-level lumped equations, with different component families, topology, and routing.
Deceptive similarity
XMECH-10
Reference-Aligned
Pass
0.00
A recognizable serial youBot arm topology extends from the base pivot to the two-finger gripper, but defects in its hierarchy, actuation, and routing cause severely inadequate performance; all performance dimensions are zero.
Deceptive similarity
XCOMM-01
Reference-Aligned
Pass
0.00
The organization and output paths resemble a single-core radar, but target-range and velocity estimation are defective. Objective accuracy and operating-domain robustness are zero.
Table 22: Layout-only optimization of eight high-performing but severely disordered Simulink models. Lower values are better for all three layout indicators.
Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.
Jianing Yin, Ziyi Chen, Hao Wang +4
University of Pennsylvania · University of Illinois at Urbana-Champaign · University of California, Berkeley +2
AI agents are increasingly used for simulation-driven engineering. Physical system modeling presents different requirements from general-purpose code generation in software engineering, because correctness depends not only on syntax and executability but also on physical consistency and scenario-dependent behavior. We study this challenge in Modelica, an equation-based modeling language in which a model may compile and simulate while still violating its intended physics or engineering requirements. Across successive revisions, an agent may lose track of requirements or rely on simulation evidence produced by an outdated candidate. To address this challenge, we present Pufibara, an agent harness that maintains persistent engineering state across revisions, associates execution and simulation evidence with the candidate that produced it, and makes submission an explicit agent action. To evaluate end-to-end Modelica agent workflows, we also propose a source-grounded method for constructing realistic and independently evaluable tasks. We use this method to build the 232-task Modelica Agent Workflow Benchmark, spanning Model Repair, Model Generation, and Model Tuning. Each submitted candidate is scored by a benchmark-owned evaluator outside the agent loop. We compare Pufibara with Claude Code as complete harnesses under two matched large language model (LLM) backends. With DeepSeek v4 Flash, Pufibara passes 202 tasks, compared with 185 for Claude Code. With Claude Sonnet 5, Pufibara passes 202 tasks, compared with 187 for Claude Code. Under the repository-reported token accounting, Pufibara records 76.4%-82.5% lower logical-token totals. Its sequential runtime is 6.1%-58.4% lower. These findings show that, even under matched LLM backends, complete agent harnesses can differ substantially in both task success and resource use for physical system modeling.
Large language models can generate scientific simulation code, but the generated code silently fails on most non-textbook problems. We show that classical mathematical validation -- well-posedness, convergence, and error certification -- can be fully automated by a Judge Agent, reducing the silent-failure rate from 42% to 1.5% across 134 test cases spanning 12 scientific domains. The headline result comes from a prospective benchmark: 72 blinded tasks submitted by 12 independent scientists yield an 89% success rate (95% CI: [80%, 95%]) with automated error bounds, versus 53% without the Judge. On clinical CT (the only powered experiment, n = 200), the pipeline reaches 99% of expert quality. The residual 1.5% concentrates at bifurcation points where certifiability breaks down. We formalize this boundary through the simulability class S and introduce spec.md, a structured specification format that makes any scientific computation problem machine-readable and solver-independent. Code, data, and all 72 benchmark tasks are publicly archived.