SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation

Ruiqi Zhang; Jiahao Wang; Mingxuan Li; Haichen Luo; Chaoting Wang; Guoyu Mou; Keyu Lai; Hanchao Lv

cs.SEOct 1, 2026

SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation

Authors: Ruiqi Zhang, Jiahao Wang, Mingxuan Li, Haichen Luo, Chaoting Wang, Guoyu Mou, Keyu Lai, Hanchao Lv,

Organizations: Xi’an Jiaotong University

Abstract

Existing Simulink benchmarks mainly evaluate whether generated models compile, execute, or resemble a reference model. These criteria do not establish whether a model satisfies its engineering requirements. We introduce SimuVerity, a benchmark of 101 text-to-executable Simulink model-generation tasks across ten engineering domains. For each task, executable-system profiles ground the engineering specification and four families of native simulation scenarios. A hierarchical evaluator first checks artifact delivery, native executability, and engineering qualification, then scores qualified models across six dimensions covering accuracy, output quality, mechanistic fidelity, control and causal integrity, operating-domain robustness, and dynamic response. We evaluate six agent systems with SimuVerity. The best system achieves an overall score of only 42.86. The results show that structural similarity is a poor proxy for engineering performance: capability bottlenecks arise both in producing qualified implementations and in satisfying multidimensional requirements after qualification. Meanwhile, some high-scoring models still exhibit severe visual-layout disorder. SimuVerity provides a systematic basis for assessing agents' engineering capabilities and diagnosing failures in executable Simulink model generation.

Figures & tables

Figure 1: Task composition and agent performance in SimuVerity . (a) Task distribution across ten engineering domains; outer-ring line lengths indicate mean domain scores. (b) Six-dimensional performance and overall scores of six agents.

Figure 2: The SimuVerity pipeline. Benchmark construction, native simulation testing, and six-dimensional performance evaluation.

Engineering domains (101 tasks)
Power electronics & control	9	Automotive systems	11
Mechanical systems	11	Battery systems	10
Robotics	10	Aerospace systems	10
Thermal systems	11	Communication systems	9
Fluid systems	11	Biomedical systems	9

Table 1: Task distribution across the ten engineering domains of SimuVerity .

Figure 3: Evaluation of the closed-loop tiltrotor VTOL model. Native simulation evidence, six-dimensional scores, and failure diagnosis for the candidate model. Four of six scenarios are shown.

Agent system	Overall	Delivered	Executable	G pass
Opus 4.8 + Claude Code	42.86	94.72%	86.47%	76.90%
GPT-5.5 + Codex	41.72	98.68%	94.72%	75.58%
DeepSeek-V4-Pro + Claude Code	28.40	97.03%	88.12%	61.39%
Qwen3.8-Max + Claude Code	25.01	89.77%	80.86%	53.80%
GLM-5.3-Flash + Claude Code	4.98	96.04%	48.18%	13.20%
Qwen3.8-27B-FP8 + Claude Code	1.60	30.36%	13.53%	6.27%

Table 2: Overall scores and prerequisite pass rates. Agent results are three-run means; Reference reports the mean score of 101 task-specific reference systems.

Agent system	Scope	A	Q	M	C	R	D
Opus 4.8 + Claude Code	End-to-end	47.95	63.24	52.34	48.82	46.72	45.90
	Post-G	62.28	82.21	65.25	58.92	60.73	58.65
GPT-5.5 + Codex	End-to-end	48.80	61.94	56.34	44.87	46.05	44.81
	Post-G	64.57	81.96	72.43	53.62	60.92	59.07
DeepSeek-V4-Pro + Claude Code	End-to-end	36.08	49.31	38.05	38.57	33.75	33.93
	Post-G	58.78	80.32	59.32	57.59	55.07	54.50

Table 3: End-to-end and post-G performance on the six engineering dimensions. Each dimension is averaged over its registered applicable-task set.

Structural relation	$n$	G pass	End-to-end	Post-G	Top quartile	Zero
Reference-Aligned	39	87.18%	52.13	59.80	12	7
Partially Similar	43	83.72%	44.05	52.61	9	10
Structurally Distinct	13	69.23%	42.37	61.20	3	4

Table 4: Structural relation and engineering performance. Results cover 95 delivered Opus Run 1 candidates; post-G means include only candidates that pass engineering qualification.

Figure 4: Robotic ball-balancing system before and after layout optimization. Only block positions and existing line routes are adjusted; the model implementation remains unchanged.

Tool condition	Artifact delivery	Native executability	G pass	Overall score
Full MCP	100.00%	100.00%	100.00%	78.46
No-Simulation MCP	53.33%	33.33%	13.33%	7.88
Batch-only	60.00%	50.00%	40.00%	18.23

Table 5: Ablations of native simulation feedback and tool access. Results report three-run mean prerequisite pass rates and overall scores for Claude Opus 4.8 with Claude Code on ten cross-domain, high-performing tasks.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Task scope	Native execution	Evaluation focus
SLGPT ( Shrestha & Csallner, 2021 )	Simulink file generation	Toolchain execution	Fault discovery
SimuGen ( Ren et al., 2025 )	Image-to-model reconstruction	Basic execution	Blocks and connections
ASWE-Bench ( Abdalla et al., 2026 )	Automotive control logic	Simulink simulation	Boundary and time-series tests
SimuBench ( Liang & Zhao, 2026 )	Multi-domain text-to-model	Basic execution	Reference similarity and completeness
VerilogEval ( Liu et al., 2023 )	RTL code generation	Transient simulation	Functional correctness
RTLLM ( Lu et al., 2024 )	RTL design generation	Compilation and simulation	Syntax, function, and design quality

Appendix

Table 6: Comparison with related executable-engineering and Simulink benchmarks.

XAERO-04: Closed-Loop Tiltrotor VTOL Flight
Engineering goal	Build a VTOL model coupling flight missions, closed-loop control, tilt actuation, and six-degree-of-freedom vehicle dynamics. Support hover waypoint flight, hover-to-fixed-wing transition, and fixed-wing waypoint or orbit flight.
Inputs and outputs	Inputs are mission type, initial flight mode, mission profile, payload, wind enablement, and simulation duration. Outputs are position-tracking, transition-state, vehicle-state, and actuator-command time series.
Core constraints	Rotor propulsion, independent front-rotor tilt, aerodynamic control surfaces, and vehicle dynamics must form connected paths, with vehicle states participating in feedback control. The model must cover the specified payload and wind conditions, maintain bounded tracking errors and continuous transitions, and avoid prescribed or replayed flight trajectories.
Validation scenarios	Six scenarios: nominal hover, payload boundary, transition with wind, forward transition, combined payload and wind, and a fixed-wing turn.
Applicable dimensions	A/Q/M/R/D.

Appendix

Table 11

XAERO-07: Electric-Aircraft Mission and Powertrain
Engineering goal	Build an aircraft powertrain coupling the flight mission, aerodynamic loads, electric propulsion, and power distribution. Evaluate how battery capacity and payload affect mission execution.
Inputs and outputs	Inputs are battery capacity and payload. Outputs are flight range, battery state, motor current, and low-voltage auxiliary-load power.
Core constraints	Battery capacity affects both stored energy and mass, while payload affects power demand. Electrical and mechanical paths must actively participate in execution.
Validation scenarios	Five scenarios: nominal design, increased capacity, increased payload, joint variation, and an intermediate design point.
Applicable dimensions	A/Q/M/C/R/D.

Appendix

Table 12

XAUTO-07: Electric Axle Drive with Regenerative Braking
Engineering goal	Build a battery-powered electric axle drive coupling closed-loop motor control, the drivetrain, and longitudinal vehicle dynamics for acceleration and regenerative braking.
Inputs and outputs	Inputs scale acceleration and braking demand. Outputs include actual pedal inputs; reference, measured, and estimated torque and torque limits; three-phase currents; and vehicle speed.
Core constraints	A high-voltage source, DC bus, controlled three-phase converter, interior permanent-magnet synchronous motor, reduction gear, and vehicle load form a connected electromechanical path. Phase currents and rotor states participate in feedback control. Vehicle speed follows finite-mass vehicle dynamics, without prescribed speed or replayed torque and current trajectories.
Validation scenarios	Five scenarios: nominal startup and regeneration transition, reduced acceleration demand, reduced regenerative braking, moderate-demand cyclic operation, and asymmetric pedal excitation.
Applicable dimensions	A/Q/M/C/R/D.

Appendix

Table 13

Primary scenario category	Count
Operating-envelope scenarios	205
Interaction and fault scenarios	189
Temporal-process scenarios	79
Mechanism and causal-check scenarios	122
Total	595

Appendix

Table 7: Formal scenarios across the 101 tasks, counted by primary category.

Performance dimension	A	Q	M	C	R	D
Applicable tasks	101	101	93	35	101	98

Appendix

Table 8: Task counts for the six applicable performance dimensions.

Primary category	Scenario	Engineering checks	Main dims.
Temporal process	Nominal startup and regeneration transition	Startup torque establishment, transition timing from motoring to regeneration, and torque tracking.	D
Mechanism and causal check	Reduced acceleration demand	Paired comparison with the nominal scenario of torque and speed responses to acceleration demand, together with electromechanical coupling.	M/C
Mechanism and causal check	Reduced regenerative braking	Paired comparison with the nominal scenario of generating torque, final vehicle speed, and braking-response delay.	C/D
Operating envelope	Moderate-demand cyclic operation	Torque tracking, ripple, and current balance at a moderate operating point within the input range.	A/Q
Interaction and fault	Asymmetric pedal excitation	Output quality and performance retention relative to nominal operation under asymmetric acceleration and braking demands.	R

Appendix

Table 9: Electric-axle scenarios, engineering checks, and main performance associations. All scenarios undergo output-completeness checks and contribute to dimension scores and cross-scenario evaluation.

Dim.	Engineering metrics	Score derivation for this candidate	Score
A	Torque-tracking and torque-estimation errors	Measured-to-reference tracking error and estimated-to-measured torque error are converted to utilities, then averaged across the five scenarios.	100.00
Q	Torque ripple, peak-to-average ratio, phase-current crest factor and balance, and ride smoothness	Torque and current fluctuations, three-phase current imbalance, and vehicle jerk are evaluated. Their utilities are averaged across the five scenarios.	100.00
M	Functional-path connectivity, motoring and generating operation, and electromechanical coupling	Connectivity, bidirectional operation, torque–acceleration relationships, and current–torque relationships are combined, with greater weight on weaker components. Electromechanical coupling is the main shortfall here.	78.25
C	Acceleration/braking input fidelity and corresponding vehicle responses	Checks assess whether acceleration and braking inputs are applied as required and whether torque and speed respond accordingly. Weaker input responses receive greater weight. Braking-input fidelity and braking response are weaker here.	71.88
R	Retention of accuracy, output quality, electromechanical coupling, and bidirectional operation	Performance retention relative to nominal operation is calculated for each scenario, then aggregated through a weighted combination of the minimum and mean. Stronger motoring with light braking gives the lowest retention, although the overall decline is small.	99.90
D	Startup torque-establishment and regenerative-braking delays	Three delays are checked: nominal startup, nominal braking, and entry into generating operation under reduced braking demand.	100.00

Appendix

Table 10: Six-dimensional performance and score derivation for the Opus Run 1 electric-axle candidate. Scores are rounded to two decimal places.

Statistic	Comparison	$n$	Estimate	95% CI
Spearman $\rho$	Expert 1 vs. Expert 2	30	0.87	[0.71, 0.96]
Spearman $\rho$	Evaluator vs. Expert 1	30	0.88	[0.71, 0.96]
Spearman $\rho$	Evaluator vs. Expert 2	30	0.94	[0.85, 0.98]
Spearman $\rho$	Evaluator vs. expert mean	30	0.94	[0.88, 0.98]
Spearman $\rho$	Evaluator vs. expert mean (nonzero candidates)	18	0.88	[0.72, 0.94]

Appendix

Table 11: Rank correlations between evaluator scores and domain-expert ratings.

Setting	Claude Opus 4.8	GPT-5.5	DeepSeek V4-Pro	Qwen3.8 Max	GLM-5.3 Flash	Qwen3.8 27B-FP8
Reasoning effort	max	xhigh	max	xhigh	max	xhigh
Context window (tokens)	1M	1,050,000	1M	1M	1,048,576	262,144

Appendix

Table 12: Reasoning effort and context windows. Each model uses its highest supported reasoning level and its official native context-window setting.

System	Run 1	Run 2	Run 3	Mean	95% CI	Consistent G
Claude Opus 4.8	44.34	41.05	43.18	42.86	[37.03, 48.73]	92.08%
GPT-5.5	39.40	42.87	42.91	41.72	[35.29, 48.31]	89.11%
DeepSeek-V4-Pro	27.35	29.52	28.32	28.40	[22.01, 34.76]	82.18%
Qwen3.8-Max	24.57	25.85	24.62	25.01	[18.85, 31.37]	86.14%
GLM-5.3-Flash	3.36	6.19	5.38	4.98	[2.40, 8.04]	81.19%
Qwen3.8-27B-FP8	1.58	1.56	1.67	1.60	[0.41, 2.99]	89.11%

Appendix

Table 13: Scores for the three runs, mean, task-level 95% confidence interval, and the proportion of tasks with consistent G outcomes across runs.

Comparison	Difference	95% CI	$p$
Claude Opus 4.8 – GPT-5.5	1.13	[-5.80, 7.68]	0.86
GPT-5.5 – DeepSeek-V4-Pro	13.33	[6.93, 19.89]	$<0.001$
DeepSeek-V4-Pro – Qwen3.8-Max	3.38	[-3.63, 10.39]	0.46
Qwen3.8-Max – GLM-5.3-Flash	20.04	[13.73, 26.54]	$<0.001$
GLM-5.3-Flash – Qwen3.8-27B-FP8	3.37	[0.92, 6.03]	0.012

Appendix

Table 14: Differences between adjacent agent systems. 95% confidence intervals are obtained by paired task-level bootstrap, and

p

-values are from the Wilcoxon signed-rank test.

System	Original	Arithmetic	Equal-weight geometric	Minimum
Claude Opus 4.8	42.86	50.34	44.00	25.38
GPT-5.5	41.72	50.65	42.49	26.48
DeepSeek-V4-Pro	28.40	37.70	29.40	18.87
Qwen3.8-Max	25.01	33.01	25.61	16.06
GLM-5.3-Flash	4.98	7.37	5.38	3.20
Qwen3.8-27B-FP8	1.60	3.29	1.67	0.74

Appendix

Table 15: Overall scores under the original and three alternative aggregation rules; candidates failing G receive zero.

Condition	Run 1	Run 2	Run 3	Mean	95% CI	Tasks below Full MCP
Full MCP	82.19	76.04	77.16	78.46	[76.13, 80.93]	—
No-Simulation MCP	6.63	10.85	6.17	7.88	[0.00, 20.06]	10/10
Batch-only	16.81	20.67	17.22	18.23	[0.00, 40.73]	8/10

Appendix

Table 16: Scores for the three runs, mean, task-level 95% confidence interval, and tasks with scores below Full MCP on the tool-ablation subset.

(a) End-to-end overall score
Domain	Claude Opus 4.8	GPT-5.5	DeepSeek V4-Pro	Qwen3.8 Max	GLM-5.3 Flash	Qwen3.8 27B-FP8
Power electronics & control	25.99	30.51	21.63	1.79	5.72	0.00
Mechanical	32.85	34.25	23.74	35.91	0.00	0.00
Robotics	55.49	49.79	47.75	36.83	0.00	0.00
Thermal	49.44	59.48	36.13	36.52	3.52	0.00
Fluid	48.92	46.35	39.79	24.81	19.48	5.87

Appendix

Table 17: End-to-end and post-G overall scores across ten domains for six systems. An em dash indicates no qualified candidate in any run; 0.00 denotes a defined zero score.

Engineering domain	$N$	Overall	G pass	Post-G	Capability regime
Power electronics & control	9	25.99	44.44%	58.49	Qualification dominated
Mechanical systems	11	32.85	69.70%	47.46	Compound
Robotics	10	55.49	86.67%	64.05	—
Thermal systems	11	49.44	78.79%	62.90	—
Fluid systems	11	48.92	72.73%	67.26	Qualification dominated
Automotive systems	11	53.03	90.91%	58.33	—

Appendix

Table 18: Cross-domain engineering profiles for Claude Opus 4.8 with Claude Code. “Post-G” averages include only candidates that pass engineering qualification; an em dash marks domains with no dominant bottleneck type.

Figure 5: Summary of the experimental results. (a) Overall performance, (b) structural relation, (c) tool access, and (d) prerequisite-gate outcomes.

Case	A	Q	M	C	R	D
Wheel-loader coordination ( XMECH-11 )	95.55	99.89	100.00	50.00	91.40	100.00
Electric-aircraft mission and powertrain ( XAERO-07 )	100.00	33.33	75.00	100.00	66.67	100.00
Serial robotic arm ( XMECH-10 )	0.00	0.00	0.00	—	0.00	0.00
Radar system ( XCOMM-01 )	0.00	55.18	25.00	—	0.00	63.69

Appendix

Table 19: Six-dimensional scores of the four representative candidates. All four candidates pass the G gate; an em dash indicates a non-applicable dimension.

Relation	Decision rule
Reference-Aligned (RA)	The candidate preserves a recognizable correspondence in major block choices and abstraction level, system topology, subsystem decomposition, and signal/feedback routing. Differences are limited mainly to local substitutions, parameterization, auxiliary blocks, or bounded reorganization.
Partially Similar (PS)	The candidate retains some central components or engineering paths, but materially changes the overall architecture, abstraction level, subsystem organization, or signal/feedback routing. Similarity is local rather than system-wide.
Structurally Distinct (SD)	The candidate differs substantially in system-level formulation, abstraction level, topology, subsystem organization, or signal/feedback routing. Shared elements are mainly task-mandated interfaces, generic domain components, or the engineering objective.

Appendix

Table 20: Three-level structural relation criteria.

Phenomenon	Task	Relation	G	Score	Engineering evidence
Valid diversity	XMECH-11	Structurally Distinct	Pass	85.10	Reference: detailed physical loader, engine, transmission, and surface subsystems; candidate: a lumped MATLAB-function plant, a different system-level structural implementation.
Valid diversity	XAERO-07	Structurally Distinct	Pass	80.07	Reference: physical Simscape aircraft distribution network; candidate: signal-level lumped equations, with different component families, topology, and routing.
Deceptive similarity	XMECH-10	Reference-Aligned	Pass	0.00	A recognizable serial youBot arm topology extends from the base pivot to the two-finger gripper, but defects in its hierarchy, actuation, and routing cause severely inadequate performance; all performance dimensions are zero.
Deceptive similarity	XCOMM-01	Reference-Aligned	Pass	0.00	The organization and output paths resemble a single-core radar, but target-range and velocity estimation are defective. Objective accuracy and operating-domain robustness are zero.

Appendix

Table 21: Representative cases illustrating valid structural diversity and deceptive structural similarity.

Candidate model	Domain	Score	Overlap	Detour	Crossings / 100 segments
QPSK Communication Link	Communication	83.33	0.950 $\rightarrow$ 0.000	1.283 $\rightarrow$ 1.009	71.0 $\rightarrow$ 19.6
Two-Body Point-Absorber Wave-Energy Converter	Fluid	84.22	0.966 $\rightarrow$ 0.000	5.998 $\rightarrow$ 1.060	295.6 $\rightarrow$ 70.8
Closed-Loop CPU Cooling System	Thermal	81.02	0.907 $\rightarrow$ 0.000	1.493 $\rightarrow$ 1.078	204.8 $\rightarrow$ 30.6
Grid-Connected Two-Level Inverter	Power electronics & control	74.81	0.240 $\rightarrow$ 0.000	1.085 $\rightarrow$ 1.080	55.9 $\rightarrow$ 43.2
Dual-Patient Ventilation System	Biomedical	73.46	0.952 $\rightarrow$ 0.000	1.388 $\rightarrow$ 1.013	120.1 $\rightarrow$ 36.7
Power-Split Hybrid-Electric Vehicle	Automotive	72.48	0.918 $\rightarrow$ 0.000	1.040 $\rightarrow$ 1.011	309.9 $\rightarrow$ 70.1

Appendix

Table 22: Layout-only optimization of eight high-performing but severely disordered Simulink models. Lower values are better for all three layout indicators.

Figure 7: Two-Body Point-Absorber Wave-Energy Converter

Figure 8: Closed-Loop CPU Cooling System

Figure 9: Dual-Patient Ventilation System

Figure 10: Grid-Connected Two-Level Inverter

Figure 11: Power-Split Hybrid-Electric Vehicle

Task	Domain	Full MCP	No-Simulation MCP	Batch-only
XELEC-07	Power electronics & control	74.55	0.00	0.00
XAERO-07	Aerospace systems	75.35	0.00	0.00
XAUTO-07	Automotive systems	85.13	0.00	21.49
XBATT-05	Battery systems	73.33	0.00	75.01
XCOMM-06	Communication systems	79.91	60.87	0.00
XFLUID-08	Fluid systems	79.78	0.00	85.81

Appendix

Table 23: Task-level engineering scores under three tool conditions across ten cross-domain tasks. Each entry is the arithmetic mean over three runs.

Tool condition	A	Q	M	C	R	D
Full MCP	78.09	86.82	81.35	97.08	84.50	75.58
No-Simulation MCP	6.56	9.71	14.07	0.00	8.38	5.86
Batch-only	21.70	32.89	38.53	18.76	26.08	21.12

Appendix

Table 24: End-to-end six-dimensional tool-ablation results for Claude Opus 4.8 with Claude Code on ten cross-domain tasks.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,497 papers

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
Oct 19, 2026Jianing Yin, Ziyi Chen, Hao Wang +4LLM Agent Verification AI Agent Benchmarks
University of Pennsylvania · University of Illinois at Urbana-Champaign · University of California, Berkeley +2
Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modeling
Aug 24, 2026Zizhe WangLLM Agent Harnesses AI Agent Benchmarks
A Judge Agent Closes the Reliability Gap in AI-Generated Scientific Simulation
Mar 26, 2026Chengshuai YangScientific Code Generation LLM Reliability
NextGen PlatformAI C Corp, USA

Oct 19, 2026J/K move · Enter open · S save

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Jianing Yin, Ziyi Chen, Hao Wang, Yiyou Sun+3

University of Pennsylvania · University of Illinois at Urbana-Champaign · University of California, Berkeley+2

LLM Agent Verification AI Agent Benchmarks Automated Agent Evaluation

Open paper AskPDF arXiv

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.

Abstract

AI Agent Benchmarks

Explore similar work

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modeling

A Judge Agent Closes the Reliability Gap in AI-Generated Scientific Simulation