cs.AISep 29, 2026
SaveEngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?
Organizations: Tsinghua University · Zhiman Inc. · Chongqing University · UCAS · Shandong University · BUPT · Fudan University · Henan Polytechnic University · Xi’an Jiaotong University · Peking University · Zhejiang University
Abstract
Autonomous agents have made rapid progress in general-purpose computer use, but reliable automation of professional industrial engineering remains out of reach, as engineering workflows demand reasoning over geometric and physical constraints and dependencies preserved across software and design stages. We present EngiWorld, the first benchmark structured around the complete design loop: 1,301 expert-curated tasks spanning 6 engineering domains (CAD, CAE, CAM, BIM, EDA, and 3D visualization) and 26 professional software platforms, with both GUI and CLI interfaces and 6 task types ranging from software-selection to open-ended tasks. We further introduce an artifact-centric evaluation methodology built on a unified domain-verifier suite, which programmatically checks the geometric validity, physical feasibility, and rule compliance of final and intermediate artifacts, and scores quantitative design tasks continuously by specification attainment rather than binary success. Evaluation of seven frontier models reveals a substantial capability gap: the strongest model achieves an EngiScore of only 44.3, and just 3.6% of multi-software attempts succeed. EngiWorld provides the first rigorous foundation for measuring progress toward agents that operate professional engineering software end to end.
Figures & tables
Figure 1: Overview of EngiWorld . The benchmark covers 26 software platforms and workbenches across six engineering domains, and includes six task types that together constitute an evaluation of the complete design loop.
| Benchmark | Size | Platforms | Domain | Interface | Cross | Loop | Choice | Objective |
| Text2CAD [ 17 ] | 170k models | – | CAD | Text2Artifact | ✗ | ✗ | ✗ | ✗ |
| VerilogEval [ 22 ] | 156 tasks | – | EDA | Code Generation | ✗ | ✗ | ✗ | ✗ |
| RTLLM [ 25 ] | 30 designs | – | EDA | Code Generation | ✗ | ✗ | ✗ | ✓ |
| OSWorld [ 39 ] | 369 tasks | 9 | General Desktop | GUI+CLI | ✓ | ✓ | ✗ | ✗ |
| Spider2-V [ 3 ] | 494 tasks | 20 | Data Engineering | GUI+CLI | ✓ | ✓ | ✗ | ✗ |
| ScienceBoard [ 35 ] | 169 tasks | 6 | Scientific | GUI+CLI | ✓ | ✓ | ✗ | ✗ |
Table 1: Comparison with existing benchmarks. Cross : cross-software workflows; Loop : execution feedback; Choice : software selection; Objective : engineering performance targets. ALE * is a subset of ALE comprising 25 engineering and 3D tasks.
Figure 2: Overview of the EngiWorld framework. Agents interact with engineering applications through CLI or GUI interfaces and are evaluated from the artifacts produced during execution. Domain-specific verifiers check whether these artifacts satisfy the required engineering constraints.
Figure 3: Dataset composition. EngiWorld contains 1,301 tasks across six engineering domains and 26 software platforms and workbenches. Software coverage counts task–software associations, including all applications in a workflow and all permitted tools in software-selection tasks.
Table 2: Main results on EngiWorld . EngiScore divided by task category and interface. Steps, Tokens, and Cost are per-task means over all 300 tasks; Tokens counts output tokens generated per decision step. Higher EngiScore and lower Steps, Tokens, and Cost are preferred; best and second-best values are bold and underlined .
Figure 6: Ablation on observation and history. Accessibility and resolution use GUI tasks; history includes both interfaces. A11y adds an accessibility tree to the screenshot. Resolution scales are relative to .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Dataset construction pipeline. Representative engineering scenarios are transformed into executable benchmark tasks through task construction, task-instance construction, and evaluation-criteria construction, with expert review and validation throughout the process.
| Domain / type | Application(s) | Version | OS | Interface |
| Software-specific environments (26) | ||||
| CAD | FreeCAD | 0.21.2 | Ubuntu | CLI/GUI |
| AutoCAD | 2024 | Windows | CLI/GUI | |
| SolidWorks | 2025 | Windows | CLI/GUI | |
| BRL-CAD | 7.32.2 | Ubuntu | CLI | |
| OpenSCAD | 2021.01 | Ubuntu | CLI/GUI | |
Table B.1: Software environments. The 37 environment configurations in EngiWorld: 26 software-specific environments, 10 composite environments, and one Open-ended environment.
| Check family | Properties | Acceptance criteria |
| Global geometry | Bounding spans, volume, surface area, and solid validity | Dimensional bounds and topology checks |
| Local geometry | Profiles at specified sections in the task’s datum frame | Profile tolerances, e.g., – mm |
| Solid–void occupancy | Material, cavities, through-holes, and bore clearance | Required occupancy; residual material, e.g., mm 3 |
| Engineering assignments | Materials, loads, boundary conditions, and thermal zones | Required assignments and parameter values |
| Simulation outputs | Field quantities and physical responses | Task-specific absolute or relative error bounds |
| Structural relations | Net connectivity, entity roles, and cross-file references | Exact connectivity and semantic consistency |
Table B.2: Artifact checks. Representative properties and acceptance rules across the verifier families. Numerical tolerances are task specific.
| Test population | Scope | Accepted | Rejected | |
| Reference artifacts | Completed audit group | 96 | 96 | 0 |
| Alternative solver | Abaqus / ANSYS, CLI | 40 | 40 | 0 |
| Alternative solver | Abaqus / ANSYS, GUI | 40 | 40 | 0 |
| Alternative design | FreeCAD lightweight design | 5 | 5 | 0 |
| Empty delivery | Software-selection tasks, CLI / GUI | 21 | 0 | 21 |
| Metrics without artifacts | Software-selection tasks, CLI / GUI | 21 | 0 | 21 |
Table B.3: Verifier validation. Acceptance outcomes for reference, alternative-valid, and defective artifact populations.
Figure C.1: Tasks missed by all models. Of 300 main-evaluation tasks, 128 receive zero from all seven models. The remaining bars show how many models obtain a nonzero score, while the composition bars partition the 128 all-zero tasks by type and interface.
| Number of runs | ||||
| Model | Zero | Partial | Full | EngiScore |
| GPT-5.6 Sol | 20 | 8 | 5 | 36.4 |
| Claude Opus 5 | 21 | 8 | 4 | 32.8 |
| Gemini 3.7 Flash | 24 | 9 | 0 | 20.6 |
| Kimi K3 | 24 | 9 | 0 | 21.9 |
| Qwen 3.8 Max | 24 | 9 | 0 | 22.3 |
Table C.1: Quantitative design outcomes. Each model is evaluated on 33 tasks. Partial and full denote and , respectively; EngiScore includes zero-scored runs.
| KiCad task 01: Layout optimization | ||||||
| Model | Cost (USD) | Route (mm) | Spacing (mm) | Vias | Baseline | Score |
| Gemini 3.7 Flash | 3.2135 | 40.1688 | 4.6174 | 0 | 0.647 | 0.982 |
| ZBrush task 04: Thumb-rest surface | ||||||
| Model | RMSE | Coverage (%) | Faces | Valid | Baseline | Score |
| Gemini 3.7 Flash | 0.15098 | 100.00 | 256 | Pass | 0.880 | 0.804 |
| GPT-5.6 Sol | 0.07650 | 100.00 | 256 | Pass | 0.880 | 0.892 |
Table C.2: Representative quantitative outcomes. Task-specific quality metrics are reported together with artifact validity, baseline quality, and EngiScore.
Figure C.2: Paired Multi-software and Open-ended tasks. Tasks 02 and 09 share engineering targets but differ in workflow and delivery requirements.
| Task 02 | Task 09 | |||
| Model | Multi- software | Open- ended | Multi- software | Open- ended |
| GPT-5.6 Sol | 0 | 0 | 0 | 0 |
| Claude Opus 5 | 0 | 0 | 0 | 1 |
| Gemini 3.7 Flash | 0 | 0 | 0 | 1 |
| Kimi K3 | 0 | 0 | 0 | 0 |
| Qwen 3.8 Max | 0 | 0 | 0 | 1 |
Table C.3: Paired task outcomes. Entries are binary, with 1 indicating success. Multi-software follows the prescribed workflow; Open-ended permits free tool and strategy selection.
Figure D.1: Successful single-software GUI workflows. The four rows show representative KiCad, AutoCAD, SolidWorks, and Blender executions. Intermediate edits and final checks or artifacts are connected by dashed arrows. All four runs pass their artifact verifiers.
| Recorded outcome | Runs | Share (%) |
| DONE with zero score | 1,641 | 49.03 |
| Decision-turn limit reached | 1,499 | 44.79 |
| Explicit FAIL declaration | 167 | 4.99 |
| Environment time limit exceeded | 37 | 1.11 |
| Desktop closed or restarted by agent action | 3 | 0.09 |
Table D.1: Failure taxonomy across all experiments. Counts and shares of 3,347 distinct failed runs in the five analyzed categories. Reused runs count once; required-field and missing-file failures are excluded.
| Model | Analyzed | DONE | Turn limit | FAIL | Timeout | Desktop |
| GPT-5.6 Sol | 184 | 160 | 24 | 0 | 0 | 0 |
| Claude Opus 5 | 165 | 122 | 32 | 2 | 9 | 0 |
| Gemini 3.7 Flash | 222 | 88 | 133 | 0 | 0 | 1 |
| Kimi K3 | 226 | 122 | 94 | 10 | 0 | 0 |
| Qwen 3.8 Max | 221 | 84 | 131 | 4 | 0 | 2 |
| Qwen 3.8 Flash | 251 | 66 | 154 | 31 | 0 | 0 |
Table D.2: Failure counts by model. Main-evaluation failures on the shared 300-task set, grouped into the five analyzed categories. Required-field and missing-file failures are excluded.
Figure D.2: Failure composition by interaction interface. Decision-turn exhaustion accounts for 36.6% of CLI failures and 55.3% of GUI failures; the remaining segments show the other four analyzed outcomes.
Figure D.3: Multi-software failure cases. Recorded actions, submitted data, and observed outcomes are compared with independently rendered reference artifacts. The first three cases use the GUI interface and the fourth uses the CLI interface; all receive zero credit.
| Task | Model | Verified violation |
| OpenSCAD microfluidic lid (GUI) | Gemini 3.7 Flash | Channel cut from the wrong face; sampled material and cavity states are inverted. |
| OpenSCAD V-groove pulley (CLI) | Gemini 3.7 Flash | Groove root and section fail at the required stations in the inherited datum frame. |
| KiCad–OpenSCAD–FreeCAD–Blender (CLI) | Gemini 3.7 Flash | The mechanical map assigns a mounting-hole role inconsistent with the source board. |
| Thermal-camera mount (Open-ended, CLI) | GPT-5.6 Sol | The MH1 bore contains mm 3 of residual material, exceeding the mm 3 allowance. |
Table D.3: Verified violations after declared completion. Each run ends with DONE but receives zero credit because the submitted artifact violates a task requirement.