cs.AINov 2, 2026
SaveParametric CAD Bench v3: Evaluating Agentic Parametric CAD Authoring from Text and Engineering Drawings
Organizations: gNucleus AI, Inc.
Abstract
Producing an executable CAD script is not sufficient to establish that an agent has satisfied an engineering design request. Evaluation must also consider the resulting geometry, specified dimensions, and native CAD modeling history. We present Parametric CAD Bench v3, a 100-task benchmark comprising 30 text-to-CAD creation tasks, 30 text-to-CAD create-and-edit tasks, and 40 tasks that reconstruct mechanical parts from engineering drawings. Agents operate in a pinned FreeCAD environment and submit native CAD artifacts to a separate programmatic verifier. The primary metric is mean continuous task reward, combining geometry similarity and specification consistency rather than binary task success. A September 24, 2026 snapshot covers 13 model-agent systems; Claude Code with Claude Opus 5.5 achieves the highest overall point estimate, 61.03%, and an image-to-CAD mean reward of 84.66%. These measurements describe complete systems, not isolated model capabilities. We release versioned tasks, evaluation configurations, and inspectable result artifacts.
Figures & tables
| Workflow | Tasks | Agent input | Required CAD artifacts |
|---|---|---|---|
| Text-to-CAD Creation | 30 | Part description and specified parameters | Native parametric model |
| Text-to-CAD Create-and-Edit | 30 | Base description, edit instruction, and target requirements | Base and edited native models |
| Image-to-CAD | 40 | Engineering drawing and general execution requirements | Native model reconstructed from the drawing |
Table 1: The v3 task composition. Text-to-CAD create-and-edit is a two-stage construction workflow, not editing an externally supplied CAD document.
Figure 1: Grader reference of text-to-CAD creation example freecad-2002df3ce7 , a trussed upper plate. (b) faces the plate, showing its 18 internal openings, side mounting holes, and centered end extension.
Figure 2: Grader reference of text-to-CAD creation example freecad-fa0a1abef3 , an asymmetric clevis joint connector. (b) looks along the split-ring axis, showing the fork, the large clevis, and the slotted side arm.
Figure 3: Text-to-CAD create-and-edit example freecad-ff7883ab0c : trusted base reference (top) and edited target reference (bottom), shown in oblique and axial views. Each view uses the same angle and scale for both stages. The edit adds the front flange plate, its four bolt holes, and four rear-face openings while retaining the axisymmetric body and central bore.
Figure 4: Image-to-CAD example freecad-9749149002 , a front-wheel linkage arm. Top: the engineering drawing supplied to the agent, cropped to its frame; the scale line and general-tolerance note outside the frame are not shown. Bottom: an oblique view of the held-back grader reference.
| Workflow | Tasks | Mean features | Median | Min | Max |
|---|---|---|---|---|---|
| All final references | 100 | 6.87 | 6 | 1 | 20 |
| Text-to-CAD Creation | 30 | 5.20 | 4 | 1 | 20 |
| Text-to-CAD Create-and-Edit (target) | 30 | 5.60 | 5.5 | 2 | 10 |
| Image-to-CAD | 40 | 9.08 | 8 | 2 | 20 |
Table 3: Measured solid-modeling feature counts in the v3 grader references. Text-to-CAD create-and-edit contributes its target only to the 100-task distribution.
Figure 5: Solid-modeling features per final reference model. The aggregate contains 100 tasks, not 130 documents; dashed lines mark cohort means. Counts reflect reference construction choices rather than a calibrated difficulty measure or a required solution sequence.
| Workflow | A mean | B mean | A higher | Similar | B higher | |
|---|---|---|---|---|---|---|
| Text-to-CAD Creation | 30 | 50.87% | 49.48% | 8 | 18 | 4 |
| Text-to-CAD Create-and-Edit | 30 | 39.69% | 36.52% | 4 | 23 | 3 |
| Image-to-CAD | 40 | 84.66% | 62.64% | 22 | 15 | 3 |
| Overall | 100 | 61.03% | 50.86% | 34 | 56 | 10 |
Table 4: Claude Code held constant; A: Opus 5.5, B: Opus 5. Higher means a task reward advantage greater than 5 percentage points; Similar includes the boundary. These counts do not establish statistical equivalence.
| Workflow | A mean | B mean | A higher | Similar | B higher | |
|---|---|---|---|---|---|---|
| Text-to-CAD Creation | 30 | 50.87% | 42.79% | 9 | 20 | 1 |
| Text-to-CAD Create-and-Edit | 30 | 39.69% | 41.38% | 2 | 24 | 4 |
| Image-to-CAD | 40 | 84.66% | 81.77% | 5 | 30 | 5 |
| Overall | 100 | 61.03% | 57.96% | 16 | 74 | 10 |
Table 5: Opus 5.5 held constant; A: Claude Code, B: mini-swe-agent with multimodal bridge. Higher means a task reward advantage greater than 5 percentage points; Similar includes the boundary. These counts do not establish statistical equivalence.
| Workflow | Zero outcomes | All 13 zero | 7 zero | Any perfect |
|---|---|---|---|---|
| Text-to-CAD Creation | 22.56% | 0 | 5 | 6 |
| Text-to-CAD Create-and-Edit | 41.54% | 5 | 11 | 3 |
| Image-to-CAD | 27.50% | 0 | 8 | 6 |
| Overall | 30.23% | 5 | 24 | 15 |
Table 6: Observed difficulty across 13 evaluated systems. Zero outcomes is the fraction of final trials with zero reward. The last three columns count distinct tasks: zero in all 13 systems, zero in at least 7 systems, and reward exactly 1.0 in at least one system. Cohorts contain 30 text-to-CAD creation, 30 text-to-CAD create-and-edit, and 40 image-to-CAD tasks.
| Model | Agent | Reward | Failed | Outcome | ||
|---|---|---|---|---|---|---|
| Claude Fable 5.1 | Claude Code | 0.999 | 0.998 | 1.000 | 0 | positive |
| Grok 4.6 | Grok Build | 0.628 | 0.482 | 0.900 | 1 | positive |
| GPT-6 Astra | Codex | 0.375 | 0.500 | 0.300 | 7 | positive |
| GPT-5.6 Terra | Codex | 0.140 | 0.232 | 0.100 | 9 | positive |
| Claude Opus 5 | Claude Code | 0.000 | 0.146 | 0.000 | 13 | specification zero |
| Claude Opus 5.5 | Claude Code | 0.000 | 0.445 | 0.000 | 12 | specification zero |
Table 7: Final outcomes of the 13 evaluated systems on the trussed-plate example freecad-2002df3ce7 . and are the geometry and specification scores; Failed counts the parameters, of 86, that are inconsistent or not found. —: no scored output after a final exception.
| Model | Agent | Reward | Failed | Outcome | ||
|---|---|---|---|---|---|---|
| Claude Opus 5 | Claude Code | 1.000 | 1.000 | 1.000 | 0 | positive |
| Claude Opus 5.5 | Claude Code | 0.615 | 0.500 | 0.800 | 2 | positive |
| Claude Fable 5.1 | Claude Code | 0.419 | 0.265 | 1.000 | 0 | positive |
| Gemini 3.8 Flash | mini-swe-agent | 0.419 | 0.265 | 1.000 | 0 | positive |
| GPT-6 Astra | Codex | 0.397 | 0.264 | 0.800 | 2 | positive |
| Kimi K3 | mini-swe-agent | 0.397 | 0.264 | 0.800 | 2 | positive |
Table 8: Final outcomes of the 13 evaluated systems on the clevis-connector example freecad-fa0a1abef3 . Columns as in Table 7 ; Failed counts the parameters, of 44, that are inconsistent or not found.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value | Unit | Check |
|---|---|---|---|
| Plate outline, thickness, and side recesses | |||
| plate_thickness | 8.00 | mm | W |
| base_sketch_plane_rotation_angle | 90.0 | deg | N |
| large_outer_rectangle_width | 280.00 | mm | N |
| large_outer_rectangle_height | 447.00 | mm | N |
| large_outer_rectangle_center_y | 51.00 | mm | N |
Table 9: Key parameters of freecad-2002df3ce7 , as given to the agent. Check: W, a witness on the final geometry; C, a custom binding; N, a native sketch or feature measurement by the task’s param_check.py .
| Parameter | Value | Unit | Check |
|---|---|---|---|
| Central body | |||
| connector_body_length | 75.0 | mm | N |
| connector_body_width | 45.0 | mm | N |
| connector_body_thickness | 12.0 | mm | W |
| Large split-ring clevis | |||
| clevis_end_outer_diameter | 45.0 | mm | W |
Table 10: Key parameters of freecad-fa0a1abef3 , as given to the agent. Check: W, a witness on the final geometry; C, a custom binding; N, a native sketch or feature measurement by the task’s param_check.py .
| Parameter | Value | Unit |
|---|---|---|
| Base part (unchanged by the edit) | ||
| central_bore_diameter | 50.8 | mm |
| tubular_body_outer_diameter | 68.58 | mm |
| tubular_body_taper_end_diameter | 75.2 | mm |
| tapered_inlet_rim_small_diameter | 88.84 | mm |
| tapered_inlet_rim_large_diameter | 95.25 | mm |
Table 11: Text-to-CAD create-and-edit example parameters, reproduced from the agent instruction. The 15 base parameters remain unchanged; the edit adds the 14 target requirements below.
Figure 6: Feature-object types across the 100 final references, broken down by workflow. A task can contribute several objects of the same type.
Figure 7: Feature counts for the 30 text-to-CAD create-and-edit pairs. Left: side-by-side bars distinguish base and edited target distributions. Right: paired base/target counts; circle area and numerals show how many tasks share each pair. The dashed diagonal indicates no count change. The target-minus-base count is not necessarily the number of operations needed to perform an edit.
Figure 8: Saved sketch-object counts per final reference model. Sketch objects are excluded from the solid-modeling feature statistic.
Figure 9: Overall mean continuous reward across 100 tasks in the September 24, 2026 snapshot. Whiskers show 95% task-level intervals. Every final exception counts as zero. Ranking is by point estimate, not demonstrated statistical separation.
Figure 10: Text-to-CAD creation mean reward across 30 tasks. Systems are ordered by their creation point estimates; workflow-specific intervals are not plotted.
Figure 11: Text-to-CAD create-and-edit mean reward across 30 tasks. Each task scores both base and target artifacts through the nested harmonic aggregation in Equation 5 . Bars show point estimates, not binary success rates.
Figure 12: Image-to-CAD mean reward across 40 engineering-drawing tasks. Bars show workflow point estimates only. Different workflow populations prevent treating cross-workflow differences as a controlled modality ablation.
More in this topic
Parametric CAD Modeling
Parametric CAD modeling represents 3D designs as editable, constraint-driven construction histories and develops methods to generate, reconstruct, and revise them.
See all 85 papers