cs.CLOct 28, 2026
Save3DCodeEdit: Benchmarking Instruction-Guided 3D Program Editing
Organizations: New York University · Carnegie Mellon University · UNC, Chapel Hill · University of Illinois Urbana-Champaign ·
Abstract
Language models increasingly create 3D content through code, with practical workflows often relying on editing existing programs rather than regenerating them, for speed and precise control. Such edits require understanding what a program builds and foreseeing how a change reshapes it, yet prior benchmarks mostly evaluate generation or simple edits against a reference. In this paper, we introduce 3DCodeEdit, a benchmark of 2,592 instruction-guided editing tasks over 108 Blender programs that construct objects entirely in code. Its edits range from parameter to structural changes, style redesign, and scene composition, and are judged from renders against human-reviewed checklists. Across 17 models, the average success rate is only 37%, and even the strongest models complete fewer than 60% of structural and style edits. As models become more capable, their failures shift to geometric foresight, where the code runs but builds the wrong shape. We therefore study whether multi-turn feedback can compensate for this missing foresight. Feedback from a separate reviewer model that describes how the renders differ from the request helps most, more than maximum reasoning effort or an agent harness, and can let a 27B model surpass the single-shot rate of a 2.8T-parameter model.
Figures & tables
| Task | Programs | Evaluation | ||||
|---|---|---|---|---|---|---|
| Benchmark | Editing | End-to-End Code | Complex Code | Multiple Sources | Difficulty Levels | Reference-Free Eval. |
| CADBench ( Du et al., 2024 ) | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ |
| 3DCodeBench ( Gao et al., 2026 ) | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ |
| 3DHarnessBench ( Liu et al., 2026b ) | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ |
| Nova3D-Bench ( Noor et al., 2026 ) | ✗ | ✓ | ✓ | ✗ | ✓ | ✓ |
| BlenderGym ( Gu et al., 2025 ) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
Table 1: Comparison of benchmarks for Blender code. 3DCodeEdit combines instruction-guided editing, end-to-end object construction, complex programs from multiple sources, graded edit difficulty, and reference-free evaluation. In end-to-end code , the program builds the entire object; complex code indicates an average program length exceeding 200 lines (Appendix B ); reference-free evaluation judges edits against the instruction rather than a reference output.
Figure 2: Construction of 3DCodeEdit. (1) A generator and an evaluator iteratively reconstruct OpenSCAD designs as editable Blender programs, preserving geometry while exposing parameters, parts, and assembly logic. We combine 53 reconstructed programs with 55 from 3DCodeBench. (2) For each program and edit type, we generate edit prompts paired with checklists of requested changes and preservation requirements. After filtering, selection, and ambiguity screening under human supervision, we obtain four tasks per program and edit type, totaling 2,592 tasks.
| Type | Name | Operational definition |
| Geometric edits, graded by scope | ||
| G1 | Parameter | Change one visible attribute (size, proportion, spacing, position, or orientation) of one part or part family. |
| G2 | Component | Reconstruct the form of one part, together with a size change and one further visible change to it. |
| G3 | Multi-component | Reconstruct and resize two separate parts independently. |
| G4 | Structural | Reorganize how three or more regions meet, span, or support one another, changing the overall layout. |
| Open-ended edits | ||
Table 2: Edit types in 3DCodeEdit and their operational definitions.
| Model | Geometric | G1 | G2 | G3 | G4 | Open-ended | S | C | All | Exec. | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Proprietary | Claude Opus 5.5 | 96.5 | 76.9 | 64.4 | 58.6 | 52.5 | 70.8 | 69.9 | 99.3 | ||
| GPT-6.1-sol | 95.6 | 71.3 | 65.3 | 59.3 | 57.6 | 65.7 | 69.1 | 96.1 | |||
| Gemini 4 Argon | 94.7 | 67.4 | 54.2 | 41.0 | 41.0 | 53.0 | 58.5 | 95.4 | |||
| Gemini 3.8 Flash | 94.7 | 68.8 | 62.7 | 53.7 | 34.5 | 36.3 | 58.4 | 92.0 | |||
| Claude Sonnet 5.5 | 91.4 | 66.4 | 51.2 | 41.9 | 25.9 | 45.4 | 53.7 | 98.8 | |||
| Claude Haiku 5.5 | 90.3 | 54.4 | 42.8 | 31.5 | 16.0 | 32.9 | 44.6 | 94.3 |
Table 3: Single-shot editing performance (%) on 3DCodeEdit. All-pass rate measures the percentage of edits whose programs execute, render, and satisfy every checklist item. G1–G4 denote geometric edits of increasing scope; S and C denote style and composition edits. Exec. reports execution and rendering success. Average is the mean over the 17 models. Five Qwen3.5 models are reported in Table 12 .
Figure 3: Statistics of 3DCodeEdit. (a) The 108 source programs by object category. (b) Program length in lines of code (log scale; shaded: middle 50%; ticks: individual programs). (c) Words per edit prompt by edit type (G1–G4, S, C; defined in Table 2 ; dot: median; bar: interquartile range; whisker: 5th–95th percentile), with the number of edit instructions below.
Figure 4: Progress in 3D program editing on the analysis subset (312 tasks). (a) All-pass rate of 44 models; lines join releases within four families, and the dashed line is the best rate so far. (b) Best rate per edit type among models released before April 2026 and among all models.
| Share of failures | ||||||
|---|---|---|---|---|---|---|
| All-pass band | I | P | K | G | O | |
| 20% | 19 | 11% | 34% | 37% | 18% | 0% |
| 20–40% | 19 | 11% | 29% | 23% | 37% | 0% |
| 40–55% | 23 | 10% | 24% | 22% | 42% | 2% |
| 55% | 15 | 7% | 27% | 23% | 39% | 4% |
Table 4: Distribution of failure stages by editing performance. Each failure is assigned to the first stage at which the edit diverges from the instruction. I : instruction comprehension; P : program comprehension; K : code execution; G : geometric foresight; O : visibility in renders. As performance improves, failures shift from comprehension and coding toward geometric foresight.
| Self-check | Indep. rev. | |||||
|---|---|---|---|---|---|---|
| Editor | T0 | Code | Render | P/F | Full | Self-rev. |
| GPT-6.1-sol | 64.7 | 71.5 | 76.0 | 75.6 | 85.6 | 83.0 |
| Claude Opus 5.5 | 67.6 | 68.3 | 74.0 | 81.7 | 83.0 | 83.3 |
| Claude Sonnet 5.5 | 50.6 | 55.1 | 55.4 | 67.3 | 71.2 | 69.9 |
| GPT-6-luna | 36.9 | 42.6 | 49.7 | 59.0 | 65.7 | 61.5 |
| Qwen3.8-27B | 26.0 | 31.4 | 33.0 | 43.6 | 50.3 | 37.2 |
Table 5: All-pass rate (%) of multi-turn editing on the analysis subset (§ 4.4 ). T0 : single-shot edit. Indep. rev. : independent reviewer (GPT-6-luna, or GPT-6.1-sol for GPT-6-luna). Self-rev. : self-review. –: render self-check unavailable for text-only editors.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Reconstruction of the NopSCADlib flat hinge, shown from one oblique view. The evaluator accepted round 22. Its first review began: “The main hinge leaf outline is substantially too regular and too large/deep compared with the source CSG leaf shape.”
Figure 6: Example source objects from the four source groups of 3DCodeEdit: a pineapple (546 lines of code) from 3DCodeBench, and a board-and-batten cottage (677), a small-parts cabinet (717), and a flat hinge (760) reconstructed from OpenSCAD, rendered from the source programs.
Figure 7: Twenty-four of the 108 source programs: 3DCodeBench objects (grey) and reconstructed designs (coloured).
| Helical gear, G3, term |
| Before: enlarge each of the seven spiral-arranged openings by about 50% |
| After: enlarge each of the seven spiral-arranged openings to about 1.5 times its original diameter |
| Pulley scene, C, conflict between instructions |
| Before: add a pedestal-mounted two-link robotic arm with a large two-finger gripper reaching toward the pulley |
| After: add a two-link robotic arm rising from a pedestal block fixed to the gantry, with a large two-finger gripper reaching toward the pulley |
| Conflicting instruction: build a large open gantry spanning the workbench, carrying the robotic arm (after: the robotic arm’s pedestal block) above the turntable […] |
Table 6: Two instructions before and after ambiguity rewriting. Their checklist items were rewritten in the same way.
| Benchmark | Programs | Mean | Med. | |
| CADBench a | BlendNet | 12,059 | 38 | 36 |
| 3DCodeBench | Reference | 212 | 476 | 363 |
| 3DHarnessBench | Reference | 100 | 876 | 784 |
| Nova3D-Bench b | Generated | 54 | 489 | 402 |
| BlenderGym c | Start | 280 | 51 | 40 |
| BlenderBench | Start | 27 | 25 | 24 |
Table 7: Program length (lines) of Blender code benchmarks. Med.: median.
Figure 8: Two of the six verifier views for the task of Example C .
Figure 9: G1: rotate the fern’s outer fronds upward so they rise steeply instead of drooping.
Figure 11: G2: rebuild the dorsal fin as a smooth semicircular sail, twice as tall and longer.
Figure 13: G4: rebuild the desk legs as two A-frames with a thick shelf between them.
Figure 15: C: place the mushroom cluster on a field table under a canopy holding a lantern and a magnifying lens.
| Renders | Code | Both | ||||
|---|---|---|---|---|---|---|
| Verifier | Acc. | Acc. | Acc. | |||
| Gemini 3.8 Flash (high) | 90.4 | 0.80 | 83.5 | 0.62 | 91.7 | 0.82 |
| GPT-6.1-sol | 93.8 | 0.87 | 84.3 | 0.67 | 92.9 | 0.85 |
| GPT-6-astra | 91.9 | 0.83 | 84.3 | 0.67 | 91.9 | 0.83 |
| Claude Opus 5.5 | 91.4 | 0.82 | 85.7 | 0.69 | 91.0 | 0.81 |
| Claude Sonnet 5.5 | 85.2 | 0.69 | 81.0 | 0.57 | 84.3 | 0.67 |
Table 8: Agreement with human labels on the 210 checklist items when the verifier sees the renders (our protocol), the original and edited programs, or both. Acc. in %. Gemini 3.8 Flash averages five calls on renders and three calls otherwise; the other verifiers use one call. GPT-6-astra uses reasoning effort medium here.
| Verifier | Acc. | |
|---|---|---|
| Main verifier and reference | ||
| Gemini 3.8 Flash (low) | 87.7 | 0.74 |
| Gemini 3.8 Flash (medium) | 90.0 | 0.79 |
| Gemini 3.8 Flash (high) ∗ | 90.4 | 0.80 |
| Gemini 3.8 Flash (max) | 90.8 | 0.80 |
| GPT-6.1-sol | 93.8 | 0.87 |
Table 9: Verifier agreement with human labels on 210 checklist items from the analysis subset. Acc. in %. Single call per item; Gemini 3.8 Flash rows average five independent calls. ∗ Configuration used to score all experiments.
| Self-check | Indep. rev. | ||||
|---|---|---|---|---|---|
| Editor | Code | Render | P/F | Full | Self-rev. |
| GPT-6.1-sol | 0.3 | 0.3 | 3.4 | 2.8 | 1.5 |
| Claude Opus 5.5 | 0.2 | 0.5 | 3.5 | 2.7 | 1.6 |
| Claude Sonnet 5.5 | 0.4 | 0.6 | 4.3 | 3.8 | 4.7 |
| GPT-6-luna | 1.8 | 2.7 | 4.5 | 3.2 | 4.2 |
| Qwen3.8-27B | 1.5 | 1.6 | 5.5 | 5.0 | 2.6 |
Table 10: Mean rounds of revision in each multi-turn setting of Table 5 (analysis subset, at most 8). –: text-only editor.
| Setting | Luna | Sol |
|---|---|---|
| Single-shot edit | 36.9 | 64.7 |
| Single-shot edit, best effort | 54.2 | 74.7 |
| Codex harness | 34.6 | 74.7 |
| Codex harness + self-review | 59.6 | 84.9 |
| Multi-turn editing (self-review) | 61.5 | 83.0 |
Table 11: All-pass rate (%) of a coding agent (Codex CLI) and multi-turn editing on the analysis subset. Luna: GPT-6-luna; Sol: GPT-6.1-sol. Best effort: max for Luna, xhigh for Sol.
| All-pass rate | ||||||||
|---|---|---|---|---|---|---|---|---|
| Model | G1 | G2 | G3 | G4 | S | C | All | Exec. |
| Claude Opus 5.5 | 96.5 | 76.9 | 64.4 | 58.6 | 52.5 | 70.8 | 69.9 | 99.3 |
| GPT-6.1-sol | 95.6 | 71.3 | 65.3 | 59.3 | 57.6 | 65.7 | 69.1 | 96.1 |
| Gemini 4 Argon | 94.7 | 67.4 | 54.2 | 41.0 | 41.0 | 53.0 | 58.5 | 95.4 |
| Gemini 3.8 Flash | 94.7 | 68.8 | 62.7 | 53.7 | 34.5 | 36.3 | 58.4 | 92.0 |
| Claude Sonnet 5.5 | 91.4 | 66.4 | 51.2 | 41.9 | 25.9 | 45.4 | 53.7 | 98.8 |
Table 12: All-pass rate (%) of all 22 evaluated models by edit type (G1–G4: geometric edits graded by scope; S: style; C: composition), sorted by overall rate; the best value in each column is in bold. Exec.: percentage of edited programs that execute and render. Average: mean over the 22 models.
| Model | Effort | G1 | G2 | G3 | G4 | S | C | All | Exec. | Tokens |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | low | 94.2 | 61.5 | 48.1 | 51.9 | 50.0 | 53.8 | 59.9 | 100.0 | 11.1 |
| medium | 96.2 | 73.1 | 55.8 | 51.9 | 51.9 | 76.9 | 67.6 | 99.4 | 14.7 | |
| high | 98.1 | 71.2 | 61.5 | 57.7 | 51.9 | 73.1 | 68.9 | 99.4 | 17.9 | |
| xhigh | 94.2 | 71.2 | 69.2 | 57.7 | 61.5 | 78.8 | 72.1 | 99.7 | 38.8 | |
| GPT-6.1-sol | low | 94.2 | 73.1 | 59.6 | 59.6 | 51.9 | 63.5 | 67.0 | 96.2 | 7.1 |
| medium | 96.2 | 63.5 | 53.8 | 51.9 | 48.1 | 75.0 | 64.7 | 94.6 | 7.8 |
Table 13: All-pass rate (%) by reasoning effort on the analysis subset (312 tasks; 52 per edit type). For the six models that are also in Table 12 , the medium level is their main run (at most 64k output tokens). All other levels use the provider’s maximum output length: 128k tokens for Claude and Kimi K3, 65,536 for Gemini 3.8 Flash, and no limit for GPT models. At this limit, the Claude models at max effort are truncated on up to 54% of a 24-task pilot, so we omit that level. Exec.: percentage of edited programs that execute and render. Tokens: mean output tokens per task in thousands, including reasoning tokens. The best overall rate of each model is in bold.
| Model | Released | Family | G1 | G2 | G3 | G4 | S | C | All | Exec. |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-Coder-32B | 2024-11-12 | Qwen 27–32B | 50.0 | 3.8 | 0.0 | 0.0 | 0.0 | 0.0 | 9.0 | 57.7 |
| QwQ-32B | 2025-03-06 | Qwen 27–32B | 46.2 | 5.8 | 0.0 | 1.9 | 0.0 | 0.0 | 9.0 | 41.3 |
| Qwen3-32B | 2025-04-29 | Qwen 27–32B | 48.1 | 9.6 | 0.0 | 0.0 | 0.0 | 0.0 | 9.6 | 52.2 |
| Claude Sonnet 4 | 2025-05-22 | Claude Sonnet | 71.2 | 21.2 | 7.7 | 3.8 | 0.0 | 0.0 | 17.3 | 89.4 |
| Gemini 2.5 Flash | 2025-06-17 | Gemini Flash | 67.3 | 13.5 | 7.7 | 5.8 | 0.0 | 0.0 | 15.7 | 56.7 |
| gpt-oss-120b ∗ | 2025-08-05 | gpt-oss | 75.0 | 9.6 | 0.0 | 1.9 | 0.0 | 0.0 | 14.4 | 75.3 |
Table 14: The 44 models of the timeline analysis on the analysis subset (312 tasks), sorted by release date: all-pass rate (%) by edit type and overall, and the percentage of edited programs that execute and render (Exec.). ∗ : one of the 22 models of Table 12 . † : approximate release date. Claude Sonnet 4.6, Claude Sonnet 5, and GPT-5.6-luna are the medium levels of their effort sweeps.
| Type | Opus 5.5 | GPT-6.1-sol | Either | |
|---|---|---|---|---|
| G1 | 96.5 | 95.6 | 98.1 | 0.45 |
| G2 | 76.9 | 71.3 | 85.6 | 0.40 |
| G3 | 64.4 | 65.3 | 78.0 | 0.42 |
| G4 | 58.6 | 59.3 | 73.6 | 0.39 |
| S | 52.5 | 57.6 | 74.3 | 0.23 |
| C | 70.8 | 65.7 | 86.1 | 0.18 |
Table 15: The two strongest models on the full benchmark: all-pass rate (%) of Claude Opus 5.5 and GPT-6.1-sol, the percentage of tasks solved by at least one of them (Either), and the task-level correlation of their passes ( ).
| Code | Definition | Share |
| Instruction: what to build | ||
| I-OMIT | A requested change is not attempted | 4.1% |
| I-MISREAD | A different part, feature, or relation is built | 5.3% |
| I-LITERAL | The wording is met but the intent is defeated | 0.3% |
| I-OVERREACH | Rebuilt far beyond the request | 1.0% |
| Program: understanding the source | ||
Table 16: Codebook of the error census. Each failure receives one stage, the earliest at which the edit diverges from the instruction, and one mechanism within it. Share: percentage of all failures of the 76 configurations on the analysis subset that receive the code.
| Configuration | Pass | I | P | K | G | O | Crash |
|---|---|---|---|---|---|---|---|
| GPT-6.1-sol (xhigh) | 74.7% | 7% | 45% | 19% | 21% | 7% | 17 |
| Claude Sonnet 5.5 (xhigh) | 73.7% | 13% | 21% | 21% | 42% | 3% | 3 |
| Claude Opus 5.5 (xhigh) | 72.1% | 0% | 27% | 17% | 53% | 3% | 0 |
| GPT-6.1-sol (high) | 70.8% | 6% | 32% | 28% | 30% | 4% | 20 |
| GPT-6.1-sol (max) | 70.5% | 4% | 29% | 29% | 29% | 10% | 21 |
| Claude Opus 5.5 (high) | 68.9% | 10% | 18% | 16% | 49% | 8% | 2 |
Table 17: All-pass rate (Pass) and stage mix of failures for all 76 configurations on the analysis subset, sorted by all-pass rate. Stages: I instruction, P program, K code, G geometric foresight, O visibility. Crash: number of tasks, out of 312, in which the edited program crashes. Effort levels other than medium are given in parentheses.