Creating high-quality 3D content has traditionally required specialist expertise and experience with complex tools. The growing ecosystem of AI-assisted creativity demands accessible and intuitive 3D generation systems. Recent advances in Large Language Models (LLMs) offer promising directions, yet extending their generative capabilities to 3D remains fundamentally challenging because existing 3D representations are not naturally compatible with language-based reasoning. Current approaches either depend on expensive fine-tuning or impose restrictive structural priors that sacrifice expressiveness and exclude organic shapes. We argue that the mismatch between LLM-native reasoning and conventional 3D parameterizations is a core bottleneck of LLM-driven 3D generation. To address this, we introduce the Hierarchical Patch Scaffold (HPS), a representation designed around semantically meaningful, interpretable part-level abstractions that are naturally understood by both humans and LLMs, enabling expressive, editable, and open-domain 3D generation without imposing restrictive geometric priors or expensive fine-tuning. Building upon HPS, we introduce SurfScaff3D, an interactive framework for text and image-to-3D generation. HPS inherits the flexibility of B-spline patches, while exposing an interface that leverages open-world capabilities of foundation models to reliably generate 3D. Its hierarchical part-level semantic labels enable targeted, natural-language edits at any level of granularity without re-running generation. Further, HPS offers explicit per-patch control over surface detail, enabling more expressive geometry capable of representing arbitrary objects. Extensive experiments against state-of-the-art LLM-based 3D generation methods demonstrate strong geometric fidelity, semantic alignment, and flexible editing across diverse object categories.
Figures & tables
Figure 2: Qualitative Comparison of SurfScaff3D. We provide qualitative examples of SurfScaff3D against similar works, on the left we represent methods which allow greater geometric flexibility but require fine-tuning, in the middle we show methods which benefit from the zero-shot generalizability of modern LLMs but suffer from restrictive geometric priors. On the right we show SurfScaff3D using the open-source Kimi K2.6 [ 20 ] and Claude Sonnet 4.6 [ 4 ] as generation backbones, respectively. Each row corresponds to a single object prompt, all methods are generated using the identical prompts to ensure a fair qualitative comparison.
Figure 3: Overview of SurfScaff3D . SurfScaff3D iteratively constructs an HPS representation by programmatically generating surface geometry and verifying its local geometry through rendered views. HPS organizes geometry as a hierarchical semantic tree of B-spline surface patches with explicit patch-level control over surface variation, enabling both structured mechanical objects and organic shapes. The examples illustrate how frequency and amplitude help in modelling diverse objects. A critic evaluates rendered outputs and provides targeted feedback, forming a closed-loop refinement process that progressively improves prompt alignment and geometric fidelity. The critic loop executes for a fixed number of N iterations unless an early stopping criterion is met.
Figure 4: Surface Variation . An organic broccoli model generated by SurfScaff3D, demonstrating the expressive capacity of the HPS smoothness parameters. By modulating the local smoothness parameters sf and ωf , the model synthesizes complex, high-frequency organic geometry without losing geometric coherence.
Figure 5: Human-as-the-Critic Refinement. SurfScaff3D allows integration of the user as a real-time critic within the generative loop. By providing natural language feedback, the human critic can drive targeted and personalized refinements.
ShapeNetCore
Toys4K
OmniObject3D
Prompt Align
3D Qual
Prompt Align
3D Qual
Prompt Align
3D Qual
Failure
Text-to-3D Model
CLIP ↑
ULIP ↑
VQA ↑
FPD ↓
Elo ↑
CLIP ↑
ULIP ↑
VQA ↑
FPD ↓
Elo ↑
CLIP ↑
ULIP ↑
VQA ↑
FPD ↓
Elo ↑
(%)
BlenderLLM [ 14 ]
24.9
21.9
38.3
930
0 763
24.2
20.6
35.3
0 397
0 670
25.2
19.1
41.9
991
0 780
36.4
LLaMA-Mesh [ 37 ]
23.4
16.6
24.4
305
0 696
23.2
13.2
21.8
0 298
0 648
23.8
15.8
37.9
298
0 646
3 0.9
NURBGen [ 35 ]
23.6
0 7.3
23.0
736
0 716
23.4
0 6.6
19.6
1198
0 759
24.1
0 7.6
23.7
706
0 660
0 0.9
ShapeCraft [ 50 ]
24.6
20.6
38.8
378
0 910
24.9
20.8
39.6
0 279
1030
25.1
20.1
42.0
251
0 922
0 1.3
Table 1: Evaluation of Text-to-3D Generation. We evaluate the prompt fidelity using CLIP, ULIP scores, and VQA pass rate and 3D quality using FPD [ 32 ] and an Elo scoring based on GPT-5.4-mini [ 26 ] preferences (GPT represents an independent LLM family which was not a part of any of the evaluated pipelines). We evaluate generation across three 3D datasets on captions from MARVEL-40M+ [ 33 ] . The zero-failure rate of SurfScaff3D across different backbone variants demonstrates the reliability of HPS as an intermediate representation.
Table 2: Image-to-3D. We evaluate image-conditioned 3D generation on OmniObject3D and DeepCAD, reporting geometric accuracy (Chamfer, [email protected]), conditioning fidelity (CLIP-I), and novel-view reconstruction quality (LPIPS).
Figure 6: Qualitative Image-to-3D Comparison. We present qualitative comparisons of various organic, synthetic and CAD objects. The source image is provided without any textual description of the object. HPS allows greater flexibility in geometry.
Model
CLIP ↑
ULIP ↑
VQA ↑
FPD ↓
Fail%
LLaMA-Mesh [ 37 ]
26.6
21.0
32.4
740
62.2
NURBGen [ 35 ]
24.9
0 9.4
42.6
786
0 1.1
ShapeCraft [ 50 ]
25.9
21.1
59.7
909
31.1
SurfScaff3D-K
29.5
25.6
75.6
426
0 0.0
Table 3: Editing. We evaluate the editing capability of various LLM-based text-to-3D models. We employ CLIP, ULIP and VQA scores along with FPD to evaluate the quality of the edited 3D assets and utilise edit prompts from ShapeTalk [ 1 ] .
Qwen3.6
KimiK2.6
Dataset
HPS
w/o SV
HPS
w/o SV
ShapeNetCore
30
25
31
24
Toys4K
64
41
60
45
OmniObject3D
46
25
41
30
Table 4: Impact of Surface Variation. Pairwise preference (GPT-5.4-mini [ 26 ] , same protocol as the Elo scores in Table 1 ) between the HPS output and the same output exported without surface variation (w/o SV), per dataset, across two open-source backbones. For each backbone, the two columns add up to the number of objects in the dataset (55, 105 and 71). Higher is better ↑ .
Figure 10
Qwen3.6 35B-A3B
KimiK2.6 1T-A32B
Setting
FPD ↓
Elo Score ↑
FPD ↓
Elo Score ↑
Cloud
929
936
628
863
Control
1192
960
745
930
HPS
571
1103
391
1206
Table 5: Isolating Impact of HPS. In Cloud, the agent outputs a point cloud without additional structure. In Control, each patch is specified by control points instead of surface points. HPS is far more effective than both. All settings are evaluated without critic iterations and with the same set of in-context objects, to isolate the impact of the representations.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Prompt for editing existing HPS representations.
Figure 10: Qualitative Samples for text-to-3D generation. We show samples generated by SurfScaff3D from the object categories of ShapeNetCore [ 8 ] , Toys4K [ 34 ] and OmniObject3D [ 40 ] datasets. The corresponding MARVEL-40M+ level-3 text prompts are displayed below each result.
Figure 11: Qualitative samples for image-to-3D. We demonstrate the performance of SurfScaff3D on diverse image inputs. The source conditioning images are provided below each resulting 3D asset.
Figure 12: Editing capabilities of our model. We display the original 3D object alongside the edited result. The corresponding ShapeTalk-based text prompts used to guide the edits are shown on the arrows.
Figure 13: System Prompt for SurfScaff3D Generator Agent.
Figure 14: System prompt for SurfScaff3D critic agent.
Figure 15: Prompt appended to SurfScaff3D critic agent when an image is provided.
Figure 16: Failure cases of SurfScaff3D. The model can misjudge the orientation and relative positioning of object parts (left), and in some cases leaves parts detached from the body, producing visible gaps in the generated surface (right).
Method
TF
PS
IR
PE
TG-Gen
TG-Edit
IM
Representation
BlenderLLM [ 14 ]
×
×
×
×
✓
×
×
CAD scripts (Blender Python/bpy)
LLaMA-Mesh [ 37 ]
×
×
×
×
✓
×5
×
Text-serialized mesh (vertices + faces)
NURBGen [ 35 ]
×
×
×
×
✓
×5
×
NURBS patches and analytic primitives
CAD-Coder [ 13 ]
×
×
×
×
×1
×
✓
CadQuery (Python)
CADCodeVerify [ 2 ]
✓
×
✓
×
✓
×
×
CAD scripting code (Python)
ShapeCraft [ 50 ]
✓ 2
✓
✓
✓
✓
✓
×
Graph-based Procedural Shape (GPS)
Appendix
Table 6: Comparison of capabilities and representations of related 3D generation and editing frameworks. TF = training-free; PS = decomposition into explicitly labelled semantic parts; IR = inference-time iterative/interactive refinement; PE = part-level/local editing without regenerating the complete object; TG-Gen = text-to-3D generation from scratch; TG-Edit = text-guided editing of an existing asset; IM = image-conditioned input. 1 CAD-Coder’s text input is a fixed instruction template rather than an object description, and all its reported results are image-conditioned. 2 Neither method trains a task-specific model, but both rely on per-instance test-time optimization (ShapeCraft uses score distillation for texturing, and MvDrag3D uses deformation and score distillation). 3 The Articraft paper describes only local edits made during generation, though its released code also supports text-guided editing of a completed asset. 4 Prox-E’s superquadric primitives carry only a numeric index and colour rather than semantic part labels. 5 Neither method supports editing natively. For Table 3 of the main paper, we provide the source shape in each model’s own format along with the edit instruction.
Backbone
Total Params
Act. Params
CLIP ↑
ULIP ↑
VQA ↑
FPD ↓
Qwen3.6-35B-A3B [ 28 ]
0 35B
0 3B
28.1
22.8
60.5
384
MiMo-V2.5 [ 43 ]
310B
15B
29.1
26.1
62.4
344
DeepSeek-V4-flash-0731 [ 12 ]
284B
13B
29.3
25.9
64.8
313
Appendix
Table 7: SurfScaff3D with other backbones. SurfScaff3D with each backbone, evaluated on OmniObject3D under the same protocol as Table 1 of the main paper. The MiMo and DeepSeek runs use three critic iterations, with Gemini 2.5 Pro [ 11 ] as the critic. Active parameter counts are reported since all three backbones are mixture-of-experts models.
Text-to-3D generation has advanced rapidly, yet state-of-the-art models, encompassing both optimization-based and feed-forward architectures, still face two fundamental limitations. First, they struggle with coarse semantic alignment, often failing to capture fine-grained prompt details. Second, they lack robust 3D spatial understanding, leading to geometric inconsistencies and catastrophic failures in part assembly and spatial relationships. To address these challenges, we propose VLM3D, a general framework that repurposes large vision-language models (VLMs) as powerful, differentiable semantic and spatial critics. Our core contribution is a dual-query critic signal derived from the VLM's Yes or No log-odds, which assesses both semantic fidelity and geometric coherence. We demonstrate the generality of this guidance signal across two distinct paradigms: (1) As a reward objective for optimization-based pipelines, VLM3D significantly outperforms existing methods on standard benchmarks. (2) As a test-time guidance module for feed-forward pipelines, it actively steers the iterative sampling process of SOTA native 3D models to correct severe spatial errors. VLM3D establishes a principled and generalizable path to inject the VLM's rich, language-grounded understanding of both semantics and space into diverse 3D generative pipelines.
Understanding and generating 3D objects as compositions of meaningful parts is fundamental to human perception and reasoning. However, most text-to-3D methods overlook the semantic and functional structure of parts. While recent part-aware approaches introduce decomposition, they remain largely geometry-focused, lacking semantic grounding and failing to model how parts align with textual descriptions or their inter-part relations. We propose DreamPartGen, a framework for semantically grounded, part-aware text-to-3D generation. DreamPartGen introduces Duplex Part Latents (DPLs) that jointly model each part's geometry and appearance, and Relational Semantic Latents (RSLs) that capture inter-part dependencies derived from language. A synchronized co-denoising process enforces mutual geometric and semantic consistency, enabling coherent, interpretable, and text-aligned 3D synthesis. Across multiple benchmarks, DreamPartGen delivers state-of-the-art performance in geometric fidelity and text-shape alignment.
Recently, Large Language Models (LLMs) have emerged as promising layout agents for 3D scene generation. Existing layout agents still suffer from implausible layout generation because most of them convert 3D assets and 3D layouts into textual descriptions as inputs and outputs, which involves severe information loss due to the modality gap between texts and 3D assets and 3D layouts. We propose NaLA, a native 3D LLM layout Agent for high-quality 3D scene generation by placing 3D assets in the scene. For the inputs, NaLA encodes 3D scene boundaries and 3D assets directly into the LLM, preserving fine-grained geometry and enabling explicit reasoning over relationships like collisions, surface supporting, and containment. To accurately output the positions and orientations of assets, NaLA adopts a coarse-to-fine prediction mechanism that first predicts discrete poses in an autoregressive manner and then refines the discrete poses with a continuous regression. Trained on diverse layout datasets, NaLA attains strong geometric perception and layout coherence. Experiments demonstrate that NaLA outperforms prior layout agents in both generation quality and inference efficiency, with comprehensive ablation studies to verify each component's effectiveness.
Cheng Wan, Yongsen Mao, Wenzheng Wu +7
Hong Kong University of Science and Technology · Shenzhen Loop Area Institute · University of Science and Technology of China +2