Organizations: Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University, Shenzhen, China · Math Magic, Beijing, China · Peking University, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong, China · Dalian University of Technology, Dalian, China
Recent advances in 3D generation have improved the fidelity and geometric details of synthesized 3D assets. However, due to the inherent ambiguity of single-view observations and the lack of robust global structural priors caused by limited 3D training data, the unseen regions generated by existing models are often stochastic and difficult to control, which may sometimes fail to align with user intentions or produce implausible geometries. In this paper, we propose Know3D, a novel framework that incorporates rich knowledge from multimodal large language models into 3D generative processes via latent hidden-state injection, enabling language-controllable generation of the back-view for 3D assets. We utilize a VLM-diffusion-based model, where the VLM is responsible for semantic understanding and guidance. The diffusion model acts as a bridge that transfers semantic knowledge from the VLM to the 3D generation model. In this way, we successfully bridge the gap between abstract textual instructions and the geometric reconstruction of unobserved regions, transforming the traditionally stochastic back-view hallucination into a semantically controllable process, demonstrating a promising direction for future 3D generation models.
Figures & tables
Figure 1: We propose Know3D, a knowledge-guided 3D generation framework that enables semantic control over the back-view of target objects. As shown in the coffee-cup scene, by simply varying the text, Know3D can synthesize diverse and structurally consistent back-view components. The bottom-row results demonstrate the potential of our method to improve the structural plausibility of unseen parts in 3D generation.
Figure 2: Text annotation pipeline. Given paired front and back views of a 3D asset, a VLM generates initial part-level descriptions.
Figure 3: Overview . Given an input image and a text prompt, Know3D leverages Qwen-Image-Edit-2511 [ 56 ] , which integrates Qwen2.5-VL with a Diffusion Transformer (DiT). Qwen2.5-VL [ 2 ] encodes the multimodal inputs to guide a latent denoising process, enabling semantically aligned backside generation. The latent hidden states from intermediate DiT layers, which embed rich semantic and structural priors, are subsequently injected as conditioning signals into the 3D generation model to guide complete shape synthesis.
Figure 4: Back-View Semantic Control Visualization. The leftmost column shows the input front-view condition images. The remaining columns present back-view normal renderings of meshes generated under different text prompts, demonstrating flexible semantic control over back-side components.
Figure 5: Visual Comparison between Know3D and Hunyuan3D-2mv. As shown in (a) and (b), both Know3D and Hunyuan3D-2mv [ 48 ] achieve reasonably good alignment between the input and generated viewpoints. However, for novel viewpoints, Hunyuan3D-2mv produces implausible structures(d), whereas Know3D still yields reasonable results(e).
Figure 6: Visual Comparison between Know3D and Current SOTA Baselines. By leveraging semantic knowledge, our method taps into the intrinsic understanding of object attributes within pre-trained multimodal models, showing potential for improving the structural plausibility of unseen parts in 3D generation.
HY3D-Bench-test
HY3D-Bench-val
Model
ULIP ↑
Uni3D ↑
ULIP ↑
Uni3D ↑
Direct3D-S2 [ 57 ]
0.2034
0.3292
0.2040
0.3303
Hi3DGen [ 64 ]
0.2082
0.3206
0.2030
0.3213
Step1X-3D [ 28 ]
0.2068
0.3233
0.2033
0.3265
TRELLIS [ 60 ]
0.2143
0.3421
0.2133
0.3464
TRELLIS.2 [ 59 ]
0.1948
0.3308
0.1967
0.3358
Table 1: Quantitative comparison on HY3D-Bench [ 21 ] . The best results are in bold and the second-best are underlined. ↑ indicates higher is better.
Figure 7: Visualization of Different Feature Representations to Prompt 3D Generation. When Qwen-Image generates an unreasonable back-view, using different feature representations to prompt 3D generation will lead to different results. The hidden states from the intermediate layers of MMDiT during the denoising process still generate reasonable results, while the VAE fails to inject the back-view information, making it quite similar to the unreasonable results of the original TRELLIS.2. Additionally, when the generated image is directly passed through DINOv3, DINOv3 tends to fit the erroneous results.
Text-to-3D generation has advanced rapidly, yet state-of-the-art models, encompassing both optimization-based and feed-forward architectures, still face two fundamental limitations. First, they struggle with coarse semantic alignment, often failing to capture fine-grained prompt details. Second, they lack robust 3D spatial understanding, leading to geometric inconsistencies and catastrophic failures in part assembly and spatial relationships. To address these challenges, we propose VLM3D, a general framework that repurposes large vision-language models (VLMs) as powerful, differentiable semantic and spatial critics. Our core contribution is a dual-query critic signal derived from the VLM's Yes or No log-odds, which assesses both semantic fidelity and geometric coherence. We demonstrate the generality of this guidance signal across two distinct paradigms: (1) As a reward objective for optimization-based pipelines, VLM3D significantly outperforms existing methods on standard benchmarks. (2) As a test-time guidance module for feed-forward pipelines, it actively steers the iterative sampling process of SOTA native 3D models to correct severe spatial errors. VLM3D establishes a principled and generalizable path to inject the VLM's rich, language-grounded understanding of both semantics and space into diverse 3D generative pipelines.
Generative models have achieved success in producing semantically plausible 2D images, but it remains challenging in 3D generation due to the absence of spatial geometry constraints. Typically, existing methods utilize geometric features as conditions to enhance spatial awareness. However, these methods can only model relative relationships and are prone to scale inconsistency of absolute geometry. Thus, we argue that semantic information and absolute geometry empower 3D cognition, thereby enabling controllable 3D generation for the physical world. In this work, we propose Cog2Gen3D, a 3D cognition-guided diffusion framework for 3D generation. Our model is guided by three key designs: 1) Cognitive Feature Embeddings. We encode different modalities into semantic and geometric representations and further extract logical representations. 2) 3D Latent Cognition Graph. We structure different representations into dual-stream semantic-geometric graphs and fuse them via common-based cross-attention to obtain a 3D cognition graph. 3) Cognition-Guided Latent Diffusion. We leverage the fused 3D cognition graph as the condition to guide the latent diffusion process for 3D Gaussian generation. Under this unified framework, the 3D cognition graph ensures the physical plausibility and structural rationality of 3D generation. Moreover, we construct a validation subset based on the Marble World Labs. Extensive experiments demonstrate that our Cog2Gen3D significantly outperforms existing methods in both semantic fidelity and geometric plausibility.
Haonan Wang, Hanyu Zhou, Haoyue Liu +2
School of Artificial and Automation, Huazhong University of Science and Technology · School of Computing, National University of Singapore · School of Computing, Macquarie University
Recent progress in 3D generation has been driven largely by models conditioned on images or text, while readily available 3D priors are still underused. In many real-world scenarios, the visible-region point cloud are easy to obtain from active sensors such as LiDAR or from feed-forward predictors like VGGT, offering explicit geometric constraints that current methods fail to exploit. In this work, we introduce Points-to-3D, a diffusion-based framework that leverages point cloud priors for geometry-controllable 3D asset and scene generation. Built on a latent 3D diffusion model TRELLIS, Points-to-3D first replaces pure-noise sparse structure latent initialization with a point cloud priors tailored input formulation.A structure inpainting network, trained within the TRELLIS framework on task-specific data designed to learn global structural inpainting, is then used for inference with a staged sampling strategy (structural inpainting followed by boundary refinement), completing the global geometry while preserving the visible regions of the input priors. In practice, Points-to-3D can take either accurate point-cloud priors or VGGT-estimated point clouds from single images as input. Experiments on both objects and scene scenarios consistently demonstrate superior performance over state-of-the-art baselines in terms of rendering quality and geometric fidelity, highlighting the effectiveness of explicitly embedding point-cloud priors for achieving more accurate and structurally controllable 3D generation. Project page: https://jiatongxia.github.io/points2-3D/
Jiatong Xia, Zicheng Duan, Anton van den Hengel +1
Australian Institute for Machine Learning, University of Adelaide, Australia