Organizations: Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University, Shenzhen, China · Math Magic, Beijing, China · Peking University, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong, China · Dalian University of Technology, Dalian, China
Recent advances in 3D generation have improved the fidelity and geometric details of synthesized 3D assets. However, due to the inherent ambiguity of single-view observations and the lack of robust global structural priors caused by limited 3D training data, the unseen regions generated by existing models are often stochastic and difficult to control, which may sometimes fail to align with user intentions or produce implausible geometries. In this paper, we propose Know3D, a novel framework that incorporates rich knowledge from multimodal large language models into 3D generative processes via latent hidden-state injection, enabling language-controllable generation of the back-view for 3D assets. We utilize a VLM-diffusion-based model, where the VLM is responsible for semantic understanding and guidance. The diffusion model acts as a bridge that transfers semantic knowledge from the VLM to the 3D generation model. In this way, we successfully bridge the gap between abstract textual instructions and the geometric reconstruction of unobserved regions, transforming the traditionally stochastic back-view hallucination into a semantically controllable process, demonstrating a promising direction for future 3D generation models.
Figures & tables
Figure 1: We propose Know3D, a knowledge-guided 3D generation framework that enables semantic control over the back-view of target objects. As shown in the coffee-cup scene, by simply varying the text, Know3D can synthesize diverse and structurally consistent back-view components. The bottom-row results demonstrate the potential of our method to improve the structural plausibility of unseen parts in 3D generation.
Figure 2: Text annotation pipeline. Given paired front and back views of a 3D asset, a VLM generates initial part-level descriptions.
Figure 3: Overview . Given an input image and a text prompt, Know3D leverages Qwen-Image-Edit-2511 [ 56 ] , which integrates Qwen2.5-VL with a Diffusion Transformer (DiT). Qwen2.5-VL [ 2 ] encodes the multimodal inputs to guide a latent denoising process, enabling semantically aligned backside generation. The latent hidden states from intermediate DiT layers, which embed rich semantic and structural priors, are subsequently injected as conditioning signals into the 3D generation model to guide complete shape synthesis.
Figure 4: Back-View Semantic Control Visualization. The leftmost column shows the input front-view condition images. The remaining columns present back-view normal renderings of meshes generated under different text prompts, demonstrating flexible semantic control over back-side components.
Figure 5: Visual Comparison between Know3D and Hunyuan3D-2mv. As shown in (a) and (b), both Know3D and Hunyuan3D-2mv [ 48 ] achieve reasonably good alignment between the input and generated viewpoints. However, for novel viewpoints, Hunyuan3D-2mv produces implausible structures(d), whereas Know3D still yields reasonable results(e).
Figure 6: Visual Comparison between Know3D and Current SOTA Baselines. By leveraging semantic knowledge, our method taps into the intrinsic understanding of object attributes within pre-trained multimodal models, showing potential for improving the structural plausibility of unseen parts in 3D generation.
HY3D-Bench-test
HY3D-Bench-val
Model
ULIP ↑
Uni3D ↑
ULIP ↑
Uni3D ↑
Direct3D-S2 [ 57 ]
0.2034
0.3292
0.2040
0.3303
Hi3DGen [ 64 ]
0.2082
0.3206
0.2030
0.3213
Step1X-3D [ 28 ]
0.2068
0.3233
0.2033
0.3265
TRELLIS [ 60 ]
0.2143
0.3421
0.2133
0.3464
TRELLIS.2 [ 59 ]
0.1948
0.3308
0.1967
0.3358
Table 1: Quantitative comparison on HY3D-Bench [ 21 ] . The best results are in bold and the second-best are underlined. ↑ indicates higher is better.
Figure 7: Visualization of Different Feature Representations to Prompt 3D Generation. When Qwen-Image generates an unreasonable back-view, using different feature representations to prompt 3D generation will lead to different results. The hidden states from the intermediate layers of MMDiT during the denoising process still generate reasonable results, while the VAE fails to inject the back-view information, making it quite similar to the unreasonable results of the original TRELLIS.2. Additionally, when the generated image is directly passed through DINOv3, DINOv3 tends to fit the erroneous results.
School of Artificial and Automation, Huazhong University of Science and Technology · School of Computing, National University of Singapore · School of Computing, Macquarie University