Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors

Jiatong Xia; Zicheng Duan; Anton van den Hengel; Lingqiao Liu

cs.CVMar 19, 2026

Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors

Authors: Jiatong Xia, Zicheng Duan, Anton van den Hengel, Lingqiao Liu

Organizations: Australian Institute for Machine Learning, University of Adelaide, Australia

Abstract

Recent progress in 3D generation has been driven largely by models conditioned on images or text, while readily available 3D priors are still underused. In many real-world scenarios, the visible-region point cloud are easy to obtain from active sensors such as LiDAR or from feed-forward predictors like VGGT, offering explicit geometric constraints that current methods fail to exploit. In this work, we introduce Points-to-3D, a diffusion-based framework that leverages point cloud priors for geometry-controllable 3D asset and scene generation. Built on a latent 3D diffusion model TRELLIS, Points-to-3D first replaces pure-noise sparse structure latent initialization with a point cloud priors tailored input formulation.A structure inpainting network, trained within the TRELLIS framework on task-specific data designed to learn global structural inpainting, is then used for inference with a staged sampling strategy (structural inpainting followed by boundary refinement), completing the global geometry while preserving the visible regions of the input priors. In practice, Points-to-3D can take either accurate point-cloud priors or VGGT-estimated point clouds from single images as input. Experiments on both objects and scene scenarios consistently demonstrate superior performance over state-of-the-art baselines in terms of rendering quality and geometric fidelity, highlighting the effectiveness of explicitly embedding point-cloud priors for achieving more accurate and structurally controllable 3D generation. Project page: https://jiatongxia.github.io/points2-3D/

Figures & tables

Figure 2: Overall framework. Given point cloud priors—either pre-existing or predicted by VGGT from input image—we first voxelize and VAE-encode it to obtain an SS latent, where the empty regions are filled with random noise and concatenated with an extracted mask to form the input paradigm for our model. During training, the input training data is fed into our inpainting flow transformer $\mathcal{G}_{inp}$ , which is optimized via a conditional flow matching loss. During inference, the input test data is processed by the trained $\mathcal{G}_{inp}$ through a two-stage sampling procedure: (1) a structural inpainting stage with $s$ sampling steps to inpaint the global structure. And (2) a boundary refinement stage with remaining $(t-s)$ steps to refine the inpainting boundaries, yielding the final output SS latent.

Figure 3: Training data processing. We preserve the visible portion of the complete point cloud and convert it into training inputs.

	Rendering				Geometry
Method	$\text{PSNR}\uparrow$	$\text{SSIM}(\%)\uparrow$	$\text{LPIPS}\downarrow$	$\text{DINO}(\%)\downarrow$	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
GaussianAnything [ 33 ]	20.08	89.31	0.183	26.74	0.084	0.513	20.99	0.199
Real3D [ 24 ]	19.55	90.65	0.169	27.65	0.065	0.574	21.31	0.178
LGM [ 61 ]	20.55	89.98	0.181	23.45	0.075	0.487	20.04	0.202
VoxHammer [ 36 ] (3D Inversion)	20.51	90.01	0.123	15.10	0.046	0.724	20.28	0.158
TRELLIS [ 72 ]	21.94	91.46	0.105	7.82	0.034	0.832	23.81	0.105

Table 1: Comparison on single-object generation on Toy4K dataset. We showcase the performance of our method in two scenarios: one where explicit point cloud priors are provided, and another where point cloud are inferred from condition images using VGGT [ 66 ] .

Figure 4: Single-object generation on Toys4K. For the explicit point cloud priors results, we use point cloud extracted strictly from the visible region of input images, whereas the “VGGT-estimated” results use point clouds inferred from the condition images by VGGT.

	Rendering				Geometry
Method	$\text{PSNR}\uparrow$	$\text{SSIM}(\%)\uparrow$	$\text{LPIPS}\downarrow$	$\text{DINO}(\%)\downarrow$	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
TRELLIS [ 72 ]	18.21	83.12	0.239	12.33	0.094	0.478	18.76	0.258
VoxHammer [ 36 ] (3D Inversion)	19.29	84.70	0.179	18.41	0.051	0.686	20.43	0.181
SceneGen [ 42 ]	18.32	83.35	0.231	14.43	0.086	0.485	19.08	0.229
MIDI [ 22 ]	19.23	85.59	0.166	14.25	0.075	0.513	20.82	0.164
Points-to-3D (Ours-VGGT Esti.)	20.52	86.51	0.152	8.90	0.040	0.743	20.97	0.160

Table 2: Comparison on scene-level generation on 3D-FRONT dataset. Points-to-3D consistently outperforms state-of-the-art multi-object generation methods across all evaluation metrics.

Figure 5: Scene-level generation on 3D-FRONT. The input point cloud priors setting is the same as in Fig. 4 .

Methods	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
TRELLIS [ 72 ] -O.	0.034	0.832	23.81	0.105
TRELLIS [ 72 ] -V.	0.032	0.854	24.77	0.093
SAM3D [ 8 ] -O.	0.033	0.835	23.85	0.101
SAM3D [ 8 ] -V.	0.031	0.841	24.81	0.090
Points-to-3D -O.	0.013	0.964	27.10	0.053
Points-to-3D -V.	0.007	0.998	29.00	0.036

Table 3: Comparison on visible and overall geometry results on Toys4K. We present the comparison between our method and TRELLIS. For each method, the upper row (O.) shows the overall results, while the lower row (V.) shows the visible region results.

Inp.	Ref.	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
50	0	0.014	0.960	25.88	0.065
40	10	0.013	0.962	26.49	0.059
30	20	0.013	0.963	26.89	0.056
25	25	0.013	0.963	27.10	0.053
20	30	0.013	0.962	27.03	0.055
10	40	0.014	0.961	26.72	0.061

Table 4: Ablation study. We evaluate the number of inpainting steps (Inp.) and refinement steps (Ref.) in our sampling strategy.

Figure 6: Ablation study. Allocating the full sampling to inpainting (Inp.) results in geometric “holes” along the inpainting edge.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Generation results with 3 input views on Toys4K. The first column of our results uses sampled point-cloud priors extracted from the visible regions of the three input images, whereas the “VGGT-estimated” results rely on point clouds inferred from the input images by VGGT.

Figure 9: Input point cloud priors examples. We show the observable point cloud priors examples for the two input modes with single-view input in this paper, along with their corresponding generation results.

Figure 10: More image-to-3D examples. More single-image to 3D generation visualization results on Toy4K (row 1-3) and 3D-Front dataset (row 4-6).

	Rendering				Geometry
Method	$\text{PSNR}\uparrow$	$\text{SSIM}(\%)\uparrow$	$\text{LPIPS}\downarrow$	$\text{DINO}(\%)\downarrow$	$~\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
TRELLIS [ 72 ]	23.19	92.63	0.075	5.79	0.025	0.904	26.22	0.066
Points-to-3D (Ours-VGGT Esti.)	23.44	93.21	0.057	5.58	0.015	0.971	28.35	0.035
Points-to-3D (Ours-P.C.Priors)	23.98	94.02	0.050	5.26	0.009	0.988	30.45	0.028

Appendix

Table 5: Comparison on single-object generation with 3 views input on Toy4K dataset.

Figure 11: Example of cluttered table with depth sensor.

Figure 12: More real-world image generation examples.

Figure 13: Text-to-3D generation examples.

Methods	$\text{CLIP}\uparrow$	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
LGM [ 61 ]	0.247	0.086	0.412	19.55	0.223
TRELLIS [ 72 ]	0.298	0.047	0.639	21.25	0.159
Points-to-3D	0.299	0.022	0.892	24.75	0.094

Appendix

Table 6: Comparison of text-to-3D generation on Toys4K.

Methods	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
P.C. priors	0.013	0.964	27.10	0.053
+ 5% perturbation	0.022	0.910	24.32	0.082
+ 10% perturbation	0.031	0.817	22.87	0.109
+ 15% perturbation	0.045	0.667	20.36	0.181
+ 10% perturbation & repair	0.027	0.855	23.47	0.098
+ 15% perturbation & repair	0.036	0.791	21.85	0.132

Appendix

Table 7: Noisy point cloud priors. We add different levels of perturbation to the accurate point-cloud priors to evaluate the impact of noisy 3D priors, and present the results of our simple repair process for noisy point cloud inputs below.

3D Asset Generation

3D asset generation creates and edits objects, characters, materials, and scenes from text, images, video, or geometric constraints.

See all 167 papers

Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models
Mar 24, 2026Wenyue Chen, Wenjue Chen, Peng Li +63D Asset Generation 3D Reconstruction
Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University, Shenzhen, China · Math Magic, Beijing, China · Peking University, Beijing, China +2
Extend3D: Town-Scale 3D Generation
Mar 31, 2026Seungwoo Yoon, Jinmo Kim, Jaesik ParkImage-to-3D Scene Generation 3D Scene Generation
Seoul National University
Structured 3D Latents Are Surprisingly Powerful: Unleashing Generalizable Style with 2D Diffusion
May 6, 2026Yiran Qiao, Yiren Lu, Yunlai Zhou +53D Asset Generation
Case Western Reserve University

Mar 24, 2026J/K move · Enter open · S save

Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models

Wenyue Chen, Wenjue Chen, Peng Li, Qinghe Wang+5

Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University, Shenzhen, China · Math Magic, Beijing, China · Peking University, Beijing, China+2

3D Asset Generation 3D Reconstruction Single-View 3D Reconstruction 3D Diffusion Models

Open paper AskPDF arXiv

Recent advances in 3D generation have improved the fidelity and geometric details of synthesized 3D assets. However, due to the inherent ambiguity of single-view observations and the lack of robust global structural priors caused by limited 3D training data, the unseen regions generated by existing models are often stochastic and difficult to control, which may sometimes fail to align with user intentions or produce implausible geometries. In this paper, we propose Know3D, a novel framework that incorporates rich knowledge from multimodal large language models into 3D generative processes via latent hidden-state injection, enabling language-controllable generation of the back-view for 3D assets. We utilize a VLM-diffusion-based model, where the VLM is responsible for semantic understanding and guidance. The diffusion model acts as a bridge that transfers semantic knowledge from the VLM to the 3D generation model. In this way, we successfully bridge the gap between abstract textual instructions and the geometric reconstruction of unobserved regions, transforming the traditionally stochastic back-view hallucination into a semantically controllable process, demonstrating a promising direction for future 3D generation models.

Abstract

3D Asset Generation

Explore similar work

Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models

Extend3D: Town-Scale 3D Generation

Structured 3D Latents Are Surprisingly Powerful: Unleashing Generalizable Style with 2D Diffusion