Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors

Jiatong Xia; Zicheng Duan; Anton van den Hengel; Lingqiao Liu

cs.CVMar 19, 2026

Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors

Authors: Jiatong Xia, Zicheng Duan, Anton van den Hengel, Lingqiao Liu

Organizations: Australian Institute for Machine Learning, University of Adelaide, Australia

Abstract

Recent progress in 3D generation has been driven largely by models conditioned on images or text, while readily available 3D priors are still underused. In many real-world scenarios, the visible-region point cloud are easy to obtain from active sensors such as LiDAR or from feed-forward predictors like VGGT, offering explicit geometric constraints that current methods fail to exploit. In this work, we introduce Points-to-3D, a diffusion-based framework that leverages point cloud priors for geometry-controllable 3D asset and scene generation. Built on a latent 3D diffusion model TRELLIS, Points-to-3D first replaces pure-noise sparse structure latent initialization with a point cloud priors tailored input formulation.A structure inpainting network, trained within the TRELLIS framework on task-specific data designed to learn global structural inpainting, is then used for inference with a staged sampling strategy (structural inpainting followed by boundary refinement), completing the global geometry while preserving the visible regions of the input priors. In practice, Points-to-3D can take either accurate point-cloud priors or VGGT-estimated point clouds from single images as input. Experiments on both objects and scene scenarios consistently demonstrate superior performance over state-of-the-art baselines in terms of rendering quality and geometric fidelity, highlighting the effectiveness of explicitly embedding point-cloud priors for achieving more accurate and structurally controllable 3D generation. Project page: https://jiatongxia.github.io/points2-3D/

Figures & tables

Figure 2: Overall framework. Given point cloud priors—either pre-existing or predicted by VGGT from input image—we first voxelize and VAE-encode it to obtain an SS latent, where the empty regions are filled with random noise and concatenated with an extracted mask to form the input paradigm for our model. During training, the input training data is fed into our inpainting flow transformer $\mathcal{G}_{inp}$ , which is optimized via a conditional flow matching loss. During inference, the input test data is processed by the trained $\mathcal{G}_{inp}$ through a two-stage sampling procedure: (1) a structural inpainting stage with $s$ sampling steps to inpaint the global structure. And (2) a boundary refinement stage with remaining $(t-s)$ steps to refine the inpainting boundaries, yielding the final output SS latent.

Figure 3: Training data processing. We preserve the visible portion of the complete point cloud and convert it into training inputs.

	Rendering				Geometry
Method	$\text{PSNR}\uparrow$	$\text{SSIM}(\%)\uparrow$	$\text{LPIPS}\downarrow$	$\text{DINO}(\%)\downarrow$	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
GaussianAnything [ 33 ]	20.08	89.31	0.183	26.74	0.084	0.513	20.99	0.199
Real3D [ 24 ]	19.55	90.65	0.169	27.65	0.065	0.574	21.31	0.178
LGM [ 61 ]	20.55	89.98	0.181	23.45	0.075	0.487	20.04	0.202
VoxHammer [ 36 ] (3D Inversion)	20.51	90.01	0.123	15.10	0.046	0.724	20.28	0.158
TRELLIS [ 72 ]	21.94	91.46	0.105	7.82	0.034	0.832	23.81	0.105

Table 1: Comparison on single-object generation on Toy4K dataset. We showcase the performance of our method in two scenarios: one where explicit point cloud priors are provided, and another where point cloud are inferred from condition images using VGGT [ 66 ] .

Figure 4: Single-object generation on Toys4K. For the explicit point cloud priors results, we use point cloud extracted strictly from the visible region of input images, whereas the “VGGT-estimated” results use point clouds inferred from the condition images by VGGT.

	Rendering				Geometry
Method	$\text{PSNR}\uparrow$	$\text{SSIM}(\%)\uparrow$	$\text{LPIPS}\downarrow$	$\text{DINO}(\%)\downarrow$	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
TRELLIS [ 72 ]	18.21	83.12	0.239	12.33	0.094	0.478	18.76	0.258
VoxHammer [ 36 ] (3D Inversion)	19.29	84.70	0.179	18.41	0.051	0.686	20.43	0.181
SceneGen [ 42 ]	18.32	83.35	0.231	14.43	0.086	0.485	19.08	0.229
MIDI [ 22 ]	19.23	85.59	0.166	14.25	0.075	0.513	20.82	0.164
Points-to-3D (Ours-VGGT Esti.)	20.52	86.51	0.152	8.90	0.040	0.743	20.97	0.160

Table 2: Comparison on scene-level generation on 3D-FRONT dataset. Points-to-3D consistently outperforms state-of-the-art multi-object generation methods across all evaluation metrics.

Figure 5: Scene-level generation on 3D-FRONT. The input point cloud priors setting is the same as in Fig. 4 .

Methods	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
TRELLIS [ 72 ] -O.	0.034	0.832	23.81	0.105
TRELLIS [ 72 ] -V.	0.032	0.854	24.77	0.093
SAM3D [ 8 ] -O.	0.033	0.835	23.85	0.101
SAM3D [ 8 ] -V.	0.031	0.841	24.81	0.090
Points-to-3D -O.	0.013	0.964	27.10	0.053
Points-to-3D -V.	0.007	0.998	29.00	0.036

Table 3: Comparison on visible and overall geometry results on Toys4K. We present the comparison between our method and TRELLIS. For each method, the upper row (O.) shows the overall results, while the lower row (V.) shows the visible region results.

Inp.	Ref.	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
50	0	0.014	0.960	25.88	0.065
40	10	0.013	0.962	26.49	0.059
30	20	0.013	0.963	26.89	0.056
25	25	0.013	0.963	27.10	0.053
20	30	0.013	0.962	27.03	0.055
10	40	0.014	0.961	26.72	0.061

Table 4: Ablation study. We evaluate the number of inpainting steps (Inp.) and refinement steps (Ref.) in our sampling strategy.

Figure 6: Ablation study. Allocating the full sampling to inpainting (Inp.) results in geometric “holes” along the inpainting edge.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Generation results with 3 input views on Toys4K. The first column of our results uses sampled point-cloud priors extracted from the visible regions of the three input images, whereas the “VGGT-estimated” results rely on point clouds inferred from the input images by VGGT.

Figure 9: Input point cloud priors examples. We show the observable point cloud priors examples for the two input modes with single-view input in this paper, along with their corresponding generation results.

Figure 10: More image-to-3D examples. More single-image to 3D generation visualization results on Toy4K (row 1-3) and 3D-Front dataset (row 4-6).

	Rendering				Geometry
Method	$\text{PSNR}\uparrow$	$\text{SSIM}(\%)\uparrow$	$\text{LPIPS}\downarrow$	$\text{DINO}(\%)\downarrow$	$~\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
TRELLIS [ 72 ]	23.19	92.63	0.075	5.79	0.025	0.904	26.22	0.066
Points-to-3D (Ours-VGGT Esti.)	23.44	93.21	0.057	5.58	0.015	0.971	28.35	0.035
Points-to-3D (Ours-P.C.Priors)	23.98	94.02	0.050	5.26	0.009	0.988	30.45	0.028

Appendix

Table 5: Comparison on single-object generation with 3 views input on Toy4K dataset.

Figure 11: Example of cluttered table with depth sensor.

Figure 12: More real-world image generation examples.

Figure 13: Text-to-3D generation examples.

Methods	$\text{CLIP}\uparrow$	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
LGM [ 61 ]	0.247	0.086	0.412	19.55	0.223
TRELLIS [ 72 ]	0.298	0.047	0.639	21.25	0.159
Points-to-3D	0.299	0.022	0.892	24.75	0.094

Appendix

Table 6: Comparison of text-to-3D generation on Toys4K.

Methods	$\text{CD}\downarrow$	$\text{F-Score}\uparrow$	$\text{PSNR-N}\uparrow$	$\text{LPIPS-N}\downarrow$
P.C. priors	0.013	0.964	27.10	0.053
+ 5% perturbation	0.022	0.910	24.32	0.082
+ 10% perturbation	0.031	0.817	22.87	0.109
+ 15% perturbation	0.045	0.667	20.36	0.181
+ 10% perturbation & repair	0.027	0.855	23.47	0.098
+ 15% perturbation & repair	0.036	0.791	21.85	0.132

Appendix

Table 7: Noisy point cloud priors. We add different levels of perturbation to the accurate point-cloud priors to evaluate the impact of noisy 3D priors, and present the results of our simple repair process for noisy point cloud inputs below.

3D Asset Generation

3D asset generation creates and edits objects, characters, materials, and scenes from text, images, video, or geometric constraints.

See all 167 papers

Mar 24, 2026

Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models

Recent advances in 3D generation have improved the fidelity and geometric details of synthesized 3D assets. However, due to the inherent ambiguity of single-view observations and the lack of robust global structural priors caused by limited 3D training data, the unseen regions generated by existing models are often stochastic and difficult to control, which may sometimes fail to align with user intentions or produce implausible geometries. In this paper, we propose Know3D, a novel framework that incorporates rich knowledge from multimodal large language models into 3D generative processes via latent hidden-state injection, enabling language-controllable generation of the back-view for 3D assets. We utilize a VLM-diffusion-based model, where the VLM is responsible for semantic understanding and guidance. The diffusion model acts as a bridge that transfers semantic knowledge from the VLM to the 3D generation model. In this way, we successfully bridge the gap between abstract textual instructions and the geometric reconstruction of unobserved regions, transforming the traditionally stochastic back-view hallucination into a semantically controllable process, demonstrating a promising direction for future 3D generation models.

Wenyue Chen, Wenjue Chen, Peng Li +6

Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University, Shenzhen, China · Math Magic, Beijing, China · Peking University, Beijing, China +2

3D Asset Generation 3D Reconstruction Single-View 3D Reconstruction 3D Diffusion Models

Ask PDF

Mar 31, 2026

Extend3D: Town-Scale 3D Generation

In this paper, we propose Extend3D, a training-free pipeline for 3D scene generation from a single image, built upon an object-centric 3D generative model. To overcome the limitations of fixed-size latent spaces in object-centric models for representing wide scenes, we extend the latent space in the

x

and

y

directions. Then, by dividing the extended latent space into overlapping patches, we apply the object-centric 3D generative model to each patch and couple them at each time step. Since patch-wise 3D generation with image conditioning requires strict spatial alignment between image and latent patches, we initialize the scene using a point cloud prior from a monocular depth estimator and iteratively refine occluded regions through SDEdit. We discovered that treating the incompleteness of 3D structure as noise during 3D refinement enables 3D completion via a concept, which we term under-noising. Furthermore, to address the sub-optimality of object-centric models for sub-scene generation, we optimize the extended latent during denoising, ensuring that the denoising trajectories remain consistent with the sub-scene dynamics. To this end, we introduce 3D-aware optimization objectives for improved geometric structure and texture fidelity. We demonstrate that our method yields better results than prior methods, as evidenced by human preference and quantitative experiments.

Seungwoo Yoon, Jinmo Kim, Jaesik Park

Seoul National University

Image-to-3D Scene Generation 3D Scene Generation 3D Diffusion Models 3D Scene Completion

Ask PDF

May 6, 2026

Structured 3D Latents Are Surprisingly Powerful: Unleashing Generalizable Style with 2D Diffusion

3D asset generation plays a pivotal role in fields such as gaming and virtual reality, enabling the rapid synthesis of high-fidelity 3D objects from a single or multiple images. Building on this capability, enabling style-controllable generation naturally emerges as an important and desirable direction. However, existing approaches typically rely on style images that lie within or are similar to the training distribution of 3D generation models. When presented with out-of-distribution (OOD) styles, their performance degrades significantly or even fails. To address this limitation, we introduce DiLAST: 2D Diffusion-based Latent Awakening for 3D Style Transfer. Specifically, we leverage a pretrained 2D diffusion model as a teacher to provide rich and generalizable style priors. By aligning rendered views with the target style under diffusion-based guidance, our method optimizes the structured 3D latent representations for stylization. We observe that this limitation stems not from insufficient model capacity, but from the underutilization of structured 3D latents, which are inherently expressive. Despite being trained on comparatively limited data, 3D generation models can leverage 2D diffusion guidance to steer denoising toward specific directions in latent space, thereby producing diverse, OOD styles. Extensive experiments across diverse data and multiple 3D generation backbones demonstrate the effectiveness and plug-and-play nature of our approach.

Yiran Qiao, Yiren Lu, Yunlai Zhou +5

Case Western Reserve University

3D Asset Generation

Ask PDF

Abstract

3D Asset Generation

Explore similar work

Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models

Extend3D: Town-Scale 3D Generation

Structured 3D Latents Are Surprisingly Powerful: Unleashing Generalizable Style with 2D Diffusion