cs.CVFeb 27, 2026

BuildAnyPoint: 3D Building Structured Abstraction from Diverse Point Clouds

Authors: Tongyan Hua, Haoran Gong, Yuan Liu, Di Wang, Ying-Cong Chen, Wufan Zhao

Organizations: HKUST(GZ) · Xi’an Jiaotong University · HKUST

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce BuildAnyPoint, a novel generative framework for structured 3D building reconstruction from point clouds with diverse distributions, such as those captured by airborne LiDAR and Structure-from-Motion. To recover artist-created building abstraction in this highly underconstrained setting, we capitalize on the role of explicit 3D generative priors in autoregressive mesh generation. Specifically, we design a Loosely Cascaded Diffusion Transformer (Loca-DiT) that initially recovers the underlying distribution from noisy or sparse points, followed by autoregressively encapsulating them into compact meshes. We first formulate distribution recovery as a conditional generation task by training latent diffusion models conditioned on input point clouds, and then tailor a decoder-only transformer for conditional autoregressive mesh generation based on the recovered point clouds. Our method delivers substantial qualitative and quantitative improvements over prior building abstraction methods. Furthermore, the effectiveness of our approach is evidenced by the strong performance of its recovered point clouds on building point cloud completion benchmarks, which exhibit improved surface accuracy and distribution uniformity.

Figures & tables

Figure 2: (a) Point2Building [ 29 ] aggressively compresses point clouds into a low-resolution feature grid to compensate for missing regions in the raw inputs. (b) ArcPro [ 23 ] is constrained by a fixed architectural grammar that can only generate vertically extruded geometric primitives, limiting flexibility in representing common structures such as slanted roofs. (c) BuildAnyPoint probabilistically models the diverse building distributions within a high-resolution latent grid.
Figure 3: Overview of BuildAnyPoint, implemented using our generative framework Loca-DiT, which retrieves building abstraction from the input in two sequential steps via latent space transformations: (a) The hierarchical latent diffusion model θ\theta generates an intermediate representation Pout\mathcal{P}_{out} on condition of the input point cloud Pin\mathcal{P}_{in} , where the finer level of the latent representation Gs\mathcal{G}_{s} is conditioned on the coarser one Gd\mathcal{G}_{d} . (b) Pout\mathcal{P}{out} is then tokenized into TP\mathcal{T}{P} to condition a decoder-only transformer ϕ\phi , which autoregressively generates the mesh token sequence TM\mathcal{T}_{M} . The final artist-created mesh M\mathcal{M} is obtained by applying the Mesh Detokenization step MD\mathcal{MD} to TM\mathcal{T}_{M} .
Figure 4: Overview of the training process for each Loca-DiT module. Our generative framework can be summarized as maintaining a set of latent spaces, with each tailored generative model learning how to sample and form a consistent feature space.
Figure 5: 3D Building Structured Abstraction Comparison. Qualitative comparison on three common urban point cloud distributions against City3D [ 22 ] and Point2Building (abbreviated as P2B.) [ 29 ] . Our generative framework achieves more complete and faithful structural recovery than the alternatives, a result attributed to its robust intermediate dense points (abbreviated as Inter.) reconstructed from the 3D generative prior, which ensures consistency across heterogeneous input scenarios.
Figure 6: 3D Building Point Cloud Completion Comparison. Our method is compared qualitatively against several established baselines known for their strong performance on building point cloud completion [ 16 ] . Our generative 3D priors faithfully capture intricate geometries and thin structures, and consistently produce more complete and uniform results.
Methods# V ↓\downarrow# F ↓\downarrow# P ↓\downarrowFR ↓\downarrowCD ↓\downarrow
City3D [ 22 ]17372146 %0.167
Point2Building [ 29 ]2034181 %0.043
Ours101680 %0.036
Table 1: Quantitative comparison on structured mesh abstraction. Our method achieves the best performance across all metrics, calculated as the average of three scenarios. Refer to the Supplementary Material for the full set of experimental records.
MethodsF-score ↑\uparrowCD ↓\downarrowUniformity ↓\downarrowEMD ↓\downarrow
FoldingNet [ 51 ]0.411.472.430.29
PCN [ 55 ]0.860.740.690.15
TopNet [ 40 ]0.250.970.610.35
GRNet [ 49 ]0.741.411.130.19
SnowFlakeNet [ 48 ]0.601.381.520.17
PoinTr [ 52 ]0.850.410.250.12
Table 2: Quantitative comparison on point cloud completion. Our method achieves the best performance across all metrics, demonstrating superior completeness (F-score), geometric accuracy (CD, EMD), and point distribution uniformity, measured for LiDAR point cloud scenarios.
Figure 7: Ablation on Cascaded Designs in Loca-DiT: Diffusion. When critical modules are removed (top row, left and middle), the intermediate point clouds deviate significantly from the target surface (mesh). Our full model (top row, right) corrects this, producing a point distribution that faithfully captures the target geometry. The bottom row shows the final meshes generated autoregressively from these point clouds.
Figure 8: Ablation on Cascaded Designs in Loca-DiT: Transformer. While provided with the same intermediate point cloud, the traditional solver [ 22 ] still cannot generate a valid surface, unlike our autoregressive transformer.
Settings# V ↓\downarrow# F ↓\downarrow# P ↓\downarrowFR ↓\downarrowCD ↓\downarrow
w/o priors7812753<1 %0.107
Ours3870230 %0.034
Table 3: Ablation on 3D generative priors in autoregressive Transformer. We test MeshAnything V2 on both the Pin\mathcal{P}_{in} (w/o priors) and Pout\mathcal{P}_{out} (ours).
 LiDAR Point CloudSfM Point CloudSparse Sample Point Cloud
Methods# V ↓\downarrow# F ↓\downarrow# P ↓\downarrowFR ↓\downarrowCD ↓\downarrow# V ↓\downarrow# F ↓\downarrow# P ↓\downarrowFR ↓\downarrowCD ↓\downarrow# V ↓\downarrow# F ↓\downarrow# P ↓\downarrowFR ↓\downarrowCD ↓\downarrow
City3D [ 22 ]11342141 %0.06422193162 %0.118180781316 %0.342
Point2Building [ 29 ]2034181 %0.0532136182 %0.0531933161 %0.044
Ours (w/o fine-tune)4277280 %0.0413768230 %0.0293769240 %0.034
Table 4: Quantitative comparison on structured mesh abstraction for all scenarios without fine-tuning. Although the absence of a fine-tuning procedure leads to slightly over-tessellated outputs, our method still achieves the best performance in precision-related metrics.
Figure 9: More Results. Our method consistently produces high-quality outputs, regardless of the input density.
More in this topic

3D Mesh Generation

3D mesh generation creates and edits connected 3D surfaces for digital content, reconstruction, animation, and simulation.

See all 83 papers

Explore similar work

CardsList
Mar 19, 2026

Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors

Recent progress in 3D generation has been driven largely by models conditioned on images or text, while readily available 3D priors are still underused. In many real-world scenarios, the visible-region point cloud are easy to obtain from active sensors such as LiDAR or from feed-forward predictors like VGGT, offering explicit geometric constraints that current methods fail to exploit. In this work, we introduce Points-to-3D, a diffusion-based framework that leverages point cloud priors for geometry-controllable 3D asset and scene generation. Built on a latent 3D diffusion model TRELLIS, Points-to-3D first replaces pure-noise sparse structure latent initialization with a point cloud priors tailored input formulation.A structure inpainting network, trained within the TRELLIS framework on task-specific data designed to learn global structural inpainting, is then used for inference with a staged sampling strategy (structural inpainting followed by boundary refinement), completing the global geometry while preserving the visible regions of the input priors. In practice, Points-to-3D can take either accurate point-cloud priors or VGGT-estimated point clouds from single images as input. Experiments on both objects and scene scenarios consistently demonstrate superior performance over state-of-the-art baselines in terms of rendering quality and geometric fidelity, highlighting the effectiveness of explicitly embedding point-cloud priors for achieving more accurate and structurally controllable 3D generation. Project page: https://jiatongxia.github.io/points2-3D/

Jiatong Xia, Zicheng Duan, Anton van den Hengel +1

Australian Institute for Machine Learning, University of Adelaide, Australia

3D Asset Generation3D Shape Generation3D Scene Generation3D Diffusion Models
AskPDF
Sep 20, 2026

ProxyBuild: Text-Guided Structured 3D Building Generation with Mesh-Anchored Procedural Proxies

Text-guided 3D building generation holds tremendous application potential, yet existing generative models typically output inseparable single meshes or non-interactive rendered representations. While procedural modeling can generate editable buildings with hierarchical structures, rule authoring is laborious, and even with the aid of large language models (LLMs), it remains challenging to effectively solve procedural rules under geometric constraints. In this paper, we propose ProxyBuild, a novel hybrid framework for structured building generation. We introduce the Mesh-Anchored Procedural Proxy (MAPP) as a novel intermediate representation, which tightly anchors building components onto geometric shells, thereby decoupling the generation task into two phases: proxy prediction and proxy-to-asset instantiation. First, we construct a building dataset with MAPP annotations to train our designed face-edge bigraph encoder. By explicitly modeling the feature interactions of topological elements on heterogeneous mesh graphs, this encoder accurately infers the semantic roles of faces and edges. Subsequently, conditioned on textual styles and attribute parameters parsed by LLMs, we accomplish high-precision asset retrieval and assembly by integrating a spatial placement logic with hard constraints. Extensive experiments show that ProxyBuild not only significantly mitigates common issues in building generation such as over-smoothing, component collisions, and structural corruptions, but also accurately parses semantic-free shells from diverse sources. Outperforming prior baselines across various metrics, our method can robustly generate structurally clear, detail-rich, and post-editable 3D buildings from text, thereby providing a reliable and interactive content foundation for downstream applications such as virtual reality and digital twins.

Xiang Tang, Ruotong Li, Xiaopeng Fan

Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · Harbin Institute of Technology +1

3D Asset Generation3D Scene GenerationProcedural Content GenerationText-to-3D Generation
AskPDF
Aug 8, 2026

SUMI: Scalable Unified Model for 3D Point Cloud Inference

Point cloud completion commonly follows a coarse-to-fine paradigm, where a low-density coarse shape is first predicted and then upsampled to the target resolution. Although recent methods have improved global structure recovery, the fine stage often remains limited by simple upsampling and insufficient interaction with coarse structural features, making local detail reconstruction challenging. We propose SUMI, a diffusion-enhanced refinement module for coarse-to-fine point cloud completion. Unlike prior diffusion-based completion methods that use diffusion as a standalone point generator, SUMI injects noisy geometric features into cross-attention with coarse structural features, enabling reverse denoising to refine local geometry while preserving global consistency. SUMI can also be integrated into existing coarse-to-fine models as a flexible refinement module. Experiments on PCN, ShapeNet-55/34, and MVP demonstrate consistent improvements over strong baselines. SUMI achieves the best overall CD and F1-score on PCN, reduces CD by up to 16.1% on ShapeNet-55, and obtains the best CD across all output densities on MVP.

Yanlong LI, Kanchana Thilakarathna

Point Cloud Completion3D Shape Completion3D Diffusion Models
AskPDF