We introduce BuildAnyPoint, a novel generative framework for structured 3D building reconstruction from point clouds with diverse distributions, such as those captured by airborne LiDAR and Structure-from-Motion. To recover artist-created building abstraction in this highly underconstrained setting, we capitalize on the role of explicit 3D generative priors in autoregressive mesh generation. Specifically, we design a Loosely Cascaded Diffusion Transformer (Loca-DiT) that initially recovers the underlying distribution from noisy or sparse points, followed by autoregressively encapsulating them into compact meshes. We first formulate distribution recovery as a conditional generation task by training latent diffusion models conditioned on input point clouds, and then tailor a decoder-only transformer for conditional autoregressive mesh generation based on the recovered point clouds. Our method delivers substantial qualitative and quantitative improvements over prior building abstraction methods. Furthermore, the effectiveness of our approach is evidenced by the strong performance of its recovered point clouds on building point cloud completion benchmarks, which exhibit improved surface accuracy and distribution uniformity.
Figures & tables
Figure 2: (a) Point2Building [ 29 ] aggressively compresses point clouds into a low-resolution feature grid to compensate for missing regions in the raw inputs. (b) ArcPro [ 23 ] is constrained by a fixed architectural grammar that can only generate vertically extruded geometric primitives, limiting flexibility in representing common structures such as slanted roofs. (c) BuildAnyPoint probabilistically models the diverse building distributions within a high-resolution latent grid.
Figure 3: Overview of BuildAnyPoint, implemented using our generative framework Loca-DiT, which retrieves building abstraction from the input in two sequential steps via latent space transformations: (a) The hierarchical latent diffusion model θ generates an intermediate representation Pout on condition of the input point cloud Pin , where the finer level of the latent representation Gs is conditioned on the coarser one Gd . (b) Pout is then tokenized into TP to condition a decoder-only transformer ϕ , which autoregressively generates the mesh token sequence TM . The final artist-created mesh M is obtained by applying the Mesh Detokenization step MD to TM .
Figure 4: Overview of the training process for each Loca-DiT module. Our generative framework can be summarized as maintaining a set of latent spaces, with each tailored generative model learning how to sample and form a consistent feature space.
Figure 5: 3D Building Structured Abstraction Comparison. Qualitative comparison on three common urban point cloud distributions against City3D [ 22 ] and Point2Building (abbreviated as P2B.) [ 29 ] . Our generative framework achieves more complete and faithful structural recovery than the alternatives, a result attributed to its robust intermediate dense points (abbreviated as Inter.) reconstructed from the 3D generative prior, which ensures consistency across heterogeneous input scenarios.
Figure 6: 3D Building Point Cloud Completion Comparison. Our method is compared qualitatively against several established baselines known for their strong performance on building point cloud completion [ 16 ] . Our generative 3D priors faithfully capture intricate geometries and thin structures, and consistently produce more complete and uniform results.
Methods
# V ↓
# F ↓
# P ↓
FR ↓
CD ↓
City3D [ 22 ]
173
72
14
6 %
0.167
Point2Building [ 29 ]
20
34
18
1 %
0.043
Ours
10
16
8
0 %
0.036
Table 1: Quantitative comparison on structured mesh abstraction. Our method achieves the best performance across all metrics, calculated as the average of three scenarios. Refer to the Supplementary Material for the full set of experimental records.
Methods
F-score ↑
CD ↓
Uniformity ↓
EMD ↓
FoldingNet [ 51 ]
0.41
1.47
2.43
0.29
PCN [ 55 ]
0.86
0.74
0.69
0.15
TopNet [ 40 ]
0.25
0.97
0.61
0.35
GRNet [ 49 ]
0.74
1.41
1.13
0.19
SnowFlakeNet [ 48 ]
0.60
1.38
1.52
0.17
PoinTr [ 52 ]
0.85
0.41
0.25
0.12
Table 2: Quantitative comparison on point cloud completion. Our method achieves the best performance across all metrics, demonstrating superior completeness (F-score), geometric accuracy (CD, EMD), and point distribution uniformity, measured for LiDAR point cloud scenarios.
Figure 7: Ablation on Cascaded Designs in Loca-DiT: Diffusion. When critical modules are removed (top row, left and middle), the intermediate point clouds deviate significantly from the target surface (mesh). Our full model (top row, right) corrects this, producing a point distribution that faithfully captures the target geometry. The bottom row shows the final meshes generated autoregressively from these point clouds.
Figure 8: Ablation on Cascaded Designs in Loca-DiT: Transformer. While provided with the same intermediate point cloud, the traditional solver [ 22 ] still cannot generate a valid surface, unlike our autoregressive transformer.
Settings
# V ↓
# F ↓
# P ↓
FR ↓
CD ↓
w/o priors
78
127
53
<1 %
0.107
Ours
38
70
23
0 %
0.034
Table 3: Ablation on 3D generative priors in autoregressive Transformer. We test MeshAnything V2 on both the Pin (w/o priors) and Pout (ours).
LiDAR Point Cloud
SfM Point Cloud
Sparse Sample Point Cloud
Methods
# V ↓
# F ↓
# P ↓
FR ↓
CD ↓
# V ↓
# F ↓
# P ↓
FR ↓
CD ↓
# V ↓
# F ↓
# P ↓
FR ↓
CD ↓
City3D [ 22 ]
113
42
14
1 %
0.064
221
93
16
2 %
0.118
180
78
13
16 %
0.342
Point2Building [ 29 ]
20
34
18
1 %
0.053
21
36
18
2 %
0.053
19
33
16
1 %
0.044
Ours (w/o fine-tune)
42
77
28
0 %
0.041
37
68
23
0 %
0.029
37
69
24
0 %
0.034
Table 4: Quantitative comparison on structured mesh abstraction for all scenarios without fine-tuning. Although the absence of a fine-tuning procedure leads to slightly over-tessellated outputs, our method still achieves the best performance in precision-related metrics.
Figure 9: More Results. Our method consistently produces high-quality outputs, regardless of the input density.