Abstracting structured buildings from limited observations is essential for 3D urban modeling and digital twins. Partial point clouds and overhead images of the same building provide incomplete 3D surface measurements and visual cues to its footprint, roof geometry, and rooftop structures, respectively. This motivates a unified framework that aligns geometric and visual cues in 3D and reconstructs buildings from either or both modalities. Building on this connection, we present Any2Building, a unified model for Level of Detail 2 (LoD2) building reconstruction from partial point clouds, overhead images, or both, with optional coarse geometric priors. At its core is an occupancy-grounded shared spatial queries representation that organizes available observations within a common 3D grid. Supervised by complete building surface voxels, these queries connect observational evidence to target geometry, enabling a shared generator to learn structural priors across tasks and providing spatial guidance for recovering missing structure. A trained voxel conditional surface-generation and low-poly remeshing pipeline then converts the predicted structure into compact LoD2 building models. To support this framework, we introduce BuildingWorld-Image, comprising approximately one million paired samples of partial point clouds, overhead images, and LoD2 building models across 18 cities. Experiments on multiple benchmarks demonstrate that Any2Building outperforms existing methods in geometric reconstruction accuracy across unimodal and multimodal point-cloud completion, single-image building generation, and prior-assisted generation, while maintaining detailed and robust recovery when point cloud input modality is degraded.
Figures & tables
Figure 2: Any2Building overview. (a) Available observations and optional priors guide building generation through shared spatial queries, followed by surface generation and low-poly remeshing. (b) Queries on a coarse 163 voxel grid aggregate observation features and optional priors. Voxel occupancy supervision encourages an intermediate representation of overall building structure, which conditions the DiT through QE tokens and spatial prompts.
Figure 3: Spatial prompt injection. (a) Zero-initialized projections inject aligned residuals. QE tokens condition every block. (b) A/B show RMS residual deviations from different spatial prompt block’s surface mean. A−B shows their difference.
Figure 4: Real imagery with textured building ground truth in BuildingWorld-Image. Remote-sensing observations, source textured meshes, and instance-colored geometry.
Figure 5: Completion and low-poly examples. Partial inputs, completed point sets, and compact geometric outputs are shown alongside the reference. These selected examples illustrate how point completion quality affects the recovered architectural form.
Point cloud (PC)
Low-poly mesh (LP)
Method
CD ↓
F-score (%) ↑
EMD ↓
Uni. ↓
CD ↓
F-score (%) ↑
rV(eV↓)
rF(eF↓)
City3D [ 15 ]
-
-
-
-
1.49
45.43
1.62 (0.81)
1.47 (0.63)
Point2Building [ 22 ]
-
-
-
-
1.20
52.92
1.29 (0.38)
1.26 (0.41)
FoldingNet [ 37 ]
1.47
41.27
0.29
2.43
1.89
35.14
1.42 (0.61)
1.58 (0.74)
PCN [ 40 ]
0.74
86.14
0.15
0.69
1.08
79.32
1.28 (0.39)
1.36 (0.47)
TopNet [ 27 ]
0.97
25.43
0.35
0.61
1.60
19.28
1.35 (0.48)
1.41 (0.52)
Table 1: Point-cloud completion and low-poly reconstruction on Building-PCC. rV and rF are mean per-instance predicted-to-reference vertex and face count ratios. The parenthesized values eV and eF are their mean absolute deviations from 1.
Figure 6: One model, four observation settings. Any2Building reconstructs buildings from (a) partial points, (b) images with partial points, (c) a single overhead image, and (d) images with a rough prior. The examples highlight roof articulation, coherent building massing, and the use of complementary evidence. All compared models that support training (with fire logo) were trained under the corresponding task-specific experimental settings.
Point cloud (PC)
Low-poly mesh (LP)
Method
Input
CDL1↓
F-score (%) ↑
CDL1↓
CDL2↓
F-score (%) ↑
EMD ↓
Uni. ↓
DCD ↓
(a) Unimodal point-cloud completion
PoinTr [ 38 ]
P
11.24
91.00
12.68
0.79
88.00
92.42
0.34
0.52
AnchorFormer [ 3 ]
P
12.35
88.00
13.87
1.04
85.00
140.98
0.60
0.55
SnowflakeNet [ 33 ]
P
10.42
93.00
11.69
0.78
91.00
80.68
0.08
0.47
AdaPoinTr [ 39 ]
P
9.81
94.00
11.05
0.62
92.00
90.57
0.14
0.48
Table 2: Unimodal and multimodal point-cloud completion on BuildingWorld-Image. P denotes partial points and I denotes an overhead image. Group (b) uses sparser partial observations than group (a). PC outputs precede surface generation and low-poly reconstruction. LP denotes final low-poly meshes.
Geometry
Appearance
Method
Input
CDL1↓
CDL2↓
F-score (%) ↑
EMD ↓
Uni. ↓
DCD ↓
PSNR (dB) ↑
SSIM ↑
(a) Single-image generation
TRELLIS [ 31 ]
I
54.89
9.14
49.20
111.08
0.03
0.66
10.91
0.26
TRELLIS.2 [ 32 ]
I
65.09
9.90
31.05
124.57
0.03
0.79
10.80
0.28
SAM 3D [ 1 ]
I
55.82
8.53
44.34
117.59
0.04
0.71
9.26
0.23
Ours
I
38.90
6.32
62.82
93.16
0.03
0.60
12.63
0.34
Table 3: Image-based building generation on BuildingWorld-Image. I denotes an overhead image and R denotes a coarse geometric prior. Backgrounds are retained in (a) and removed in (b). PSNR and SSIM compare matched four-view renderings of predicted and reference textured meshes.
(a) Core components: LP F@2% ↑
Variant
P
I
P+I
I+R
Spatial-query conditioning
Condition concat
91.86
53.34
60.71
57.32
w/o occupancy supervision
94.75
54.83
71.08
64.03
w/o query read-back
93.78
51.17
72.44
70.38
w/o spatial prompts
92.15
51.47
69.13
57.02
Table 4: Ablation of the unified generation pipeline. (a) Conditioning components and joint versus single-task training. (b) Paired P and P+I inference under point degradation. (c) Surface generation in the point-cloud-input setting. Both surface variants use predicted voxel coordinates, and one additionally receives point-cloud conditioning. F-scores are reported as percentages.
Figure 7: From spatial queries to compact geometry. (a) Surface generation reduces voxel staircase artifacts before low-poly remeshing. (b) Query and decoder occupancy alongside target surface voxels. (c) Predictions from single-task and jointly trained models. Quantitative component comparisons are reported in Table 4 (a).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
City
Geometry
Image
Boston
167,410
63,857
Calgary
322,649
224,190
Cambridge
17,041
11,191
Cape Town
274,598
125,202
Edmonton
370,673
246,540
Greater Geelong
886
596
Appendix
Table S1 : BuildingWorld data pool before partitioning. Geometry counts unique city-qualified low-poly targets. Image counts targets with a paired overhead image after quality filtering, without tier quotas. Image is a subset of Geometry; counts exclude augmentation multiplicity. These are pre-partition counts, not final training-set sizes.
Figure S1 : BuildingWorld-Image across 15 cities. Each city panel pairs a geographic image window with source building geometry and shows a selected instance crop with its corresponding mesh. The examples illustrate geographic and architectural diversity; they are not restricted to the training subset. Road traces under rendered meshes provide schematic visual context and are not surveyed road annotations. Building geometry is unchanged.
Figure S2 : Preserving building structure during image alignment. Six cities, with initial contours and boxes (red), registered candidates (green), candidate cutouts, and source-mesh top views. The shared image window preserves visual scale across the first three columns. Source meshes use a strict top-down view to check structure, not an estimated satellite camera. Local refinement preserves polygon structure; rows can retain an already registered input when no additional update is accepted. These curation examples are separate from the observations used to train the evaluated checkpoints; they do not provide independent mask annotations.
Figure S3 : Textured ground truth from neighborhood to instance. Each group shows the remote-sensing context and a selected building, with corresponding source textured mesh and instance-colored geometry. Geometry, camera, and framing are shared between the textured and colored mesh views. Source textures are retained; instance colors are display attributes, and road context is schematic. These are source references, not model predictions.
PolyFit
RoLoPM
Method
Time (s) ↓
CD ↓
F-score ↑
Time (s) ↓
CD ↓
F-score ↑
FoldingNet [ 37 ]
121.29
48.02
0.42
239.08
51.76
0.31
PCN [ 40 ]
116.92
55.86
0.33
219.13
56.53
0.38
TopNet [ 27 ] ‡
217.59
87.90
0.09
382.77
64.25
0.21
GRNet [ 34 ]
125.69
70.88
0.33
215.34
54.77
0.30
SnowflakeNet [ 33 ]
62.57
35.98
0.67
186.02
11.69
0.91
Appendix
Table S2 : Low-poly reconstruction. Arithmetic means on the full 2,000-building BuildingWorld-Image test set, with the TopNet CD exception specified below. CD denotes CDL1×103 . F-scores are reported as fractions at a normalized distance threshold of 0.02, to four decimal places for Any2Building and two for the baselines. Time sums recorded inference and conversion times.
Variant
PC F-score ↑
PC CD ↓
Uniformity ↓
EMD ↓
LP F-score ↑
LP CD ↓
TRELLIS.2 VAE encoder
0.89
0.38
0.08
0.18
0.86
0.48
Pointwise MLP
0.91
0.33
0.07
0.15
0.88
0.44
Point Transformer
0.93
0.27
0.05
0.12
0.90
0.38
EdgeConv w/o PE
0.95
0.22
0.04
0.09
0.93
0.33
EdgeConv (Ours)
0.96
0.20
0.03
0.07
0.94
0.30
Appendix
Table S3 : Point-cloud encoder ablation on Building-PCC. Metrics follow the Building-PCC comparison in the main text (PC: point cloud; LP: low-poly reconstruction). F-scores are reported as fractions rounded to two decimal places. Each variant replaces the point-cloud conditioning encoder or removes its additive positional embedding (PE). The spatial-query module, generative backbone, output decoder, and downstream reconstruction stages remain unchanged.
Query grid
P
I
P+I
I+R
83
0.7703
0.2213
0.7671
0.2681
163
0.7723
0.2272
0.7676
0.2987
Appendix
Table S4 : Query-resolution comparison on the full 2,000-building BuildingWorld-Image test set. Voxel F1 measures surface-support overlap; higher is better.
Method
Geometry
Appearance
IoU ↑
CD ↓
F-score ↑
PSNR ↑
LPIPS ↓
CLIP ↑
(a) LoD0 priors
SpaceControl [ 7 ]
0.6318
0.0402
0.5924
16.2318
0.2714
0.6253
Hunyuan3D-Omni [ 28 ]
0.8705
0.0268
0.7920
17.4162
0.2187
0.7468
SatSkylines [ 18 ]
0.8982
0.0224
0.8236
17.9248
0.2015
0.7601
Ours
0.9307
0.0172
0.8702
19.5238
0.1624
0.7889
Appendix
Table S5 : Prior-assisted building generation with LoD0 and LoD1 priors. Results follow the 500-asset evaluation setting and are compared separately within each prior level. F-scores are reported as fractions. PSNR is reported in dB.
Figure S4 : Geometric priors reduce image-to-3D ambiguity. Selected image-only and image-plus-prior reconstructions, each shown with its reference geometry. Priors help anchor building extent and massing while the generator recovers architectural detail. These examples illustrate the additional-input setting rather than image-only performance.
Figure S5 : Spatial queries provide an interpretable geometric intermediate. From left to right: decoder support without queries, query anchors with occupancy support, occupied query cells, decoder support with queries, and reference support. The query lattice is 163 , with occupied cells visualized at probability at least 0.5. Each row preserves a common view. Quantitative component comparisons are reported in Table 4 (a).
Figure S6 : Downstream textured building visualization. Each group shows the input image, untextured mesh (White Mesh), reference appearance, and final textured output. The examples illustrate the appearance of the textured buildings alongside their geometric structure.
We introduce BuildAnyPoint, a novel generative framework for structured 3D building reconstruction from point clouds with diverse distributions, such as those captured by airborne LiDAR and Structure-from-Motion. To recover artist-created building abstraction in this highly underconstrained setting, we capitalize on the role of explicit 3D generative priors in autoregressive mesh generation. Specifically, we design a Loosely Cascaded Diffusion Transformer (Loca-DiT) that initially recovers the underlying distribution from noisy or sparse points, followed by autoregressively encapsulating them into compact meshes. We first formulate distribution recovery as a conditional generation task by training latent diffusion models conditioned on input point clouds, and then tailor a decoder-only transformer for conditional autoregressive mesh generation based on the recovered point clouds. Our method delivers substantial qualitative and quantitative improvements over prior building abstraction methods. Furthermore, the effectiveness of our approach is evidenced by the strong performance of its recovered point clouds on building point cloud completion benchmarks, which exhibit improved surface accuracy and distribution uniformity.
Automated LOD3 building modeling typically relies on purpose-built geometric or learning-based pipelines, limiting flexibility across heterogeneous buildings and input evidence conditions. This study investigates whether Astra, a general-purpose multimodal foundation model, can address these limitations through zero-shot reconstruction of LOD3 building models within an agentic framework under bounded autonomy. AstraLOD3 combines multi-view images, calibrated cameras, and a filtered sparse SfM point cloud with a natural-language reconstruction specification, while the Astra agent dynamically selects and executes computational procedures using Python and Blender. Across 35 runs, including 24 benchmark buildings, AstraLOD3 achieved a mean FRDS of 0.9647 and geometric agreement comparable to that of previous purpose-built methods. Controlled ablations further revealed the effects of reconstruction guidance, evidence modalities, model configuration, and run-to-run variability. The results demonstrate that structured LOD3 reconstruction can be formulated as a constrained agentic process rather than as a fixed pipeline. Future work will investigate adaptive refinement, user-guided correction, task-specific specialization, and damage-aware reconstruction.
Bryan G. Pantoja-Rosero
Digital Intelligence for Structures and Construction (DISC) Laboratory, School of Civil and Environmental Engineering, Nanyang Technological University, Singapore
We present the P3 dataset, a large-scale multimodal benchmark for building vectorization, constructed from aerial LiDAR point clouds, high-resolution aerial imagery, and vectorized 2D building outlines, collected across three continents. The dataset contains over 10 billion LiDAR points with decimeter-level accuracy and RGB images at a ground sampling distance of 25 centimeter. While many existing datasets primarily focus on the image modality, P3 offers a complementary perspective by also incorporating dense 3D information. We demonstrate that LiDAR point clouds serve as a robust modality for predicting building polygons, both in hybrid and end-to-end learning frameworks. Moreover, fusing aerial LiDAR and imagery further improves accuracy and geometric quality of predicted polygons. The P3 dataset is publicly available, along with code and pretrained weights of three state-of-the-art models for building polygon prediction at https://github.com/raphaelsulzer/PixelsPointsPolygons .
Raphael Sulzer, Liuyun Duan, Nicolas Girard +1
Université Côte d’Azur, INRIA, Sophia-Antipolis, France · LuxCarta Technology, Mouans-Sartoux, France