We introduce PixARMesh, a method to autoregressively reconstruct complete 3D indoor scene meshes directly from a single RGB image. Unlike prior methods that rely on implicit signed distance fields and post-hoc layout optimization, PixARMesh jointly predicts object layout and geometry within a unified model, producing coherent and artist-ready meshes in a single forward pass. Building on recent advances in mesh generative models, we augment a point-cloud encoder with pixel-aligned image features and global scene context via cross-attention, enabling accurate spatial reasoning from a single image. Scenes are generated autoregressively from a unified token stream containing context, pose, and mesh, yielding compact meshes with high-fidelity geometry. Experiments on synthetic and real-world datasets show that PixARMesh achieves state-of-the-art reconstruction quality while producing lightweight, high-quality meshes ready for downstream applications.
Figures & tables
Figure 1: Comparison of PixARMesh with recent compositional scene reconstruction methods. PixARMesh predicts object poses and reconstructs native meshes in a single autoregressive decoding process, without relying on SDF-based surface extraction or layout optimization, producing compact and artist-ready mesh outputs.
Figure 2: Pipeline overview. Given an RGB image, we use pretrained models to extract the depth point cloud and image features for both the target object i and the global scene. These local and global cues are fed into the Pixel-Aligned PC-Encoder to produce the fused latent code, which is then aggregated into a single latent vector via cross-attention. This latent vector conditions the Transformer Decoder, which predicts the object’s pose followed by its mesh token sequence.
Method
Scene-level
Object-level
CD ( ×10−3 , ↓ )
CD-S ( ×10−3 , ↓ )
F-Score ( % , ↑ )
CD ( ×10−3 , ↓ )
F-Score ( % , ↑ )
SDF-based
InstPIFu [ 27 ]
213.4
124.9
13.72
44.74
29.63
Uni-3D [ 47 ]
218.3
113.3
12.99
—
—
Gen3DSR [ 13 ]
222.4
137.5
13.52
9.74
31.42
DeepPriorAssembly [ 51 ]
191.8
76.2
16.72
20.13
27.83
Table 1: Qualitative comparison with state-of-the-art methods on the 3D-FRONT [ 14 ] dataset. Following DepR [ 49 ] and DeepPriorAssembly [ 51 ] , we report object- and scene-level Chamfer Distance (CD; lower is better) and F-Score (higher is better). We additionally include the single-direction Chamfer Distance (CD-S) to account for missing instances.
Figure 3: Qualitative comparisons on the 3D-FRONT [ 14 ] dataset. For PixARMesh, we also show the mesh wireframe to highlight geometric quality.
Figure 4: Qualitative results on real images from Pix3D [ 40 ] , Matterport3D [ 3 ] , and ScanNet [ 10 ] datasets.
Method
Scene-level
Object-level
CD
CD-S
F-Score
CD
F-Score
( ×10−3 , ↓ )
( ×10−3 , ↓ )
( % , ↑ )
( ×10−3 , ↓ )
( % , ↑ )
Two-stage
99.8
50.6
33.32
4.75
80.85
EdgeRunner-FT
119.8
48.0
27.81
4.75
80.57
PixARMesh (Ours)
98.8
49.1
33.55
4.04
82.27
Table 2: Ablation study on joint pose-mesh modeling.
Img Feat
Ctx Agg
Scene-level
Object-level
CD
CD-S
F-Score
CD
F-Score
( ×10−3 , ↓ )
( ×10−3 , ↓ )
( % , ↑ )
( ×10−3 , ↓ )
( % , ↑ )
57.78
23.75
41.02
5.29
77.22
✓
55.44
27.16
42.84
5.56
78.14
✓
39.30
16.12
44.67
3.64
84.64
✓
✓
39.88
18.52
46.15
4.04
82.27
Table 3: Ablation studies on our point-cloud encoder design. Img Feat , Ctx Agg denote pixel-aligned image features and scene context aggregation, respectively.
GT Depth
GT Layout
CD
F-Score
( ×10−3 , ↓ )
( % , ↑ )
4.13
81.64
✓
3.04
86.66
✓
4.04
82.27
✓
✓
2.93
87.19
Table 4: Effects of depth and layout in object-level metrics.
GT Inputs
CD
CD-S
F-Score
Depth
Segm
Layout
( ×10−3 , ↓ )
( ×10−3 , ↓ )
( % , ↑ )
98.84
49.08
33.55
✓
95.69
48.18
39.04
✓
39.88
18.52
46.15
✓
✓
16.30
4.40
63.91
✓
✓
25.07
10.05
59.63
Table 5: Effects of upstream (depth, segmentation, and layout) errors in scene-level metrics. Note that ground-truth layout implies ground-truth segmentation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Additional qualitative results on real images from Pix3D [ 40 ] , Matterport3D [ 3 ] and ScanNet [ 10 ] .
Method
∣F∣
∣V∣
InstPIFu [ 27 ]
1,942,029
970,951
Uni-3D [ 47 ]
141,130
70,844
BUOL [ 8 ]
55,493
27,750
Gen3DSR [ 13 ]
364,285
216,797
DeepPriorAssembly [ 51 ]
250,875
125,420
MIDI [ 19 ]
1,936,248
967,886
Appendix
Table A.1 : Face count ∣F∣ and vertex count ∣V∣ comparisons.
Method
GT Depth
Box IoU ( % )
MIDI [ 19 ]
−
31.07
DepR [ 49 ]
33.98
DepR [ 49 ]
✓
36.67
PixARMesh-EdgeRunner (Ours)
56.76
PixARMesh-EdgeRunner (Ours)
✓
70.37
PixARMesh-BPT (Ours)
54.54
Appendix
Table A.2 : Layout accuracy comparisons.
Method
Runtime
InstPIFu [ 27 ]
19.8s
Uni-3D [ 47 ]
3.1s
BUOL [ 8 ]
6.5s
Gen3DSR [ 13 ]
15.5min
DeepPriorAssembly [ 51 ]
6.7min
MIDI [ 19 ]
1.6min
Appendix
Table A.3 : Inference runtime comparisons.
Img Feat
Ctx Agg
Scene-level
Object-level
CD
CD-S
F-Score
CD
F-Score
( ×10−3 , ↓ )
( ×10−3 , ↓ )
( % , ↑ )
( ×10−3 , ↓ )
( % , ↑ )
68.36
25.43
35.96
5.34
76.54
✓
56.80
23.49
37.98
5.85
76.31
✓
57.03
23.70
41.51
4.25
81.25
✓
✓
49.82
22.06
42.18
4.57
80.30
Appendix
Table A.4 : Ablation studies on our point-cloud encoder design with BPT-based model. Img Feat , Ctx Agg denote pixel-aligned image features and scene context aggregation, respectively.
Reconstructing complete, high-quality meshes of large-scale scenes from unposed multi-view images requires faithful geometric grounding while plausibly completing surfaces that are unobserved or only sparsely captured. We present WorldRecon, a generative multi-view reconstruction framework for recovering complete, large-scale 3D scene meshes from unposed RGB images. WorldRecon uses a coarse, noisy global point cloud and camera poses estimated from the input images by a reconstruction model as a global scaffold for whole-scene generation and partitions it into overlapping spatial blocks. To robustly leverage these imperfect geometric priors, we equip the native 3D generative model TRELLIS.2 with a dedicated reconstruction-based conditioning architecture that generates each block from its local point-cloud crop and block-relevant image observations, producing sparse structure, detailed geometry, and PBR materials. To maintain scene-level coherence, we introduce neighbour-aware latent inpainting and a causal world-generation procedure that propagates structural, geometric, and material context across overlapping blocks. Together, these designs scale multi-view generative reconstruction to large-scale scenes, yielding coherent, unified meshes. Extensive experiments demonstrate that WorldRecon achieves state-of-the-art reconstruction performance, reducing Chamfer distance by 14.9% and 43.1% while improving F-score by 17.3% and 12.0% over the strongest baselines on synthetic and real-world indoor scenes, respectively, thereby demonstrating faithful and complete generative reconstruction as well as strong sim-to-real generalization. Project page: https://kszpxxzmc.github.io/WorldRecon-website/.
Mengchen Zhang, Yang Li, Chunshi Wang +2
Zhejiang University · Tencent Hunyuan · Fudan University
Single-image 3D scene reconstruction must complete partially observed objects and place them coherently in a shared observation-aligned scene frame. Object-level generative priors offer strong completion ability, but their centered, scale-normalized outputs are typically expressed in an object frame, creating a fundamental representation gap between object generation and scene reconstruction. We introduce SceneReGen, a generative reconstruction framework that reinterprets scene reconstruction as the generation and assembly of complete object assets in a shared observation-aligned scene frame. SceneReGen addresses the generation-reconstruction gap through selective pose factorization: each object's observed orientation is encoded directly in the generated mesh, while translation and scale are estimated from instance-level and global scene evidence. Given a scene image and instance masks, a geometry encoder extracts dense cues; learnable shape queries condition a pretrained DiT-based 3D generator to produce complete meshes in their observed orientations, while position queries fuse object and scene features to assemble them in the shared frame. On the 3D-FUTURE evaluation subset, SceneReGen achieves the best scene-level CD, scene-level F-Score, and 3D bounding-box IoU among the evaluated methods, ties the best object-level CD, and ranks second in object-level F-Score. Qualitative outputs in autonomous-driving and embodied-AI scenes further illustrate the potential of asset-centric reconstruction beyond indoor furniture.
We introduce a new approach to high-fidelity 3D scene reconstruction from multi-view RGB images that tightly couples reconstruction with a strong generative 3D prior. We cast scene reconstruction as conditional 3D generation over a set of spatially-localized, overlapping chunks that together tile the scene, scaling generation to large scene extents. Crucially, we inherit the fidelity and completeness of state-of-the-art generative shape models -- we use Trellis.2 as an example -- which we generalize to the scene level. To this end, we propose a projection-based conditioning mechanism that lifts posed multi-view image features into a coherent 3D representation aligned with the generative model, independent of view ordering and spatially anchored to the scene, yielding high-fidelity, multi-view consistent generated geometry. This enables lifting the strong object-level prior of Trellis.2 to multi-view, scene-scale generation, producing faithful, editable PBR mesh reconstructions of indoor environments. As a result, we obtain high-fidelity results that outperform cutting-edge reconstruction methods by 16%.
Katharina Schmid, Nicolas von Lützow, Jozef Hladký +2
Technical University of Munich · Computing Systems Lab, Huawei Technologies, Switzerland