Learning 3D Representations for Spatial Intelligence from Unposed Multi-View Images

Bo Zhou; Qiuxia Lai; Zeren Sun; Xiangbo Shu; Yazhou Yao; Wenguan Wang

cs.CVApr 12, 2026

Learning 3D Representations for Spatial Intelligence from Unposed Multi-View Images

Authors: Bo Zhou, Qiuxia Lai, Zeren Sun, Xiangbo Shu, Yazhou Yao, Wenguan Wang

Organizations: Nanjing University of Science and Technology · Communication University of China · Zhejiang University

Abstract

Robust 3D representation learning forms the perceptual foundation of spatial intelligence, enabling downstream tasks in scene understanding and embodied AI. However, learning such representations directly from unposed multi-view images remains challenging. Recent self-supervised methods attempt to unify geometry, appearance, and semantics in a feed-forward manner, but they often suffer from weak geometry induction, limited appearance detail, and inconsistencies between geometry and semantics. We introduce UniSplat, a feed-forward framework designed to address these limitations through three complementary components. First, we propose a dual-masking strategy that strengthens geometry induction in the encoder. By masking both encoder and decoder tokens, and targeting decoder masks toward geometry-rich regions, the model is forced to infer structural information from incomplete visual cues, yielding geometry-aware representations even under unposed inputs. Second, we develop a coarse-to-fine Gaussian splatting strategy that reduces appearance-semantics inconsistencies by progressively refining the radiance field. Finally, to enforce geometric-semantic consistency, we introduce a pose-conditioned recalibration mechanism that interrelates the outputs of multiple heads by re-projecting predicted 3D point and semantic maps into the image plane using estimated camera parameters, and aligning them with corresponding RGB and semantic predictions to ensure cross-task consistency, thereby resolving geometry-semantic mismatches. Together, these components yield unified 3D representations that are robust to unposed, sparse-view inputs and generalize across diverse tasks, laying a perceptual foundation for spatial intelligence.

Figures & tables

Figure 1: Comparison of three key 3D representation learning paradigms: supervised geometry reconstruction, self-supervised neural reconstruction, and our novel feed-forward framework.

Figure 2: Overview of the proposed UniSplat framework. UniSplat integrates a dual-masking strategy for geometry induction (§ 3.2 ), a coarse-to-fine Gaussian splatting strategy for appearance refinement (§ 3.3 ), and a pose-conditioned recalibration mechanism for geometry–semantic consistency (§ 3.4 ).

Figure 3: Pose-conditioned recalibration mechanism (§ 3.4 ).

			Source View				Target View
	Recon. Time $\downarrow$		OVSS		DE		OVSS		NVS
Methods	SfM	Per-Scene	mIoU $\uparrow$	Acc. $\uparrow$	rel $\downarrow$	$\tau\uparrow$	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
LSeg [ 34 ]	N/A	N/A	0.4701	0.7891	-	-	0.4819	0.7927	-	-	-
NeRF-DFF [ 31 ]	20.52s	1min	0.4540	0.7173	27.68	9.61	0.4037	0.6755	19.86	0.6650	0.3629
Feature-3DGS [ 87 ]	20.52s	18mins	0.4453	0.7276	12.95	21.07	0.4223	0.7174	24.49	0.8132	0.2293
pixelSplat [ 6 ]	20.52s	0.064s	-	-	-	-	-	-	24.89	0.8392	0.1641

Table 1: Quantitative comparison on ScanNet [ 14 ] . (§ 4.2 ) We evaluate performance on novel view synthesis (NVS), depth estimation (DE), and open-vocabulary semantic segmentation (OVSS).

Methods					Vision-Centric		Multi-Modal			Embodied-Specific
Benchmarks		MAE	DINOv2	CLIP	EVA	InternViT	MVP	VC-1	SPA	Ours
VC-1	AD (2)	$58.0_{\pm 2.0}$	$47.3_{\pm 3.1}$	$48.7_{\pm 3.1}$	$58.0_{\pm 6.0}$	$53.3_{\pm 3.1}$	$53.3_{\pm 4.2}$	$54.0_{\pm 4.0}$	$60.0_{\pm 4.0}$	$\mathbf{{61.7_{\pm 4.3}}}$
	MW (5)	$90.0_{\pm 4.6}$	$84.0_{\pm 3.7}$	$77.1_{\pm 3.2}$	$90.7_{\pm 0.9}$	$84.0_{\pm 3.7}$	$93.6_{\pm 5.2}$	$87.5_{\pm 3.8}$	$93.3_{\pm 2.0}$	$\mathbf{94.3_{\pm 3.1}}$
	DMC (5)	$74.4_{\pm 1.8}$	$64.5_{\pm 2.5}$	$53.9_{\pm 3.6}$	$62.7_{\pm 2.8}$	$53.3_{\pm 0.4}$	$69.4_{\pm 2.6}$	$65.3_{\pm 3.6}$	$71.1_{\pm 5.0}$	$\mathbf{75.8_{\pm 4.5}}$
	TF (2)	$73.0_{\pm 0.5}$	$68.5_{\pm 0.4}$	$56.1_{\pm 1.6}$	$67.2_{\pm 0.2}$	$65.2_{\pm 1.6}$	$73.2_{\pm 0.8}$	$70.9_{\pm 1.1}$	$73.6_{\pm 2.0}$	$\mathbf{75.6_{\pm 1.7}}$
RLBench	Group 1 (35)	78.3	78.2	76.8	75.2	74.1	76.2	80.1	80.5	81.2

Table 2: Quantitative comparison on various embodied AI tasks. (§ 4.3 ) The number in parentheses denotes the number of tasks.

	RealEstate10K			ACID
Methods	5 ∘ $\uparrow$	10 ∘ $\uparrow$	20 ∘ $\uparrow$	5 ∘ $\uparrow$	10 ∘ $\uparrow$	20 ∘ $\uparrow$
DUSt3R	0.329	0.537	0.691	0.113	0.273	0.469
MASt3R	0.351	0.557	0.701	0.159	0.362	0.524
NoPoSplat	0.568	0.737	0.839	0.342	0.504	0.653
SelfSplat	0.223	0.413	0.589	0.213	0.372	0.541
Ours	0.607	0.748	0.842	0.354	0.516	0.661

Table 3: Quantitative comparison of relative pose estimation on RealEstate10K [ 88 ] and ACID [ 38 ] . (§ 4.2 )

	ACID			DTU
Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
pixelSplat	25.477	0.770	0.207	15.067	0.539	0.341
MVSplat	25.525	0.773	0.199	14.542	0.537	0.324
NoPoSplat	25.765	0.776	0.199	17.899	0.629	0.279
SelfSplat	22.204	0.686	0.316	13.249	0.434	0.441
Ours	25.983	0.786	0.188	16.852	0.587	0.269

Table 4: Quantitative comparison of Cross-dataset Generalization on ACID [ 38 ] and DTU [ 25 ] . (§ 4.2 ) All models are trained on RealEstate10K [ 88 ] (indoor scenes) and evaluated on ACID (outdoor scenes) and DTU (object-centric scenes) datasets.

Figure 4: Qualitative comparison of novel-view segmentation on ScanNet. (§ 4.2 ) Follwing [ 16 ] , we map thousands of ScanNet labels into 8 common categories for visualization. The dense 2D GT labels are obtained by projecting sparse 3D annotations, so some regions are inevitably incomplete or missing.

Figure 5: Qualitative comparison of NVS on RE10K. (§ 4.2 )

Figure 6: Visualizations of feature and depth on ScanNet.

Variant	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	rel $\downarrow$
Full UniSplat	0.5625	0.8334	25.65	3.10
w/o Self-sup.	0.5263	0.8110	24.40	3.74
w/o Dual Mask	0.5462	0.8275	24.74	3.27
w/o Coarse-to-Fine	0.5374	0.8239	24.93	3.34
w/o $\mathcal{L}_{\text{recalib}}$	0.5287	0.8186	24.35	3.52
w/o $\mathcal{L}_{\text{sem}}$	0.0214	0.0811	24.82	3.24

Table 5: Ablation on core components. (§ 4.4 ) Each row removes one component to evaluate its contribution.

Exp ID	Datasets	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	rel $\downarrow$
(1)	ScanNet	0.5603	0.8297	25.48	3.18
(2)	(1) + ScanNet++	0.5625	0.8334	25.65	3.10
(3)	(2) + RE10K	0.5717	0.8414	25.79	3.02
(4)	(3) + DL3DV	0.5755	0.8437	25.83	2.93

Table 6: Ablation on data scale. (§ 4.4 ) We evaluate the effect of progressively adding more training data from different datasets, including ScanNet++ [ 80 ] , RealEstate10K [ 88 ] , and DL3DV [ 37 ] .

#Views	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	rel $\downarrow$
3	0.5574	0.8292	23.83	0.7683	4.12
6	0.5846	0.8451	24.47	0.8138	3.79
8	0.6027	0.8454	25.03	0.8652	3.30
10	0.6227	0.8514	25.12	0.8723	3.21

Table 7: Ablation on the number of input views. (§ 4.4 ) We fix the view range and evaluate under varying numbers of input views.

Type of $\mathcal{M}_{e}$	$\rho_{e}$	Type of $\mathcal{M}_{d}$	$\rho_{d}$	mIoU $\uparrow$	PSNR $\uparrow$	rel $\downarrow$
Random	0.50	N/A	0	0.5412	24.64	3.45
Croco	0.90	N/A	0	0.5498	25.12	3.37
Random	0.50	3D GS	0.30	0.5511	25.42	3.23
Random	0.50	3D GS	0.50	0.5625	25.65	3.10
Random	0.50	3D GS	0.75	0.5470	25.10	3.38
Random	0.75	3D GS	0.50	0.5284	24.55	3.56

Table 8: Ablation on different mask Strategies. (§ 4.4 )

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

	Small			Medium			Large			Average
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Pose-Required
pixelSplat	20.277	0.719	0.265	23.726	0.811	0.180	27.152	0.880	0.121	23.859	0.808	0.184
MVSplat	20.371	0.725	0.250	23.808	0.814	0.172	27.466	0.885	0.115	24.012	0.812	0.175
Supervised Pose-Free
MASt3R	16.305	0.516	0.451	18.106	0.561	0.377	17.975	0.524	0.402	17.617	0.539	0.403

Appendix

Table A1 : Quantitative comparison of novel view synthesis on the RE10K dataset under various overlap settings (§ B.1 ).

	PnP CabToCounter		PnP CounterToCab		PnP CounterToMicrowave		PnP CounterToSink		PnP CounterToStove		PnP MicrowaveToCounter


BC-transformer	2	18	6	28	2	18	2	44	2	6	2	8
GWM	18	32	4	22	14	44	20	38	2	18	20	26
Ours	25	39	10	34	17	51	26	47	8	22	29	36
	PnP SinkToCounter		PnP StoveToCounter		Open SingleDoor		Open DoubleDoor		Close DoubleDoor		Close SingleDoor

Appendix

Table A2 : Multi-Task Imitation Learning Results in Robocasa (§ B.2 ). Average success rates (%) of multi-task agents trained with

50

human demonstrations or

3000

generated demonstrations per task. Results are evaluated over

50

episodes.

	RE10K			ACID
Method	5 ∘ $\uparrow$	10 ∘ $\uparrow$	20 ∘ $\uparrow$	5 ∘ $\uparrow$	10 ∘ $\uparrow$	20 ∘ $\uparrow$
$\mathcal{G}_{\text{geo}}$	0.510	0.664	0.796	0.305	0.472	0.612
$\mathcal{G}_{\text{anchor}}$	0.578	0.714	0.822	0.336	0.495	0.648
$\mathcal{G}_{\text{app}}$	0.592	0.736	0.834	0.344	0.504	0.653
$C_{\text{coarse}}$	0.524	0.693	0.805	0.311	0.487	0.625
$C_{\text{final}}$	0.607	0.748	0.842	0.354	0.516	0.661

Appendix

Table A3 : Relative Pose Estimation with Gaussian-based PnP Poses and Pose Heads (§ B.3 ). We evaluate AUC with various thresholds on RealEstate10K [ 88 ] and ACID [ 38 ] datasets.

# $T_{\text{coarse}}$	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	rel $\downarrow$
64	0.5469	0.8237	25.05	0.8602	3.35
128	0.5587	0.8284	25.33	0.8684	3.23
256	0.5625	0.8334	25.65	0.8782	3.10
512	0.5617	0.8312	25.53	0.8756	3.11

Appendix

Table A4: Ablation on the Number of Gaussian Latent Tokens (§ B.4 ). We evaluate the effect of varying the number of Gaussian latent tokens on reconstruction and segmentation performance.

Method	Input	AP 25	AR 25	AP 50	AR 50
ImVoxelNet [ 56 ]	RGB	6.15	20.39	2.41	6.31
VoteNet [ 51 ]	Depth	3.20	6.11	0.38	1.22
FCAF3D [ 55 ]	Depth	9.07	44.23	4.11	20.22
EmbodiedScan [ 68 ]	RGB-D	16.85	51.07	9.77	28.21
Ours	RGB	28.69	62.24	15.34	39.57

Appendix

Table A5: Multi-view 3D object detection (§ B.5 ).

Method	Input	mIoU
OccNet [ 62 ]	RGB	8.07
SurroundOcc [ 70 ]	RGB	9.10
EmbodiedScan [ 68 ]	RGB	10.48
EmbodiedScan [ 68 ]	RGB-D	19.97
Ours	RGB	27.45

Appendix

Table A6: Multi-view Semantic Occupancy Prediction (§ B.5 ).

Method	Input	Overall	Easy	Hard
ScanRefer [ 8 ]	RGB-D	12.85	13.78	9.12
BUTD-DETR [ 23 ]	RGB-D	22.14	23.12	18.23
L3Det [ 89 ]	RGB-D	23.07	24.01	18.34
EmbodiedScan [ 68 ]	RGB-D	25.72	27.11	20.12
Ours	RGB	36.88	38.13	31.42

Appendix

Table A7: Multi-view 3D Visual Grounding (§ B.5 ).

Figure A2: More qualitative comparisons on RE10K with small image overlap (§ C.2 ).

Figure A3: More qualitative comparisons on RE10K with medium image overlap (§ C.2 ).

Figure A4: More qualitative comparisons on RE10K with large image overlap (§ C.2 ).

Figure A5: More qualitative comparisons on ACID (§ C.2 ).

Figure A6: More qualitative comparison of novel-view segmentation on ScanNet (§ C.3 ).

Figure A7: More visualizations of feature maps and depth maps (§ C.4 ).

Figure A8: More qualitative results of 3D object detection (§ C.5 ).

3D Representation Learning

3D representation learning builds geometric features for recognizing, reconstructing, generating, and reasoning about objects and scenes.

See all 191 papers

Mar 18, 2026

UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images

Semantic-aware 3D reconstruction from sparse, unposed images remains challenging for feed-forward 3D Gaussian Splatting (3DGS). Existing methods often predict an over-complete set of Gaussian primitives under sparse-view supervision, leading to unstable geometry and inferior depth quality. Meanwhile, they rely solely on 2D segmenter features for semantic lifting, which provides weak 3D-level and limited generalizable supervision, resulting in incomplete 3D semantics in novel scenes. To address these issues, we propose UniSem, a unified framework that jointly improves depth accuracy and semantic generalization via two key components. First, Error-aware Gaussian Dropout (EGD) performs error-guided capacity control by suppressing redundancy-prone Gaussians using rendering error cues, producing meaningful, geometrically stable Gaussian representations for improved depth estimation. Second, we introduce a Mix-training Curriculum (MTC) that progressively blends 2D segmenter-lifted semantics with the model's own emergent 3D semantic priors, implemented with object-level prototype alignment to enhance semantic coherence and completeness. Extensive experiments on ScanNet and Replica show that UniSem achieves superior performance in depth prediction and open-vocabulary 3D segmentation across varying numbers of input views. Notably, with 16-view inputs, UniSem reduces depth Rel by 15.2% and improves open-vocabulary segmentation mAcc by 3.7% over strong baselines.

Guibiao Liao, Qian Ren, Kaimin Liao +4

Moore Threads AI

3D Reconstruction Sparse-View 3D Reconstruction 3D Gaussian Splatting Open-Vocabulary 3D Segmentation Multi-View Depth Estimation

Ask PDF

Apr 10, 2026

FF3R: Feedforward Feature 3D Reconstruction from Unconstrained views

Recent advances in vision foundation models have revolutionized geometry reconstruction and semantic understanding. Yet, most of the existing approaches treat these capabilities in isolation, leading to redundant pipelines and compounded errors. This paper introduces FF3R, a fully annotation-free feed-forward framework that unifies geometric and semantic reasoning from unconstrained multi-view image sequences. Unlike previous methods, FF3R does not require camera poses, depth maps, or semantic labels, relying solely on rendering supervision for RGB and feature maps, establishing a scalable paradigm for unified 3D reasoning. In addition, we address two critical challenges in feedforward feature reconstruction pipelines, namely global semantic inconsistency and local structural inconsistency, through two key innovations: (i) a Token-wise Fusion Module that enriches geometry tokens with semantic context via cross-attention, and (ii) a Semantic-Geometry Mutual Boosting mechanism combining geometry-guided feature warping for global consistency with semantic-aware voxelization for local coherence. Extensive experiments on ScanNet and DL3DV-10K demonstrate FF3R's superior performance in novel-view synthesis, open-vocabulary semantic segmentation, and depth estimation, with strong generalization to in-the-wild scenarios, paving the way for embodied intelligence systems that demand both spatial and semantic understanding.

Chaoyi Zhou, Run Wang, Feng Luo +4

Microsoft · Clemson University · Texas A&M University

3D Reconstruction Feed-Forward 3D Reconstruction Multi-View 3D Reconstruction Open-Vocabulary 3D Segmentation

Ask PDF

May 22, 2026

LangFlash: Feed-forward 3D Language Gaussian Splatting from Sparse Unposed Images

We present LangFlash, a feed-forward framework for 3D Language Gaussian Splatting that reconstructs 3D scenes parameterized by Gaussian primitives enriched with language-aligned semantic features from sparse unposed multi-view images. Unlike optimization-based 3D methods, LangFlash directly predicts the geometry and semantics in a single forward pass, enabling low-latency 3D reconstruction and language-consistent scene understanding. To support large-scale training, we enriched the RealEstate10k dataset with coherent and dense semantic information for 3D semantic supervision. Furthermore, we propose a sparse semantic encoding scheme that combines a global semantic dictionary with locally varying per-primitive weights, preserving high-level linguistic information, while reducing representation complexity. Experimental results show that LangFlash achieves superior novel view synthesis and semantic consistency compared with previous methods. This study establishes a new paradigm for pose-free, language-grounded 3D scene reconstruction, advancing generalizable 3D vision and multimodal scene understanding. Demo is available at https://liylo.github.io/langflash.github.io/.

Yilong Liu, Wanhua Li, Chen Zhu-Tian +1

Harvard University · Tsinghua University · Nanyang Technological University +1

3D Gaussian Splatting Language-Embedded 3DGS Sparse-View 3D Reconstruction Feed-Forward 3D Reconstruction 3D Scene Understanding

Ask PDF

Abstract

3D Representation Learning

Explore similar work

UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images

FF3R: Feedforward Feature 3D Reconstruction from Unconstrained views

LangFlash: Feed-forward 3D Language Gaussian Splatting from Sparse Unposed Images