Learning 3D Representations for Spatial Intelligence from Unposed Multi-View Images

Bo Zhou; Qiuxia Lai; Zeren Sun; Xiangbo Shu; Yazhou Yao; Wenguan Wang

cs.CVApr 12, 2026

Learning 3D Representations for Spatial Intelligence from Unposed Multi-View Images

Authors: Bo Zhou, Qiuxia Lai, Zeren Sun, Xiangbo Shu, Yazhou Yao, Wenguan Wang

Organizations: Nanjing University of Science and Technology · Communication University of China · Zhejiang University

Abstract

Robust 3D representation learning forms the perceptual foundation of spatial intelligence, enabling downstream tasks in scene understanding and embodied AI. However, learning such representations directly from unposed multi-view images remains challenging. Recent self-supervised methods attempt to unify geometry, appearance, and semantics in a feed-forward manner, but they often suffer from weak geometry induction, limited appearance detail, and inconsistencies between geometry and semantics. We introduce UniSplat, a feed-forward framework designed to address these limitations through three complementary components. First, we propose a dual-masking strategy that strengthens geometry induction in the encoder. By masking both encoder and decoder tokens, and targeting decoder masks toward geometry-rich regions, the model is forced to infer structural information from incomplete visual cues, yielding geometry-aware representations even under unposed inputs. Second, we develop a coarse-to-fine Gaussian splatting strategy that reduces appearance-semantics inconsistencies by progressively refining the radiance field. Finally, to enforce geometric-semantic consistency, we introduce a pose-conditioned recalibration mechanism that interrelates the outputs of multiple heads by re-projecting predicted 3D point and semantic maps into the image plane using estimated camera parameters, and aligning them with corresponding RGB and semantic predictions to ensure cross-task consistency, thereby resolving geometry-semantic mismatches. Together, these components yield unified 3D representations that are robust to unposed, sparse-view inputs and generalize across diverse tasks, laying a perceptual foundation for spatial intelligence.

Figures & tables

Figure 1: Comparison of three key 3D representation learning paradigms: supervised geometry reconstruction, self-supervised neural reconstruction, and our novel feed-forward framework.

Figure 2: Overview of the proposed UniSplat framework. UniSplat integrates a dual-masking strategy for geometry induction (§ 3.2 ), a coarse-to-fine Gaussian splatting strategy for appearance refinement (§ 3.3 ), and a pose-conditioned recalibration mechanism for geometry–semantic consistency (§ 3.4 ).

Figure 3: Pose-conditioned recalibration mechanism (§ 3.4 ).

			Source View				Target View
	Recon. Time $\downarrow$		OVSS		DE		OVSS		NVS
Methods	SfM	Per-Scene	mIoU $\uparrow$	Acc. $\uparrow$	rel $\downarrow$	$\tau\uparrow$	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
LSeg [ 34 ]	N/A	N/A	0.4701	0.7891	-	-	0.4819	0.7927	-	-	-
NeRF-DFF [ 31 ]	20.52s	1min	0.4540	0.7173	27.68	9.61	0.4037	0.6755	19.86	0.6650	0.3629
Feature-3DGS [ 87 ]	20.52s	18mins	0.4453	0.7276	12.95	21.07	0.4223	0.7174	24.49	0.8132	0.2293
pixelSplat [ 6 ]	20.52s	0.064s	-	-	-	-	-	-	24.89	0.8392	0.1641

Table 1: Quantitative comparison on ScanNet [ 14 ] . (§ 4.2 ) We evaluate performance on novel view synthesis (NVS), depth estimation (DE), and open-vocabulary semantic segmentation (OVSS).

Methods					Vision-Centric		Multi-Modal			Embodied-Specific
Benchmarks		MAE	DINOv2	CLIP	EVA	InternViT	MVP	VC-1	SPA	Ours
VC-1	AD (2)	$58.0_{\pm 2.0}$	$47.3_{\pm 3.1}$	$48.7_{\pm 3.1}$	$58.0_{\pm 6.0}$	$53.3_{\pm 3.1}$	$53.3_{\pm 4.2}$	$54.0_{\pm 4.0}$	$60.0_{\pm 4.0}$	$\mathbf{{61.7_{\pm 4.3}}}$
	MW (5)	$90.0_{\pm 4.6}$	$84.0_{\pm 3.7}$	$77.1_{\pm 3.2}$	$90.7_{\pm 0.9}$	$84.0_{\pm 3.7}$	$93.6_{\pm 5.2}$	$87.5_{\pm 3.8}$	$93.3_{\pm 2.0}$	$\mathbf{94.3_{\pm 3.1}}$
	DMC (5)	$74.4_{\pm 1.8}$	$64.5_{\pm 2.5}$	$53.9_{\pm 3.6}$	$62.7_{\pm 2.8}$	$53.3_{\pm 0.4}$	$69.4_{\pm 2.6}$	$65.3_{\pm 3.6}$	$71.1_{\pm 5.0}$	$\mathbf{75.8_{\pm 4.5}}$
	TF (2)	$73.0_{\pm 0.5}$	$68.5_{\pm 0.4}$	$56.1_{\pm 1.6}$	$67.2_{\pm 0.2}$	$65.2_{\pm 1.6}$	$73.2_{\pm 0.8}$	$70.9_{\pm 1.1}$	$73.6_{\pm 2.0}$	$\mathbf{75.6_{\pm 1.7}}$
RLBench	Group 1 (35)	78.3	78.2	76.8	75.2	74.1	76.2	80.1	80.5	81.2

Table 2: Quantitative comparison on various embodied AI tasks. (§ 4.3 ) The number in parentheses denotes the number of tasks.

	RealEstate10K			ACID
Methods	5 ∘ $\uparrow$	10 ∘ $\uparrow$	20 ∘ $\uparrow$	5 ∘ $\uparrow$	10 ∘ $\uparrow$	20 ∘ $\uparrow$
DUSt3R	0.329	0.537	0.691	0.113	0.273	0.469
MASt3R	0.351	0.557	0.701	0.159	0.362	0.524
NoPoSplat	0.568	0.737	0.839	0.342	0.504	0.653
SelfSplat	0.223	0.413	0.589	0.213	0.372	0.541
Ours	0.607	0.748	0.842	0.354	0.516	0.661

Table 3: Quantitative comparison of relative pose estimation on RealEstate10K [ 88 ] and ACID [ 38 ] . (§ 4.2 )

	ACID			DTU
Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
pixelSplat	25.477	0.770	0.207	15.067	0.539	0.341
MVSplat	25.525	0.773	0.199	14.542	0.537	0.324
NoPoSplat	25.765	0.776	0.199	17.899	0.629	0.279
SelfSplat	22.204	0.686	0.316	13.249	0.434	0.441
Ours	25.983	0.786	0.188	16.852	0.587	0.269

Table 4: Quantitative comparison of Cross-dataset Generalization on ACID [ 38 ] and DTU [ 25 ] . (§ 4.2 ) All models are trained on RealEstate10K [ 88 ] (indoor scenes) and evaluated on ACID (outdoor scenes) and DTU (object-centric scenes) datasets.

Figure 4: Qualitative comparison of novel-view segmentation on ScanNet. (§ 4.2 ) Follwing [ 16 ] , we map thousands of ScanNet labels into 8 common categories for visualization. The dense 2D GT labels are obtained by projecting sparse 3D annotations, so some regions are inevitably incomplete or missing.

Figure 5: Qualitative comparison of NVS on RE10K. (§ 4.2 )

Figure 6: Visualizations of feature and depth on ScanNet.

Variant	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	rel $\downarrow$
Full UniSplat	0.5625	0.8334	25.65	3.10
w/o Self-sup.	0.5263	0.8110	24.40	3.74
w/o Dual Mask	0.5462	0.8275	24.74	3.27
w/o Coarse-to-Fine	0.5374	0.8239	24.93	3.34
w/o $\mathcal{L}_{\text{recalib}}$	0.5287	0.8186	24.35	3.52
w/o $\mathcal{L}_{\text{sem}}$	0.0214	0.0811	24.82	3.24

Table 5: Ablation on core components. (§ 4.4 ) Each row removes one component to evaluate its contribution.

Exp ID	Datasets	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	rel $\downarrow$
(1)	ScanNet	0.5603	0.8297	25.48	3.18
(2)	(1) + ScanNet++	0.5625	0.8334	25.65	3.10
(3)	(2) + RE10K	0.5717	0.8414	25.79	3.02
(4)	(3) + DL3DV	0.5755	0.8437	25.83	2.93

Table 6: Ablation on data scale. (§ 4.4 ) We evaluate the effect of progressively adding more training data from different datasets, including ScanNet++ [ 80 ] , RealEstate10K [ 88 ] , and DL3DV [ 37 ] .

#Views	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	rel $\downarrow$
3	0.5574	0.8292	23.83	0.7683	4.12
6	0.5846	0.8451	24.47	0.8138	3.79
8	0.6027	0.8454	25.03	0.8652	3.30
10	0.6227	0.8514	25.12	0.8723	3.21

Table 7: Ablation on the number of input views. (§ 4.4 ) We fix the view range and evaluate under varying numbers of input views.

Type of $\mathcal{M}_{e}$	$\rho_{e}$	Type of $\mathcal{M}_{d}$	$\rho_{d}$	mIoU $\uparrow$	PSNR $\uparrow$	rel $\downarrow$
Random	0.50	N/A	0	0.5412	24.64	3.45
Croco	0.90	N/A	0	0.5498	25.12	3.37
Random	0.50	3D GS	0.30	0.5511	25.42	3.23
Random	0.50	3D GS	0.50	0.5625	25.65	3.10
Random	0.50	3D GS	0.75	0.5470	25.10	3.38
Random	0.75	3D GS	0.50	0.5284	24.55	3.56

Table 8: Ablation on different mask Strategies. (§ 4.4 )

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

	Small			Medium			Large			Average
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Pose-Required
pixelSplat	20.277	0.719	0.265	23.726	0.811	0.180	27.152	0.880	0.121	23.859	0.808	0.184
MVSplat	20.371	0.725	0.250	23.808	0.814	0.172	27.466	0.885	0.115	24.012	0.812	0.175
Supervised Pose-Free
MASt3R	16.305	0.516	0.451	18.106	0.561	0.377	17.975	0.524	0.402	17.617	0.539	0.403

Appendix

Table A1 : Quantitative comparison of novel view synthesis on the RE10K dataset under various overlap settings (§ B.1 ).

	PnP CabToCounter		PnP CounterToCab		PnP CounterToMicrowave		PnP CounterToSink		PnP CounterToStove		PnP MicrowaveToCounter


BC-transformer	2	18	6	28	2	18	2	44	2	6	2	8
GWM	18	32	4	22	14	44	20	38	2	18	20	26
Ours	25	39	10	34	17	51	26	47	8	22	29	36
	PnP SinkToCounter		PnP StoveToCounter		Open SingleDoor		Open DoubleDoor		Close DoubleDoor		Close SingleDoor

Appendix

Table A2 : Multi-Task Imitation Learning Results in Robocasa (§ B.2 ). Average success rates (%) of multi-task agents trained with

50

human demonstrations or

3000

generated demonstrations per task. Results are evaluated over

50

episodes.

	RE10K			ACID
Method	5 ∘ $\uparrow$	10 ∘ $\uparrow$	20 ∘ $\uparrow$	5 ∘ $\uparrow$	10 ∘ $\uparrow$	20 ∘ $\uparrow$
$\mathcal{G}_{\text{geo}}$	0.510	0.664	0.796	0.305	0.472	0.612
$\mathcal{G}_{\text{anchor}}$	0.578	0.714	0.822	0.336	0.495	0.648
$\mathcal{G}_{\text{app}}$	0.592	0.736	0.834	0.344	0.504	0.653
$C_{\text{coarse}}$	0.524	0.693	0.805	0.311	0.487	0.625
$C_{\text{final}}$	0.607	0.748	0.842	0.354	0.516	0.661

Appendix

Table A3 : Relative Pose Estimation with Gaussian-based PnP Poses and Pose Heads (§ B.3 ). We evaluate AUC with various thresholds on RealEstate10K [ 88 ] and ACID [ 38 ] datasets.

# $T_{\text{coarse}}$	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	rel $\downarrow$
64	0.5469	0.8237	25.05	0.8602	3.35
128	0.5587	0.8284	25.33	0.8684	3.23
256	0.5625	0.8334	25.65	0.8782	3.10
512	0.5617	0.8312	25.53	0.8756	3.11

Appendix

Table A4: Ablation on the Number of Gaussian Latent Tokens (§ B.4 ). We evaluate the effect of varying the number of Gaussian latent tokens on reconstruction and segmentation performance.

Method	Input	AP 25	AR 25	AP 50	AR 50
ImVoxelNet [ 56 ]	RGB	6.15	20.39	2.41	6.31
VoteNet [ 51 ]	Depth	3.20	6.11	0.38	1.22
FCAF3D [ 55 ]	Depth	9.07	44.23	4.11	20.22
EmbodiedScan [ 68 ]	RGB-D	16.85	51.07	9.77	28.21
Ours	RGB	28.69	62.24	15.34	39.57

Appendix

Table A5: Multi-view 3D object detection (§ B.5 ).

Method	Input	mIoU
OccNet [ 62 ]	RGB	8.07
SurroundOcc [ 70 ]	RGB	9.10
EmbodiedScan [ 68 ]	RGB	10.48
EmbodiedScan [ 68 ]	RGB-D	19.97
Ours	RGB	27.45

Appendix

Table A6: Multi-view Semantic Occupancy Prediction (§ B.5 ).

Method	Input	Overall	Easy	Hard
ScanRefer [ 8 ]	RGB-D	12.85	13.78	9.12
BUTD-DETR [ 23 ]	RGB-D	22.14	23.12	18.23
L3Det [ 89 ]	RGB-D	23.07	24.01	18.34
EmbodiedScan [ 68 ]	RGB-D	25.72	27.11	20.12
Ours	RGB	36.88	38.13	31.42

Appendix

Table A7: Multi-view 3D Visual Grounding (§ B.5 ).

Figure A2: More qualitative comparisons on RE10K with small image overlap (§ C.2 ).

Figure A3: More qualitative comparisons on RE10K with medium image overlap (§ C.2 ).

Figure A4: More qualitative comparisons on RE10K with large image overlap (§ C.2 ).

Figure A5: More qualitative comparisons on ACID (§ C.2 ).

Figure A6: More qualitative comparison of novel-view segmentation on ScanNet (§ C.3 ).

Figure A7: More visualizations of feature maps and depth maps (§ C.4 ).

Figure A8: More qualitative results of 3D object detection (§ C.5 ).

3D Representation Learning

3D representation learning builds geometric features for recognizing, reconstructing, generating, and reasoning about objects and scenes.

See all 191 papers

UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images
Mar 18, 2026Guibiao Liao, Qian Ren, Kaimin Liao +43D Reconstruction Sparse-View 3D Reconstruction
Moore Threads AI
FF3R: Feedforward Feature 3D Reconstruction from Unconstrained views
Apr 10, 2026Chaoyi Zhou, Run Wang, Feng Luo +43D Reconstruction Feed-Forward 3D Reconstruction
Microsoft · Clemson University · Texas A&M University
LangFlash: Feed-forward 3D Language Gaussian Splatting from Sparse Unposed Images
May 22, 2026Yilong Liu, Wanhua Li, Chen Zhu-Tian +13D Gaussian Splatting Language-Embedded 3DGS
Harvard University · Tsinghua University · Nanyang Technological University +1

Mar 18, 2026J/K move · Enter open · S save

UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images

Guibiao Liao, Qian Ren, Kaimin Liao, Hua Wang+3

Moore Threads AI

3D Reconstruction Sparse-View 3D Reconstruction 3D Gaussian Splatting Open-Vocabulary 3D Segmentation Multi-View Depth Estimation

Open paper AskPDF arXiv

Semantic-aware 3D reconstruction from sparse, unposed images remains challenging for feed-forward 3D Gaussian Splatting (3DGS). Existing methods often predict an over-complete set of Gaussian primitives under sparse-view supervision, leading to unstable geometry and inferior depth quality. Meanwhile, they rely solely on 2D segmenter features for semantic lifting, which provides weak 3D-level and limited generalizable supervision, resulting in incomplete 3D semantics in novel scenes. To address these issues, we propose UniSem, a unified framework that jointly improves depth accuracy and semantic generalization via two key components. First, Error-aware Gaussian Dropout (EGD) performs error-guided capacity control by suppressing redundancy-prone Gaussians using rendering error cues, producing meaningful, geometrically stable Gaussian representations for improved depth estimation. Second, we introduce a Mix-training Curriculum (MTC) that progressively blends 2D segmenter-lifted semantics with the model's own emergent 3D semantic priors, implemented with object-level prototype alignment to enhance semantic coherence and completeness. Extensive experiments on ScanNet and Replica show that UniSem achieves superior performance in depth prediction and open-vocabulary 3D segmentation across varying numbers of input views. Notably, with 16-view inputs, UniSem reduces depth Rel by 15.2% and improves open-vocabulary segmentation mAcc by 3.7% over strong baselines.

Abstract

3D Representation Learning

Explore similar work

UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images

FF3R: Feedforward Feature 3D Reconstruction from Unconstrained views

LangFlash: Feed-forward 3D Language Gaussian Splatting from Sparse Unposed Images