FF3R: Feedforward Feature 3D Reconstruction from Unconstrained views

Chaoyi Zhou; Run Wang; Feng Luo; Mert D. Pesé; Zhiwen Fan; Yiqi Zhong; Siyu Huang

cs.CVApr 10, 2026

FF3R: Feedforward Feature 3D Reconstruction from Unconstrained views

Authors: Chaoyi Zhou, Run Wang, Feng Luo, Mert D. Pesé, Zhiwen Fan, Yiqi Zhong, Siyu Huang

Organizations: Microsoft · Clemson University · Texas A&M University

Abstract

Recent advances in vision foundation models have revolutionized geometry reconstruction and semantic understanding. Yet, most of the existing approaches treat these capabilities in isolation, leading to redundant pipelines and compounded errors. This paper introduces FF3R, a fully annotation-free feed-forward framework that unifies geometric and semantic reasoning from unconstrained multi-view image sequences. Unlike previous methods, FF3R does not require camera poses, depth maps, or semantic labels, relying solely on rendering supervision for RGB and feature maps, establishing a scalable paradigm for unified 3D reasoning. In addition, we address two critical challenges in feedforward feature reconstruction pipelines, namely global semantic inconsistency and local structural inconsistency, through two key innovations: (i) a Token-wise Fusion Module that enriches geometry tokens with semantic context via cross-attention, and (ii) a Semantic-Geometry Mutual Boosting mechanism combining geometry-guided feature warping for global consistency with semantic-aware voxelization for local coherence. Extensive experiments on ScanNet and DL3DV-10K demonstrate FF3R's superior performance in novel-view synthesis, open-vocabulary semantic segmentation, and depth estimation, with strong generalization to in-the-wild scenarios, paving the way for embodied intelligence systems that demand both spatial and semantic understanding.

Figures & tables

Figure 2: Architecture Overview. From unconstrained multi-view inputs, FF3R injects semantic-awareness into geometry tokens through Token-Wise Fusion, then decodes pixel-aligned features to predict feature-RGB GS, depth, and camera parameters. A Semantic–Geometry Mutual Boosting module, including Geometry-Guided Feature Warping, and Semantic-aware Voxelization, enables fully annotation-free training and yields high-quality novel view synthesis and open-vocabulary, 3D-consistent semantics.

Figure 3: Geometry-Guided Feature Warping. Before warping (middle), features show inconsistency (color shifts and boundary misalignment). After warping (right), features are spatially aligned across views and better match the ground-truth semantics, yielding crisper boundaries and fewer artifacts.

Sparse	2 Views						6 Views						16 Views
Sparse	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	Acc. $\uparrow$	Time(s) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	Acc. $\uparrow$	Time(s) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	Acc. $\uparrow$	Time(s) $\downarrow$
ScanNet [ 3 ] Dataset
LSeg [ 15 ]	–	–	–	0.543	0.810	–	–	–	–	0.551	0.804	–	–	–	–	0.540	0.800	–
Feature-3DGS [ 49 ]	17.54	0.673	0.437	0.332	0.662	18min	18.34	0.695	0.430	0.330	0.661	18min	19.12	0.713	0.419	0.349	0.648	18min
LSM [ 4 ]	14.95	0.589	0.401	0.424	0.739	0.408s	14.50	0.580	0.438	0.392	0.708	16.3s	OOM	OOM	OOM	OOM	OOM	OOM
Ours	22.70	0.787	0.285	0.486	0.754	0.884s	22.19	0.769	0.295	0.500	0.751	1.2s	22.58	0.790	0.293	0.492	0.739	2.2s

Table 1: Quantitative Results on Novel View Synthesis and Semantic Segmentation. We evaluate sparse-view NVS and segmentation on ScanNet [ 3 ] , and dense-view NVS and segmentation on DL3DV-10K [ 20 ] , reporting 3D reconstruction time, image-quality metrics, and segmentation metrics.

Figure 5: Language-based 3D Segmentation Comparison. Qualitative results across eight scenes from the ScanNet [ 3 ] dataset using varying numbers of unconstrained input views. Our FF3R produces sharper boundaries, fewer artifacts, and stronger cross-view consistency than LSM [ 4 ] , Feature-3DGS [ 49 ] , and CLIP-LSeg [ 15 ] , demonstrating effective fusion of semantic information and geometric structure into a coherent 3D feature field.

Figure 6: Novel View Synthesis Comparison. We compare results under sparse and dense view settings on the ScanNet [ 3 ] and DL3DV-10K [ 20 ] datasets using unconstrained inputs. FF3R consistently outperforms all baselines, achieving sharper details and higher visual fidelity across both sparse and dense scenarios.

Method	2 Views		6 Views
Method	Rel $\downarrow$	$\tau\uparrow$	Rel $\downarrow$	$\tau\uparrow$
LSM [ 4 ]	7.36	46.24	8.38	38.65
FF3R (Ours)	3.99	67.99	3.36	71.10

Table 2: Comparison of depth consistency under different input views. We report Absolute Relative Error (Rel

\downarrow

) and Inlier Ratio (

\tau\uparrow

) with a threshold of

1.03

Method	TW Fusion	G $\rightarrow$ S	S $\rightarrow$ G	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	Acc. $\uparrow$
Base				17.85	0.699	0.380	0.411	0.684
+ TW Fusion	✓			18.70	0.721	0.359	0.440	0.721
+ G $\rightarrow$ S	✓	✓		18.61	0.705	0.373	0.450	0.722
+ S $\rightarrow$ G	✓	✓	✓	19.12	0.718	0.357	0.449	0.725

Table 3: Ablation study of different components in FF3R.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Ours w/o distill loss	9.37	0.248	0.783
Ours w/ distill loss	17.85	0.699	0.380

Appendix

Table 4: Ablation for Geometric Distillation.

	2 Views						16 Views
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	Acc. $\uparrow$	Time(s) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	Acc. $\uparrow$	Time(s) $\downarrow$
Uni3R	15.57	0.649	0.448	0.354	0.716	0.626s	17.98	0.737	0.402	0.437	0.769	0.928s
Ours	22.70	0.787	0.285	0.486	0.754	0.884s	22.19	0.769	0.295	0.500	0.751	1.2s

Appendix

Table 5: Quantitative Results on Novel View Synthesis and Semantic Segmentation on ScanNet.

Figure 7: Qualitative Results on Novel View Synthesis and Semantic Segmentation on ScanNet.

Figure 8: Qualitative results of open-volcabulory semantic segmentation.

Figure 9: Qualitative results of novel view synthesis.

Figure 10: Qualitative results on DL3DV-10K [ 20 ] demonstrating generalization across diverse indoor and outdoor scenes.

Explore similar work

Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself
Apr 15, 2026Yuhang Dai, Xingyi YangFeed-Forward 3D Reconstruction Multi-View 3D Reconstruction
The Hong Kong Polytechnic University, Hong Kong SAR
Reliev3R: Relieving Feed-forward Reconstruction from Multi-View Geometric Annotations
Apr 1, 2026Youyu Chen, Junjun Jiang, Yueru Luo +43D Reconstruction Feed-Forward 3D Reconstruction
Harbin Institute of Technology · The Chinese University of Hong Kong, Shenzhen · Huawei
VLRC: Vision-Language Reprojection Consistency as a scalable signal for better feed-forward 3D pretraining
Jul 2, 2026Marwane Hariat, David Filliat, Antoine ManzaneraOpen-Vocabulary 3D Segmentation Feed-Forward 3D Reconstruction
U2IS, ENSTA – Institut Polytechnique de Paris · Agence Minist´erielle pour l’IA de D´efense

Apr 15, 2026J/K move · Enter open · S save

Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself

Yuhang Dai, Xingyi Yang

The Hong Kong Polytechnic University, Hong Kong SAR

Feed-Forward 3D Reconstruction Multi-View 3D Reconstruction Test-Time Adaptation Self-Supervised Learning

Open paper AskPDF arXiv

Feed-forward 3D reconstruction models are efficient but rigid: once trained, they perform inference in a zero-shot manner and cannot adapt to the test scene. As a result, visually plausible reconstructions often contain errors, particularly under occlusions, specularities, and ambiguous cues. To address this, we introduce Free Geometry, a framework that enables feed-forward 3D reconstruction models to self-evolve at test time without any 3D ground truth. Our key insight is that, when the model receives more views, it produces more reliable and view-consistent reconstructions. Leveraging this property, given a testing sequence, we mask a subset of frames to construct a self-supervised task. Free Geometry enforces cross-view feature consistency between representations from full and partial observations, while maintaining the pairwise relations implied by the held-out frames. This self-supervision allows for fast recalibration via lightweight LoRA updates, taking less than 2 minutes per dataset on a single GPU. Our approach consistently improves state-of-the-art foundation models, including Depth Anything 3 and VGGT, across 4 benchmark datasets, yielding an average improvement of 3.73% in camera pose accuracy and 2.88% in point map prediction. Code is available at https://github.com/hiteacherIamhumble/Free-Geometry .

Abstract

Explore similar work

Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself

Reliev3R: Relieving Feed-forward Reconstruction from Multi-View Geometric Annotations

VLRC: Vision-Language Reprojection Consistency as a scalable signal for better feed-forward 3D pretraining