Recent advances in vision foundation models have revolutionized geometry reconstruction and semantic understanding. Yet, most of the existing approaches treat these capabilities in isolation, leading to redundant pipelines and compounded errors. This paper introduces FF3R, a fully annotation-free feed-forward framework that unifies geometric and semantic reasoning from unconstrained multi-view image sequences. Unlike previous methods, FF3R does not require camera poses, depth maps, or semantic labels, relying solely on rendering supervision for RGB and feature maps, establishing a scalable paradigm for unified 3D reasoning. In addition, we address two critical challenges in feedforward feature reconstruction pipelines, namely global semantic inconsistency and local structural inconsistency, through two key innovations: (i) a Token-wise Fusion Module that enriches geometry tokens with semantic context via cross-attention, and (ii) a Semantic-Geometry Mutual Boosting mechanism combining geometry-guided feature warping for global consistency with semantic-aware voxelization for local coherence. Extensive experiments on ScanNet and DL3DV-10K demonstrate FF3R's superior performance in novel-view synthesis, open-vocabulary semantic segmentation, and depth estimation, with strong generalization to in-the-wild scenarios, paving the way for embodied intelligence systems that demand both spatial and semantic understanding.
Figures & tables
Figure 2: Architecture Overview. From unconstrained multi-view inputs, FF3R injects semantic-awareness into geometry tokens through Token-Wise Fusion, then decodes pixel-aligned features to predict feature-RGB GS, depth, and camera parameters. A Semantic–Geometry Mutual Boosting module, including Geometry-Guided Feature Warping, and Semantic-aware Voxelization, enables fully annotation-free training and yields high-quality novel view synthesis and open-vocabulary, 3D-consistent semantics.
Figure 3: Geometry-Guided Feature Warping. Before warping (middle), features show inconsistency (color shifts and boundary misalignment). After warping (right), features are spatially aligned across views and better match the ground-truth semantics, yielding crisper boundaries and fewer artifacts.
Figure 4: Semantic-aware Voxelization
Sparse
2 Views
6 Views
16 Views
PSNR ↑
SSIM ↑
LPIPS ↓
mIoU ↑
Acc. ↑
Time(s) ↓
PSNR ↑
SSIM ↑
LPIPS ↓
mIoU ↑
Acc. ↑
Time(s) ↓
PSNR ↑
SSIM ↑
LPIPS ↓
mIoU ↑
Acc. ↑
Time(s) ↓
ScanNet [ 3 ] Dataset
LSeg [ 15 ]
–
–
–
0.543
0.810
–
–
–
–
0.551
0.804
–
–
–
–
0.540
0.800
–
Feature-3DGS [ 49 ]
17.54
0.673
0.437
0.332
0.662
18min
18.34
0.695
0.430
0.330
0.661
18min
19.12
0.713
0.419
0.349
0.648
18min
LSM [ 4 ]
14.95
0.589
0.401
0.424
0.739
0.408s
14.50
0.580
0.438
0.392
0.708
16.3s
OOM
OOM
OOM
OOM
OOM
OOM
Ours
22.70
0.787
0.285
0.486
0.754
0.884s
22.19
0.769
0.295
0.500
0.751
1.2s
22.58
0.790
0.293
0.492
0.739
2.2s
Table 1: Quantitative Results on Novel View Synthesis and Semantic Segmentation. We evaluate sparse-view NVS and segmentation on ScanNet [ 3 ] , and dense-view NVS and segmentation on DL3DV-10K [ 20 ] , reporting 3D reconstruction time, image-quality metrics, and segmentation metrics.
Figure 5: Language-based 3D Segmentation Comparison. Qualitative results across eight scenes from the ScanNet [ 3 ] dataset using varying numbers of unconstrained input views. Our FF3R produces sharper boundaries, fewer artifacts, and stronger cross-view consistency than LSM [ 4 ] , Feature-3DGS [ 49 ] , and CLIP-LSeg [ 15 ] , demonstrating effective fusion of semantic information and geometric structure into a coherent 3D feature field.
Figure 6: Novel View Synthesis Comparison. We compare results under sparse and dense view settings on the ScanNet [ 3 ] and DL3DV-10K [ 20 ] datasets using unconstrained inputs. FF3R consistently outperforms all baselines, achieving sharper details and higher visual fidelity across both sparse and dense scenarios.
Method
2 Views
6 Views
Rel ↓
τ↑
Rel ↓
τ↑
LSM [ 4 ]
7.36
46.24
8.38
38.65
FF3R (Ours)
3.99
67.99
3.36
71.10
Table 2: Comparison of depth consistency under different input views. We report Absolute Relative Error (Rel ↓ ) and Inlier Ratio ( τ↑ ) with a threshold of 1.03 .
Method
TW Fusion
G → S
S → G
PSNR ↑
SSIM ↑
LPIPS ↓
mIoU ↑
Acc. ↑
Base
17.85
0.699
0.380
0.411
0.684
+ TW Fusion
✓
18.70
0.721
0.359
0.440
0.721
+ G → S
✓
✓
18.61
0.705
0.373
0.450
0.722
+ S → G
✓
✓
✓
19.12
0.718
0.357
0.449
0.725
Table 3: Ablation study of different components in FF3R.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Ours w/o distill loss
9.37
0.248
0.783
Ours w/ distill loss
17.85
0.699
0.380
Appendix
Table 4: Ablation for Geometric Distillation.
2 Views
16 Views
PSNR ↑
SSIM ↑
LPIPS ↓
mIoU ↑
Acc. ↑
Time(s) ↓
PSNR ↑
SSIM ↑
LPIPS ↓
mIoU ↑
Acc. ↑
Time(s) ↓
Uni3R
15.57
0.649
0.448
0.354
0.716
0.626s
17.98
0.737
0.402
0.437
0.769
0.928s
Ours
22.70
0.787
0.285
0.486
0.754
0.884s
22.19
0.769
0.295
0.500
0.751
1.2s
Appendix
Table 5: Quantitative Results on Novel View Synthesis and Semantic Segmentation on ScanNet.
Figure 7: Qualitative Results on Novel View Synthesis and Semantic Segmentation on ScanNet.
Figure 8: Qualitative results of open-volcabulory semantic segmentation.
Figure 9: Qualitative results of novel view synthesis.
Figure 10: Qualitative results on DL3DV-10K [ 20 ] demonstrating generalization across diverse indoor and outdoor scenes.
Feed-forward 3D reconstruction models are efficient but rigid: once trained, they perform inference in a zero-shot manner and cannot adapt to the test scene. As a result, visually plausible reconstructions often contain errors, particularly under occlusions, specularities, and ambiguous cues. To address this, we introduce Free Geometry, a framework that enables feed-forward 3D reconstruction models to self-evolve at test time without any 3D ground truth. Our key insight is that, when the model receives more views, it produces more reliable and view-consistent reconstructions. Leveraging this property, given a testing sequence, we mask a subset of frames to construct a self-supervised task. Free Geometry enforces cross-view feature consistency between representations from full and partial observations, while maintaining the pairwise relations implied by the held-out frames. This self-supervision allows for fast recalibration via lightweight LoRA updates, taking less than 2 minutes per dataset on a single GPU. Our approach consistently improves state-of-the-art foundation models, including Depth Anything 3 and VGGT, across 4 benchmark datasets, yielding an average improvement of 3.73% in camera pose accuracy and 2.88% in point map prediction. Code is available at https://github.com/hiteacherIamhumble/Free-Geometry .
Yuhang Dai, Xingyi Yang
The Hong Kong Polytechnic University, Hong Kong SAR
With recent advances, Feed-forward Reconstruction Models (FFRMs) have demonstrated great potential in reconstruction quality and adaptiveness to multiple downstream tasks. However, the excessive reliance on multi-view geometric annotations, e.g. 3D point maps and camera poses, makes the fully-supervised training scheme of FFRMs difficult to scale up. In this paper, we propose Reliev3R, a weakly-supervised paradigm for training FFRMs from scratch without cost-prohibitive multi-view geometric annotations. Relieving the reliance on geometric sensory data and compute-exhaustive structure-from-motion preprocessing, our method draws 3D knowledge directly from monocular relative depths and image sparse correspondences given by zero-shot predictions of pretrained models. At the core of Reliev3R, we design an ambiguity-aware relative depth loss and a trigonometry-based reprojection loss to facilitate supervision for multi-view geometric consistency. Training from scratch with the less data, Reliev3R catches up with its fully-supervised sibling models, taking a step towards low-cost 3D reconstruction supervisions and scalable FFRMs.
Youyu Chen, Junjun Jiang, Yueru Luo +4
Harbin Institute of Technology · The Chinese University of Hong Kong, Shenzhen · Huawei
Feed-forward 3D models are commonly trained using either expensive geometric supervision or self-supervised photometric objectives, both of which provide incomplete learning signals. We introduce Vision-Language Reprojection Consistency (VLRC), a scalable auxiliary objective that exploits frozen vision-language representations as semantic multi-view supervision. Given a predicted 3D reconstruction, VLRC reprojects dense vision-language features across views and enforces feature consistency between corresponding image locations, requiring no additional 3D annotations. The objective integrates seamlessly with both self-supervised monocular reconstruction and supervised-pretrained feed-forward 3D models during unlabeled adaptation. By aligning geometry with language-grounded features, VLRC not only improves depth and camera estimation but also enables more coherent multi-view semantic fusion for open-vocabulary 3D scene understanding. Experiments on indoor and outdoor benchmarks demonstrate consistent gains in 3D reconstruction accuracy and zero-shot open-vocabulary 3D semantic segmentation.
Marwane Hariat, David Filliat, Antoine Manzanera
U2IS, ENSTA – Institut Polytechnique de Paris · Agence Minist´erielle pour l’IA de D´efense