Scalable Adaptation of 3D Geometric Foundation Models via Weak Supervision from Internet Video

Zihui Gao; Ke Liu; Donny Y. Chen; Duochao Shi; Guosheng Lin; Jia-Wang Bian; Hao Chen; Chunhua Shen

cs.CVFeb 8, 2026

Scalable Adaptation of 3D Geometric Foundation Models via Weak Supervision from Internet Video

Authors: Zihui Gao, Ke Liu, Donny Y. Chen, Duochao Shi, Guosheng Lin, Jia-Wang Bian, Hao Chen, Chunhua Shen

Organizations: Zhejiang University, State Key Lab of CAD & CG · Independent Researcher · Nanyang Technological University · Zhejiang University of Technology

Abstract

Geometric foundation models show promise in 3D reconstruction. However, their progress is severely constrained by the scarcity of diverse, large-scale 3D annotations. While static-scene Internet videos are abundant and diverse, utilizing them as a scaling source for geometric learning is challenging due to the absence of ground-truth geometry and the presence of observational noise. To address this, we propose SAGE, a framework for Scalable Adaptation of GEometric foundation models from raw video streams. SAGE leverages a hierarchical mining pipeline to transform videos into training trajectories and hybrid supervisions: (1) Informative training trajectory selection; (2) Sparse Geometric Anchoring via SfM point clouds for global structural guidance; and (3) Dense Differentiable Consistency via 3D Gaussian rendering for multi-view constraints. Besides, we also introduce a regularization strategy that replays a small fraction of supervised data to prevent catastrophic forgetting. Applied to two architecturally distinct backbones, SAGE reduces Chamfer Distance on unseen benchmarks (7Scenes, TUM-RGBD, Matterport3D) by 20-42% with MV-DUSt3R and consistently improves VGGT. SAGE establishes Internet video as a viable and scalable adaptation source for geometric foundation models. Code is available at https://github.com/aim-uofa/SAGE.

Figures & tables

Figure 1: Overview of the SAGE framework for scaling a General-Purpose 3D Foundation Model. To overcome the Data Scarcity Bottleneck inherent in limited labeled 3D datasets, we propose a pipeline that leverages abundant static-scene Internet videos to achieve robust zero-shot generalization on complex unseen scenes. Bottom right: Scalability analysis demonstrates zero-shot reconstruction performance as training data scales from 100 to 10K video scenes, showing consistent improvement with increased data volume across various benchmarks (MP3D, 7-Scenes, and TUM).

Figure 2: Illustration of SAGE. For each video sequence, we sample context frames as model inputs and designate target frames for novel-view supervision, providing photometric constraints to refine the reconstructed 3D point cloud. Furthermore, we incorporate sparse 3D point clouds that provide consistent geometric constraints, complementing the photometric supervision from the target views.

(A) Foundation models trained from scratch with full 3D supervision (depth / poses)
	In-distribution		Zero-shot Generalization
Method	ScanNet		7Scenes		TUM-RGBD		Matterport3D
Method	[email protected] $\uparrow$	CD $\downarrow$	[email protected] $\uparrow$	CD $\downarrow$	[email protected] $\uparrow$	CD $\downarrow$	[email protected] $\uparrow$	CD $\downarrow$
Spann3R [ 30 ]	59.0	13.48	90.0	1.48	35.7	7.06	15.8	69.7
DUSt3R [ 35 ]	88.0	4.50	89.5	1.50	71.4	4.65	45.4	44.1
MapAnything [ 16 ]	79.5	3.05	88.1	1.56	82.0	2.96	51.8	24.0

Table 1: 8-view reconstruction results on four benchmarks at

224\times 224

resolution. Group (A) includes feed-forward foundation models trained from scratch with full 3D supervision, while Groups (B,C) adapt pretrained backbones using only Internet video, with no additional depth or pose annotations. SAGE consistently improves both MV-DUSt3R and VGGT, and matches or surpasses the strongest 3D-supervised models on the zero-shot 7Scenes and TUM-RGBD benchmarks. VDS denotes video-depth supervision using pseudo depth from Depth Any Video. SAGE (

n

K) uses

n

K video clips for adaptation. Subscripts in green / red show

\Delta

over the corresponding pretrained backbone. Underline : best within Group (A); bold : best within Groups (B,C).

	ScanNet		7Scenes		TUM		MP3D
Method	DAc $\uparrow$	CD $\downarrow$	DAc $\uparrow$	CD $\downarrow$	DAc $\uparrow$	CD $\downarrow$	DAc $\uparrow$	CD $\downarrow$
Baseline (Foundation)	88.2	1.79	89.6	0.95	87.5	2.60	30.3	43.2
Ours (Full)	91.2	1.69	98.0	0.76	93.1	1.85	48.0	24.9
Component Ablation:
w/o Sparse Anchors ( $\mathcal{L}_{\mathrm{anchor}}$ )	77.0	4.33	88.5	1.43	69.4	5.20	18.9	33.6
w/o Dense Consistency ( $\mathcal{L}_{\text{ren}}$ )	89.2	1.89	93.5	0.79	86.1	2.26	49.2	26.0

Table 2: Ablation study on hybrid supervision and training settings. We evaluate the contribution of each supervision component and the impact of different training strategies.

Figure 3: Empirical analysis of training data difficulty and its impact on model generalization. (Left) Training distribution across three datasets. (Right) Generalization performance (CD $\downarrow$ ) on three test sets under various mixing ratios of Re10K to DL3DV samples. Subplots indicate that while pure medium-hardness data (1:0) benefits simple scenes, a balanced mixture (1:1) yields the most robust generalization on large-scale environments like MP3D. The dashed red lines denote the pre-trained baseline performance.

Ratio	DAc $\uparrow$	CD $\downarrow$	DAc $\uparrow$	CD $\downarrow$	DAc $\uparrow$	CD $\downarrow$	DAc $\uparrow$	CD $\downarrow$
	ScanNet		7Scenes		TUM		MP3D
10:1	90.0	1.77	94.5	0.75	91.6	1.84	44.9	24.55
20:1	91.2	1.62	98.0	0.76	89.0	2.08	47.2	24.90
30:1	91.2	1.69	98.0	0.76	93.1	1.85	48.0	24.90
100:1	90.0	1.84	94.5	0.87	93.0	1.89	43.4	23.73

Table 3: Effect of Video-to-Supervised Data Ratio. Evaluation on four datasets with different video-to-supervised sample ratios. A 30:1 ratio is adopted as the default, balancing reconstruction quality with reliance on pretrained supervised data.

Table 7

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Standard 3D GFM Training and Inference Framework. A generic 3D GFM consists of an image encoder, a decoder for cross-view feature interaction, and optional output heads that regress various geometric representations. These models are commonly trained with explicit 3D supervision.

Figure 5: Video Data Processing Workflow for Weak Geometric Supervision. Raw Internet videos are filtered based on duration and resolution and processed using a COLMAP-based structure-from-motion pipeline. Feature extraction, exhaustive matching, and sparse reconstruction are performed to obtain sparse 3D points and camera parameters, which serve as weak geometric supervision for scalable model adaptation.

Statistic	Value
RE10K clips processed	$>$ 13K (50–200 frames per clip)
COLMAP failures	8 clips ( $<\!0.1\%$ )
Average processing time	${\sim}15$ sec / clip (170 frames), GPU-accelerated
Sparse points / frame	300–2000 (mean ${\sim}1500$ )
Total preprocessing time	${\sim}$ 5–6 h for $>$ 13K clips (4 $\times$ RTX 4090, 2 workers)
Reconstruction stage	sparse SfM only (no dense MVS)

Appendix

Table 6: COLMAP preprocessing statistics on the RealEstate10K subset. The pipeline scales to thousands of clips with negligible failure rate; preprocessing for the entire

10\text{K}{+}

scene set is a one-time, modest cost.

Method	4		12		16		20		24
Method	DAc	CD	DAc	CD	DAc	CD	DAc	CD	DAc	CD
VGGT	72.0	5.19	58.0	4.83	50.0	4.91	45.0	5.56	36.0	5.62
MV-DUSt3R	88.0	1.67	86.0	2.63	82.0	2.43	80.0	2.95	73.0	2.49
Ours	90.0	1.71	84.0	2.31	82.0	1.98	82.0	2.41	72.0	2.34

Appendix

Table 7: Multi-view 3D reconstruction performance on ScanNet. Comparison across five ScanNet scenes with varying number of images (4 to 24). Each column reports Distance Accuracy (DAc

\uparrow

) and Chamfer Distance (CD

\downarrow

Method	4		12		16		20		24
Method	DAc	CD	DAc	CD	DAc	CD	DAc	CD	DAc	CD
VGGT	44.1	40.65	24.6	46.28	19.5	57.45	13.3	57.56	13.3	78.27
MV-DUSt3R	42.9	42.08	23.0	49.28	24.0	62.29	11.7	77.89	7.1	92.19
SAGE (Ours)	62.2	19.28	35.2	30.41	30.1	34.73	18.9	39.69	14.3	52.61

Appendix

Table 8: Multi-view 3D performance on MP3D. Evaluation across different number of images. Each column reports Distance Accuracy ([email protected]

\uparrow

) and Chamfer Distance (CD

\downarrow

). All methods are evaluated without global alignment.

Method	ScanNet		7Scenes		TUM		MP3D
Method	DAc	CD	DAc	CD	DAc	CD	DAc	CD
Ours (extra)	88.0	2.31	100.0	0.74	87.5	2.53	30.6	39.91
Ours (inter)	90.0	1.88	100.0	0.48	91.7	1.78	46.4	21.80

Appendix

Table 9: Comparison of novel-view supervision strategies. Performance across four datasets using extrapolated vs. interpolated target views. Interpolated targets consistently yield better reconstruction results.

Method	ScanNet		7Scenes		TUM		MP3D
Method	DAc $\uparrow$	CD $\downarrow$	DAc $\uparrow$	CD $\downarrow$	DAc $\uparrow$	CD $\downarrow$	DAc $\uparrow$	CD $\downarrow$
A0B0: temporal jitter (Ours)	91.2	1.69	98.0	0.76	93.1	1.85	48.0	24.9
A1B0: flow-based context	91.4	1.63	95.5	0.83	93.4	1.89	49.1	24.6
A0B1: curated targets	90.6	1.70	96.4	0.89	92.3	1.86	48.0	25.1
A1B1: flow-based context + curated targets	90.4	1.63	97.9	0.78	91.2	1.86	49.5	23.4

Appendix

Table 10: Temporal vs. content-aware frame sampling. A0/B0: default temporal-jitter context sampling and random intermediate targets. A1: flow-based context selection. B1: overlap-curated targets.

Method	RRE $\downarrow$	RTE $\downarrow$	mAE $\downarrow$
$\pi^{3}$ [ 40 ]	0.3	3.2	15.4
Depth Anything 3 [ 20 ]	0.1	2.6	13.1
VGGT [ 31 ]	0.4	2.7	12.9
VGGT + SAGE	0.1	2.1	12.1

Appendix

Table 11: Cross-method pose estimation on 7Scenes. SAGE delivers competitive or superior pose accuracy over

\pi^{3}

, DA3, and VGGT, despite using only video-based weak supervision. Notably, VGGT+SAGE surpasses

\pi^{3}

on all three pose metrics.

Retention	ScanNet	7Scenes	TUM	MP3D
100% (default)	1.69	0.76	1.85	24.9
75%	1.93	1.13	1.89	29.9
50%	1.98	1.31	1.89	28.5
25%	1.79	1.12	1.94	26.6

Appendix

Table 12: Anchor density ablation: Chamfer Distance (

\downarrow

) under random subsampling of SfM points. SAGE exhibits graceful degradation: even at

25\%

retention, performance is largely preserved on three of four datasets, indicating that the dense differentiable consistency loss provides geometric supervision largely independent of anchor density.

Noise $\sigma$	ScanNet	7Scenes	TUM	MP3D
5%	1.92	0.96	1.85	27.2
20%	2.23	1.44	2.84	45.5
50%	3.15	5.03	7.28	68.3

Appendix

Table 13: Anchor quality ablation: Chamfer Distance (

\downarrow

) under additive Gaussian noise on SfM anchor points. Performance is essentially unchanged at

\sigma{=}5\%

and degrades only mildly at

\sigma{=}20\%

. Only an aggressive

50\%

perturbation causes severe drops, well beyond the noise levels expected from converged COLMAP reconstructions in practice.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
MV-DUSt3R [ 29 ]	20.72	0.66	0.188
SAGE (Ours)	20.98	0.67	0.184

Appendix

Table 14: Novel view synthesis quality on the ScanNet dataset. Comparison between MV-DUSt3R and our method (SAGE) using three standard metrics: PSNR (

\uparrow

), SSIM (

\uparrow

), and LPIPS (

\downarrow

). Higher PSNR/SSIM and lower LPIPS indicate better perceptual quality.

Figure 6: Generalization performance under a fixed number of training iterations, as the diversity of training scenes increases from 100 to 2K video scenes, demonstrating consistent performance gains from broader scene coverage.

Figure 7: Qualitative comparison of reconstructed point clouds across different methods, alongside ground-truth geometry. Zoomed-in views of representative regions are shown on the right. (Note: The 7Scenes ground-truth may appear slightly misaligned due to minor pose inaccuracies in the dataset.)

Figure 8: Qualitative results of reconstructed point clouds with 8 input views across diverse scenes, shown alongside ground-truth geometry. All examples are drawn from unseen scenes and demonstrate the generalization ability of the model.

Figure 9: Qualitative comparisons results of reconstructed point clouds views on challenging scenes, shown alongside ground-truth geometry. These examples feature low image overlap, texture-less surfaces (e.g., walls or floors), or inherently difficult matching conditions, and are drawn from previously unseen test scenes.

Figure 10: Qualitative visualization of rendered images and ground-truth views.

Figure 11: Qualitative visualization of rendered images and ground-truth views. 8 images are provided as model input, from which a 3D Gaussian representation is inferred and rendered to reconstruct the original 8 views as well as 2 novel viewpoints. Due to occlusions, novel-view renderings are limited to regions covered by the input views. The rendered images exhibit high visual fidelity and consistency with ground truth.

3D Foundation Models

3D foundation models learn reusable geometric and semantic representations for reconstruction, recognition, generation, and spatial reasoning.

See all 69 papers

Self-Geometry: GT-Free and Plug-and-Play Test-Time Adaptation for Geometrically Consistent 3D Vision Foundation Models
Aug 11, 2026Seokhyun Youn, Dahyeon Kye, Sung-Ho Bae +1Multi-View Consistency Multi-View Geometry
1CMLab, Chung-Ang University · 2Kyung Hee University
Glob3R: Global Structure-from-Motion with 3D Foundation Models
Jul 10, 2026Junyuan Deng, Heng Li, Kejie Qiu +7Bundle Adjustment Multi-View 3D Reconstruction
The Hong Kong University of Science and Technology · Tongyi Lab, Alibaba Group · Nanjing University +1
Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself
Apr 15, 2026Yuhang Dai, Xingyi YangFeed-Forward 3D Reconstruction Multi-View 3D Reconstruction
The Hong Kong Polytechnic University, Hong Kong SAR

Aug 11, 2026J/K move · Enter open · S save

Self-Geometry: GT-Free and Plug-and-Play Test-Time Adaptation for Geometrically Consistent 3D Vision Foundation Models

Seokhyun Youn, Dahyeon Kye, Sung-Ho Bae, Jihyong Oh

1CMLab, Chung-Ang University · 2Kyung Hee University

Multi-View Consistency Multi-View Geometry Camera Pose Estimation Vision Foundation Model Adaptation

Open paper AskPDF arXiv

Recent Vision Foundation Models (VFMs) predict depth, camera pose, and pointmap in a single forward pass without per-scene optimization, achieving strong generalization. However, enforcing explicit multi-view geometric consistency, e.g., through bundle adjustment, is computationally costly and is thus not imposed during VFM pretraining, so such inconsistency can arise. To address this, implicit self-consistency derived from model outputs (e.g., pointmaps, features), though enforced at test-time in prior work, delivers inherently limited performance gain, especially on scenes where the pretrained VFM is highly inaccurate. In contrast to this implicit signal, we propose Self-Geometry, a plug-and-play test-time adaptation pipeline that directly imposes explicit multi-view geometric constraints using 2D pixel correspondences as pseudo ground-truth. Our proposed Self-Geometry consists of Geometric Disentanglement Optimization, which combines Multi-View Consistency and Epipolar Consistency losses with Gradient Disentanglement to prevent gradient conflict; Frame Angular-Neighbor, a view sampler based on SO(3) geodesic distances for lightly imposing these constraints; and Lightweight TTA, which adapts VFMs via LoRA. Our method achieves consistent improvements in both pose and geometry estimation across six VFMs (VGGT,

π^3

, DA3-Giant/Large/Base/Small) and four benchmarks (7Scenes, ETH3D, ScanNet++, HiRoom).

Abstract

3D Foundation Models

Explore similar work

Self-Geometry: GT-Free and Plug-and-Play Test-Time Adaptation for Geometrically Consistent 3D Vision Foundation Models

Glob3R: Global Structure-from-Motion with 3D Foundation Models

Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself